Skip to content

Fix orbit crash loop on incorrect file permissions#40887

Open
bash-bandicoot wants to merge 3 commits intofleetdm:mainfrom
bash-bandicoot:bb/fix-orbit-metadata-file-permissions
Open

Fix orbit crash loop on incorrect file permissions#40887
bash-bandicoot wants to merge 3 commits intofleetdm:mainfrom
bash-bandicoot:bb/fix-orbit-metadata-file-permissions

Conversation

@bash-bandicoot
Copy link

@bash-bandicoot bash-bandicoot commented Mar 3, 2026

Summary

  • checkPermFile in pkg/secure/secure.go now self-heals incorrect file permissions via os.Chmod instead of returning a fatal error
  • Fixes orbit crash-looping indefinitely when /opt/orbit/updates-metadata.json has mode 755 instead of the expected 600

Problem

Orbit refuses to start when updates-metadata.json has wrong permissions (e.g. 755 instead of 600), entering an infinite restart loop (systemd restart counter observed at 3447+). The manual workaround is chmod 600 /opt/orbit/updates-metadata.json, but the root cause — an external process changing file permissions — is intermittent and hard to track.

The checkPermFile function in pkg/secure/secure.go was designed as a security check, but its behavior of fatally erroring on any permission mismatch causes a denial-of-service on the legitimate user. For comparison, checkPermPath (the directory equivalent) already tolerates permissions that are less permissive than expected.

Fix

When checkPermFile detects a permission mismatch, it now attempts os.Chmod to correct the permissions before proceeding. It only returns an error if the chmod itself fails (e.g. insufficient privileges). This preserves the security intent — files end up with correct permissions — while making orbit resilient to external permission drift.

Test plan

  • go test ./pkg/secure/ -v -run TestOpenFile — verifies self-healing behavior
  • go test ./pkg/secure/ -v -run TestMkdirAll — unchanged, verifies directory checks still work
  • Manual: create /opt/orbit/updates-metadata.json with mode 755, start orbit, confirm it self-heals and starts normally

@bash-bandicoot bash-bandicoot requested a review from a team as a code owner March 3, 2026 20:14
@bash-bandicoot bash-bandicoot force-pushed the bb/fix-orbit-metadata-file-permissions branch from 3d7e39d to da2641d Compare March 3, 2026 20:28
Copy link
Member

@lucasmrod lucasmrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Please add a changes file to changes/ (fleetctl) and orbit/changes/ (fleetd), because the involved method is used by fleetctl and by orbit

@lucasmrod lucasmrod self-assigned this Mar 3, 2026
@bash-bandicoot
Copy link
Author

@lucasmrod , see last commit

@codecov
Copy link

codecov bot commented Mar 3, 2026

Codecov Report

❌ Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.31%. Comparing base (29cb62f) to head (8efca4b).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
pkg/secure/secure.go 0.00% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #40887      +/-   ##
==========================================
- Coverage   66.31%   66.31%   -0.01%     
==========================================
  Files        2469     2469              
  Lines      197788   197846      +58     
  Branches     8748     8678      -70     
==========================================
+ Hits       131163   131199      +36     
- Misses      54759    54774      +15     
- Partials    11866    11873       +7     
Flag Coverage Δ
backend 68.10% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lucasmrod
Copy link
Member

I've created #40934 for tracking purposes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants