🔄 Topic

Finding that a scheduled Hermes watchdog was versioned and manually runnable, yet silently blocked in the actual scheduler because the deployed path was a symlink.

🎯 Goal

Make the scheduled watchdog run from the same artifact and path rules that the real Hermes cron runner enforces.

🛠 What I Did

I had created a watchdog to alert when a scheduled greeting never ran. A manual wrapper invocation worked, but the real cron runner refused the script because its resolved path escaped the allowed ~/.hermes/scripts directory through a symlink.

The repair used a real deployed file inside the scheduler’s accepted directory, retained a versioned source copy, and added a drift test that fails in either of two cases:

  • the deployed item is a symlink;
  • the deployed and versioned files differ.

The important verification was scheduler-driven: the real hermes cron run path succeeded, recorded a healthy status, and showed no delivery error. Manual execution alone would not have proved that.

🔗 Key Cybersecurity Connections

This is deployment integrity and environment parity. A task that works in an interactive shell can fail under the scheduler’s path-containment, working-directory, environment, or permission rules.

🔍 Investigation Questions

  • Which exact file path does the scheduler resolve?
  • Does path containment allow a symlink target?
  • Is the deployed copy identical to the reviewed source?
  • Did the scheduler itself run the job successfully?
  • Can a future source edit drift away from the deployed file?

🚨 Detection Opportunities

Useful checks:

  • symlink detected in scheduler-owned script paths
  • source/deployed hash mismatch
  • manual run succeeds but scheduler run fails
  • missed-task watchdog has no recent successful receipt

Example:

event=scheduled_script_refused
reason=resolved_path_outside_allowlisted_directory
action=deploy_real_file_and_verify_scheduler_path

🧭 MITRE ATT&CK Techniques

Relevant defensive context:

  • T1053 — Scheduled Task/Job, because scheduled execution has its own security boundary
  • T1574 — Hijack Execution Flow, where path resolution and unintended artifacts matter

🗺 Visual Investigation Diagram

Versioned source file
    ↓
Real deployed scheduler file
    ↓
Hash / non-symlink drift test
    ↓
Scheduler-owned execution
    ↓
Receipt or actionable failure

⚠ Challenges

The failure was easy to miss because the manual wrapper took a different path through the system and bypassed the scheduler’s containment check.

📚 What I Learned

I learned that deployment is part of the program. The actual loader, resolver, and scheduler environment are not implementation details—they are the runtime truth.

➡ Next Steps

  • Apply source/deployment parity tests to other scheduled Hermes scripts
  • Prefer scheduler-owned tests for scheduled behavior
  • Keep drift detection mechanical instead of relying on deployment memory

🧠 Reflection

“I ran it and it worked” is weak evidence for an unattended job. The job must prove itself through the same path it will use tomorrow.

🧩 Lessons Learned

What worked

Deploying a real file and testing the real scheduler.

What broke

Treating a symlink plus a manual invocation as a deployment test.

Why it broke

The scheduler had a stricter path-authority rule than the wrapper.

Fix / takeaway

Verify the exact path, loader, and environment that own unattended execution.

📈 Skill Progression Context

This reinforced a core operational-security habit: compare the reviewed source with the artifact and execution path that will actually run.

😄 TL;DR

The watchdog existed, but not in a form the scheduler trusted. A real deployed file and scheduler-driven proof fixed that.