Lab Objective

Practice the full lifecycle of handling a noisy detection, using a real case from my own tooling:

  • reproduce a false-positive alert reliably
  • read the check’s actual condition instead of its intention
  • identify the overbroad clause causing the noise
  • scope the check to its real signal
  • prove that true positives still fire after the fix

The specific script is a verifier for autonomous agent tasks, but the workflow is the same one a SOC analyst applies to any noisy rule.

Lab Environment

  • System: macOS workstation, local agent stack
  • Component: hermes-verify — a script that checks an agent task’s results after it finishes
  • Failing check: “dirty tree” — flags MISMATCH if the git working tree has unexpected changes after a task
  • Symptom: a streak of MISMATCH alerts on tasks that were verifiably fine
  • Tools: Bash, git status --porcelain, git pathspecs, a scratch repo for testing

You can reproduce the core exercise in any git repository — the verifier logic in this lab is simplified to its essential condition.

Scenario

An autonomous agent completes tasks and a verification script judges each one. Lately every task ends with:

MISMATCH: working tree dirty after task

But manual review shows the tasks are clean. The alert has fired so often that I catch myself skimming past it — which is the actual emergency. A detection nobody believes protects nothing.

The investigation question: is the working tree really dirty, or is the check looking at the wrong thing?

Commands Practiced

Command Purpose
git status --porcelain Machine-readable working tree state
git status --porcelain -- <path> Scope the same question to specific paths
git diff --stat See what actually changed, and where
bash -x script.sh Trace a script’s real execution
mktemp -d / scratch repo Build a safe reproduction environment

Step 1 - Reproduce the Alert

First rule: never tune a detection you cannot reproduce. I ran a known-clean task and captured the verifier’s output.

./hermes-verify --task last
MISMATCH: working tree dirty after task

Reliable reproduction on a task I had just manually confirmed was clean. The alert is definitely false — now it can be studied.

Step 2 - Read What the Check Actually Does

The intention was “the task should not leave unexpected changes.” The implementation was:

if [ -n "$(git status --porcelain)" ]; then
  echo "MISMATCH: working tree dirty after task"
fi

Read it literally: this fires if anything anywhere in the repository is modified or untracked. Not “changes the task made” — any change, from any source.

Step 3 - Find the Noise Source

So what was dirtying the tree?

git status --porcelain
 M memory/capability_scores.json
?? handoff/2026-07-14-session.md
?? logs/agent-run-0712.log

None of these came from the task under verification. They came from other activity in a busy repo: telemetry updates, handoff notes, log files. The repository got busier over the weeks; the check was written when a task was the only thing running.

This is the classic false-positive anatomy: the condition observes more state than the thing it verifies.

Step 4 - Scope the Check to Its Signal

The task manifest already knows which paths a task is allowed to touch. The fix: ask git about those paths only.

# before: the whole world
git status --porcelain

# after: only the task's declared paths
git status --porcelain -- "${TASK_PATHS[@]}"

Now the condition matches the intention: “did this task’s footprint change unexpectedly?”

Step 5 - Prove True Positives Still Fire

A quieter detection is only an improvement if it still catches the bad case. In a scratch repo, I simulated a misbehaving task that writes outside its manifest — and inside it.

mkdir -p /tmp/verify-lab && cd /tmp/verify-lab && git init -q
# ... create a fake task with declared paths, then dirty a declared path
echo "rogue edit" >> declared/file.txt
git status --porcelain -- declared/
 M declared/file.txt

The scoped check fires on a real violation and stays silent on unrelated repo activity. Known-bad test kept as a permanent regression test.

Step 6 - Close the Loop

  • fix committed with the investigation written up as a handoff
  • known-bad case added so the check can never silently rot again
  • alert streak ended: the next MISMATCH will mean something

Security Takeaways

  1. A noisy detection is an incident. Every false alert spends trust, and trust is what makes the true alert work.
  2. Read the condition, not the comment. The check’s intention was fine; its literal condition was overbroad. Detections do what they say, not what they meant.
  3. Scope detections to their signal. “Anything changed anywhere” is rarely the question. “Did the thing I’m verifying change” usually is.
  4. Keep a known-bad test. Tuning without a true-positive test is how detections get quietly lobotomized.
  5. Notice your own alert fatigue. The moment I started skimming past MISMATCH was the real finding of this investigation.

Where This Applies Beyond My Setup

The same pattern covers SIEM rules that alert on whole-host activity instead of the monitored process, file-integrity monitoring that watches directories full of legitimate churn, and CI checks that fail on unrelated repository state. The tools change; the triage does not: reproduce, read the condition, scope it, retest known-bad.