🔄 Topic

The personal secretary went quiet, and the root cause traced back to Google OAuth credentials expiring with no alert firing. I built an independent watchdog — then made myself prove the alarm actually rings before trusting it.

🎯 Goal

Stop silent OAuth expiry from taking down a system nobody is watching in real time, and make sure the watchdog built to catch it is itself proven to work, not just installed.

🛠 What I Did

I treated the outage as a monitoring failure first, and a credentials problem second.

Main areas covered:

  • diagnosed why the secretary died silently: a Google grant expired and nothing downstream noticed until real functionality broke
  • built an independent Google credential watchdog with loud alerts, deliberately separate from the pipeline whose health it reports on
  • delivered a one-time verdict when a grant outlives its 7-day cap, so an aging credential gets exactly one clear warning instead of silence followed by failure
  • confirmed the auth watchdog’s expiry cause with direct evidence rather than assuming the diagnosis was right
  • did not stop at “the watchdog exists” — deliberately forced the alarm condition and confirmed it actually fires, loudly, where I would see it
  • shipped a self-contained, copy-pasteable reauth command so recovery does not depend on remembering the right incantation at 11pm

🔗 Key Cybersecurity Connections

A monitor that has never been proven to fire is a monitor you believe in, not a monitor that works. This is the same lesson as the hermes-verify false-positive investigation and the fail-closed supervisor work — except inverted: instead of a check crying wolf, this was a check that stayed silent when it should have screamed. Silent failure of a monitor is worse than a noisy false positive, because nothing draws attention to it.

Building the watchdog independently of the system it watches matters too — a health check that shares a failure mode with the thing it monitors can go dark at exactly the moment it’s needed.

🔍 Investigation Questions

  • Was the credential expiry itself the root cause, or a symptom of something not watching for it?
  • Does the watchdog share any dependency with the system it monitors?
  • Has the alarm condition ever actually been triggered and observed, or only coded and assumed?
  • Is recovery documented as a copy-pasteable step, or as tribal knowledge?
  • What is the blast radius of one more silent expiry before this fix?

🚨 Detection Opportunities

Checks for a credential-health watchdog:

  • a grant approaching or past expiry with no alert raised
  • watchdog logic sharing infrastructure with the system it monitors
  • an alert path that has never been deliberately tested
  • recovery steps that exist only in memory, not in a runnable command
  • repeated silent failures of the same category before the first real fix

Example:

project=google-auth-watchdog
signal=credential_expiry_with_no_alert_raised
risk_area=unmonitored_dependency
triage=confirm_watchdog_independence_and_force_test_the_alarm

🧭 MITRE ATT&CK Techniques

No direct mapping claimed. This is availability monitoring and operational-resilience work for a credential dependency.

🗺 Visual Investigation Diagram

Google grant expires
    ↓
Old: nothing notices until real failure
    ↓
Independent watchdog built
    ↓
Deliberately force the alarm condition
    ↓
Confirm it actually fires, loudly
    ↓
One-time verdict + copy-pasteable reauth

⚠ Challenges

The uncomfortable question was “how do I know this watchdog works?” — and the honest answer, before today, was that I didn’t. Forcing the condition felt like extra work for something that “should obviously work.” It wasn’t extra work; it was the actual point.

📚 What I Learned

I learned that “I built a monitor” and “I proved the monitor fires” are different claims, and only the second one is worth anything during an actual incident.

➡ Next Steps

  • Schedule a periodic forced test of the watchdog, not just a one-time proof
  • Extend the same independence principle to other credential-backed integrations
  • Keep the one-time-verdict pattern for other aging-grant scenarios
  • Document the forced-test procedure so it survives me forgetting it exists

🧠 Reflection

An alarm that has never gone off is a hypothesis, not a safeguard. Making it ring on purpose, before I needed it to, is what turned this from a plausible fix into a trusted one.

🧩 Lessons Learned

What worked

Building the watchdog independently of the system it monitors, then deliberately forcing the alarm to prove it fires.

What broke

Google OAuth expired silently and took down the personal secretary with no warning.

Why it broke

Nothing was watching the credential’s health independently of the pipeline that depended on it.

Fix / takeaway

Monitors need proof of firing, not just proof of existing — and they should never share a failure mode with what they watch.

📈 Skill Progression Context

This supports my cybersecurity progression because credential lifecycle monitoring, independent health checks, and proving alerts actually fire are foundational SOC and reliability-engineering skills.

😄 TL;DR

Built an alarm for expiring credentials, then made myself prove it actually rings before trusting it.