The 3 A.M. Page

The story

At 2:47 a.m. on a Sunday, Halvard’s identity provider began timing out. Not failing — timing out, the worst kind of down, the kind that takes thirty seconds per attempt to tell you nothing. At 2:52, a Halvard physician at the Northshore urgent-care site had a patient in anaphylaxis and a records system that wouldn’t finish logging her in — allergies, medication history, all of it behind a spinner owned by someone else’s outage.

The night lead did the thing they had rehearsed in March, and June, and September: opened the sealed envelope in the charge station, and logged in as emergency.access@meridian.health.

Three things happened in the next four seconds, and Ana — asleep — was involved in all of them. The break-glass login fired its alarms: events onto the ledger, signed webhooks to the paging service, which is why her phone lit up with breakglass.login instead of a mystery. The account’s permissions were capped — flagged sentinel:break_glass, its grants filtered through the break-glass policy at snapshot time, so the session could read charts and medication lists but couldn’t touch admin, exports, or anything else an attacker holding a stolen envelope might prefer; the capping preserves denies, so nothing widened. And a rotation requirement was stamped: the credential was now burned by definition, drill or no drill.

The physician had the allergy list at 2:53. The patient was stable by 3:10. Ana rotated the break-glass credential at 8 a.m., resealed the envelope, and wrote the incident note over breakfast; the ledger had already written most of it. The part she kept thinking about was the September drill — the one people had grumbled through. Sentinel tracks break-glass drill freshness as a status; hers had said current, and at 2:52 a.m., “current” meant a night lead whose hands didn’t shake.

Why this is hard the traditional way

Every operations team has an emergency-access story, and most of them are embarrassing on one of two axes. Either the escape hatch doesn’t exist — and the 3 a.m. answer is a heroic engineer with production database access editing rows during a medical emergency — or it exists as a shared “admin” password that is all-powerful, never rotated, known to alumni, and logged nowhere. The first fails patients; the second fails audits, and eventually patients too.

The failure mode is structural: break-glass access is used approximately never, so nothing about it stays true on its own. The password drifts into wikis, the people who knew the procedure leave, the account quietly accumulates permissions because scoping it down was always someone’s next sprint. Untested emergency access is a prop — you find out what it actually does during the emergency, which is the only unacceptable time.

How Sentinel changes the ending

Sentinel treats break-glass as a modeled capability with a lifecycle, not a password in a drawer. The account is an ordinary user flagged sentinel:break_glass — no parallel login path to secure, every existing control still applies. The flag changes three behaviors, each tested in the suite: the capping data source filters the account’s snapshot to the policy’s patterns (broad grants cannot leak through, and denies survive: Flagged_user_with_broad_grants_only_evaluates_capped_patterns), login alerts operators and records the rotation requirement (Login_alerts_operators_and_records_events_and_rotation_requirement), and the drill clock is first-class — status goes stale when you haven’t rehearsed within the interval (Drill_status_goes_stale_after_the_interval), which turns “we should practice” into a dashboard light someone owns.

The alarms ride the same signed webhook machinery as everything else — timestamp-bound HMAC, retries with backoff — so the page that woke Ana was verifiable, deduplicated, and would have survived a transient network blip on the paging service’s end. Nothing about the night depended on anyone’s memory. That was the point.

What it costs you to ignore this

• The alternative to designed break-glass is undesigned break-glass — a DBA in production at 3 a.m., untracked, unscoped, and unrepeatable.
• An uncapped emergency account is your highest-value credential with your weakest controls: rarely rotated, widely known, never watched.
• Silent emergency access is indistinguishable from a breach — if the envelope can be used without waking anyone, you’ll learn about misuse from the consequences.
• In regulated care, both failures are reportable: the access you couldn’t grant in an emergency, and the access you granted and can’t account for.