Webhooks that survive

token.refresh_reuse_detected fires at 3 AM. Your SOC tooling needs to hear about it — and “we POST some JSON at your URL” is the easy 10%. The hard 90% is what happens when your receiver is deploying at that moment, how the receiver knows the POST is really from Sentinel, and what stops one dead endpoint from silently eating a week of security events. Sentinel’s answer is a durable outbox with signed deliveries, an explicit retry ladder, dead-lettering, and a circuit breaker — and the companion sample plays both sides of the contract in one app.

The outbox, not fire-and-forget

// file: HooksComposition.cs
services.AddSentinelWebhooks(webhooks ?? delegate { });   // the durable outbox
services.AddSentinelWebhookDispatcher();                  // the hosted pump

AddSentinelWebhooks rewires ISentinelEventSink into a composite: your sink still gets every event, and events matching a subscription are enqueued as durable delivery rows before any HTTP happens. The dispatcher — a hosted service — claims due rows in batches and attempts them. Emit and deliver are decoupled on purpose: a login never waits on your SOC’s HTTP endpoint, and a crashed process never loses an accepted event.

The timings are all options with production-shaped defaults:

WebhookDispatcherOptions	Default
PollInterval	5s
RequestTimeout	10s
RetryBackoff	1m, 5m, 30m, 2h, 12h
DisableAfterConsecutiveFailures	10

Subscribing

Subscriptions are webhook endpoints: a URL plus event-kind patterns (login.*, token.refresh_reuse_detected, * — single-star wildcards), realm-level or fenced to one org. Manage them through WebhookAdminService or its HTTP surface, MapSentinelWebhookAdmin() at /sentinel-admin/webhooks — authorization rides the same delegated-admin scopes as article 005 (sentinel:global:manage for realm-level endpoints).

POST /sentinel-admin/webhooks/
Authorization: Bearer <admin access token>

{ "url": "http://localhost:5000/billing/events", "eventKinds": ["login.*"] }

The 201 carries the endpoint view and the whsec_ secret — a 256-bit value shown exactly once, at creation or rotation. Every later read returns the endpoint without it. A_failed_login_arrives_as_a_signed_verified_delivery asserts the view never leaks whsec_ again.

The delivery contract

Every delivery is a POST with three headers and a stable envelope:

Header	Carries
X-Sentinel-Signature	t=<unix-seconds>,v1=<hmac-sha256 hex>
X-Sentinel-Event	the event kind, e.g. login.failed
X-Sentinel-Delivery	the delivery id — identical across retries, your dedupe key

{ "id": "…", "kind": "login.failed", "occurredAt": "…",
  "realmId": "…", "organizationId": "…", "subjectId": "…", "data": { } }

The body is serialized once at enqueue and re-sent byte-identical on every retry — which is what makes signature verification over exact bytes possible at all.

The receiver recipe

This is the part you write, and the sample’s BillingReceiver is the recipe verbatim:

// file: BillingReceiver.cs
// 1-5 of the recipe: exact bytes, parse t/v1, tolerance window, HMAC-SHA256 over
// "{t}.{body}", constant-time compare — all inside Verify.
if (Secret is null
    || !WebhookSignature.Verify(Secret, signatureHeader, body, DateTimeOffset.UtcNow))
{
    return StatusCodes.Status400BadRequest;
}

// Deduplicate on X-Sentinel-Delivery: retries re-send the same delivery id.
if (_seen.Add(deliveryId))
{
    _accepted.Add(new AcceptedDelivery(deliveryId, eventKind, body));
}

return StatusCodes.Status204NoContent;

Read the raw request body before any JSON parsing — re-serialization would change the bytes and kill the MAC. WebhookSignature.Verify handles the rest: header parsing, a 5-minute default timestamp tolerance (replay bound), HMACSHA256(secret, "{t}.{body}"), constant-time comparison. The receiver endpoint itself is anonymous by design — authentication IS the signature. Verify_accepts_the_exact_bytes_and_rejects_tampering_and_stale_timestamps pins all three properties with nothing but the public API.

Surviving the outage

Success is any 2xx. Anything else — a 500, a timeout, a connection refused — schedules a retry at now + RetryBackoff[attempt-1]. The sample’s outage test tells the whole story in four assertions: the receiver 500s once, heals, and

// An_outage_is_retried_with_the_same_delivery_id_until_the_receiver_heals
host.Receiver.TotalRequests.Should().Be(2);   // two HTTP attempts…
host.Receiver.Accepted.Should().HaveCount(1); // …one accepted delivery (dedupe absorbed the retry)
item.GetProperty("attemptCount").GetInt32().Should().Be(2);
item.GetProperty("deliveredAt").ValueKind.Should().NotBe(JsonValueKind.Null);

When the outage outlives the ladder, the delivery is abandoned: marked dead with its lastError, and a webhook.abandoned event goes to the (working) sinks. Nothing retries forever, and nothing disappears — the row stays in the delivery log, GET /sentinel-admin/webhooks/{id}/deliveries, with attemptCount, nextAttemptAt, lastError, abandoned.

Two more safety valves ride along. An endpoint that fails DisableAfterConsecutiveFailures times in a row — across deliveries — is disabled outright (webhook.endpoint_disabled), so a decommissioned URL can’t grind the dispatcher; re-enabling it through the admin surface resets the counter. And a webhook.* event about an endpoint is never enqueued to that endpoint — no failure-notification recursion.

Secret rotation closes the loop: After_rotation_only_the_new_secret_verifies_and_the_retry_heals_the_gap rotates, lets the next delivery fail verification against the receiver’s stale secret, updates the receiver, and watches the automatic retry land under the new one — the retry ladder doubling as the rotation grace window.

What to subscribe to

The kinds worth a pager: token.refresh_reuse_detected (article 003), abuse.account_locked (article 013), risk.blocked and risk.stepup (article 014), breakglass.login, plus the saml.* and federation.* streams. Admin mutations are not events — they land on the tamper-evident ledger instead, which is article 012’s subject. The operations page keeps the full catalog.