Security recipes · 10

Webhook forgery

The threat
POST fabricated events to a consumer's webhook endpoint, or re-send a captured legitimate delivery later, to trigger downstream actions the sender never took.
Sentinel's counter
Every delivery is signed with a timestamp bound into the HMAC (X-Sentinel-Signature t=,v1=), verified constant-time inside a freshness window, with a delivery id for idempotent consumption.
Capability
/docs/operations/

The attack

A webhook receiver is, from the network’s point of view, an unauthenticated POST endpoint that changes state. If yours deprovisions accounts on user.suspended, pages on-call on token.refresh_reuse_detected, or reconciles billing on login events, then anyone who can find its URL can drive those actions — URLs leak through config repos, error trackers, proxy logs and old tickets. The attack comes in two shapes:

  • Forgery: craft a plausible JSON body and POST it. No signature check — or a check against a guessable secret — and your SIEM fills with fiction, or your offboarding automation suspends whoever the attacker names.
  • Replay: capture one legitimate delivery (a proxy log is enough — headers and body are all there) and re-send it later, verbatim. A signature check alone passes: the bytes are authentic. Only the time is wrong.

A signature without a timestamp stops the first and not the second. Sentinel’s wire format exists to stop both with one check.

How Sentinel counters it

The timestamp is bound into the MAC. Every delivery carries X-Sentinel-Signature: t=<unix-seconds>,v1=<hex>, where v1 is HMACSHA256(secret, "<t>.<body>") — the timestamp is signed with the body, not alongside it. A replayer faces a fork with no good branch: keep t and fail the consumer’s freshness window, or move t and break the MAC. Forgery needs the endpoint’s secret — a whsec_-prefixed 256-bit random value minted by WebhookSecrets.NewSecret(), greppable in leaks like an API key.

Verification is a published recipe, and a method. The consumer side is five steps, documented on the class and encoded in WebhookSignature.Verify(secret, signatureHeader, body, now, tolerance):

  1. Read the raw request body — verify the exact bytes received, before any JSON parsing or re-serialization.
  2. Parse t and v1 from the header.
  3. Reject when |now − t| exceeds the freshness window (DefaultTolerance is 5 minutes, in both directions — future-dated deliveries fail too).
  4. Recompute HMACSHA256(secret, "<t>.<body>"), lowercase hex.
  5. Compare in constant time (CryptographicOperations.FixedTimeEquals) — no timing oracle on the match — and deduplicate on the X-Sentinel-Delivery header, the idempotency key that stays stable across retries.

Malformed headers, stale timestamps, wrong secrets and tampered bodies all return the same false (Verify_rejects_tampered_bodies_stale_timestamps_wrong_secrets_and_garbage); the wire format and the recipe’s round-trip are pinned by Signature_has_the_documented_wire_format and Consumer_can_verify_by_recomputing_the_hmac_from_the_documented_recipe, and the HTTP suite proves it end to end from the dispatcher (Emitted_event_is_delivered_to_the_receiver_with_a_verifiable_signature). The Meridian demo’s AuditForwarder is a working reference consumer: verify, dedupe, accept.

Routing metadata rides in headers, and delivery is disciplined. X-Sentinel-Event names the event kind so you can route before parsing an unverified body. Each retry of a failed delivery is re-signed with a fresh timestamp for its attempt, walking an exponential backoff (1m, 5m, 30m, 2h, 12h; then dead-letter with webhook.abandoned, and a circuit breaker that auto-disables a persistently failing endpoint) — so honoring the freshness window never costs you legitimate retries. Endpoints are realm- and org-fenced on the sending side: an org-scoped endpoint only ever receives its own org’s events.

What your app must still do

  • Actually call Verify — on the raw body, before parsing. The classic failure is verifying a re-serialized body: one reordered JSON key and honest deliveries fail while you’re tempted to “loosen” the check. Read bytes, verify, then parse.
  • Dedupe on X-Sentinel-Delivery. Retries are at-least-once by design; your handler must be idempotent. A small table (or cache with TTL beyond the retry horizon) of processed delivery ids is the standard shape.
  • Answer 2xx fast, work later. Verification is cheap; your downstream action may not be. Acknowledge, enqueue, and process out-of-band so slow work doesn’t turn into spurious retries — or an auto-disabled endpoint.
  • Guard the secret like the credential it is. Store it in your secret manager, not in the config repo next to the URL. Rotation is an admin-API mutation — pair it with a deploy of the consumer’s copy.
  • Serve the receiver over HTTPS and keep its URL boring. The signature authenticates the sender; TLS keeps captured-traffic replay material and the endpoint itself off the table in the first place.