Payment webhooks that survive retries without doing the work twice

A customer pays once. Your application receives the notification twice and dispatches two orders. The payment provider has not necessarily charged twice; your fulfilment code has repeated a side effect.

That distinction matters when designing an integration. Stripe documents retries, duplicate events and delivery that can arrive out of order. A successful test payment does not exercise those conditions. Stripe's delivery behaviour.

The following is an architecture I would use as a starting point, not a claim that adding one database table makes every integration reliable. The details depend on the database, the payment flow and the systems receiving the resulting work.

Authenticate the original message

Keep the request body exactly as received until signature verification finishes. Use the provider's maintained verification library, the signature header and the correct endpoint secret; parsing and serialising JSON first can change the signed material. Stripe's signature guidance.

Apply request-size limits, reject invalid signatures and keep timestamp verification enabled. Validate the event schema and expected account and environment before creating business work. A valid signature is not permission to associate a payment with an arbitrary customer's order. Check that relationship against trusted records. Keep secrets and unnecessary payment details out of logs.

Acknowledge durable receipt, not an intention to save

I would give the endpoint a small job: authenticate, validate and store an inbox record. A database constraint such as UNIQUE(provider, account, event_id) prevents two concurrent deliveries creating two inbox entries. Use a non-null, normalised account identity and separate test/live namespaces.

Return a successful HTTP response only after the inbox transaction commits, or after confirming that the same authenticated event already exists durably. If storage fails, return an error. Sending success and then starting an untracked background task leaves a gap where a process crash can lose the work.

A worker can poll that inbox. Claiming work needs concurrency control and recovery from expired claims, so a crashed worker cannot abandon an entry forever. The acknowledgement means “received safely”, not “the customer's order is complete”.

Make the business transition repeatable too

Deduplicating event IDs is only the first boundary. Different events can refer to the same business operation. Give fulfilment its own stable identity, such as the order and fulfilment action, rather than assuming every event deserves another dispatch.

Within one database transaction, lock or conditionally update the relevant business record, enforce the allowed transition, insert a uniquely identified outbox action, and mark the inbox item processed. Commit them together. A crash before commit changes none of them; a later retry should discover the already-completed transition.

The outbox is a durable list of external work still to send. It closes the gap between changing your database and recording that an email, shipment or accounting update is required.

The outbox cannot promise exactly-once delivery

An external service may accept a request just before your worker loses its connection. Retrying can repeat the effect unless the recipient recognises the same operation.

For outbound Stripe API requests, idempotency keys protect supported retries; this is separate from recognising incoming webhook event IDs. Reuse a key for the same logical request and respect the API's parameter and retention rules. Stripe's idempotent request contract.

For another provider, verify its own contract. Persist operation identifiers and results. If it offers no reliable deduplication or status lookup, document the ambiguity and provide reconciliation or manual review. A local “sent” flag cannot remove a crash between remote acceptance and local confirmation.

Older messages must not undo newer facts

Treat an event as a reason to examine the relevant resource. Where appropriate, retrieve current authoritative payment state, then apply explicit business rules under concurrency control. The fetch itself is not an atomic lock on either system.

A late failure notification must not blindly turn a confirmed payment into an unpaid order. Equally, a later refund or dispute is a legitimate change and must not disappear behind a simplistic “paid is final” rule. Model those transitions separately; timestamps alone are not a sufficient ordering strategy.

Test recovery, and give failures an owner

Before release, I would test concurrent duplicates, reversed delivery order, a database outage before acknowledgement, a worker crash around commit, and remote acceptance followed by a timeout. Include account mismatches and invalid signatures. Verify durable outcomes, not merely HTTP responses.

Use bounded retries with backoff, alerts for exhausted attempts and a controlled replay path. Reconcile provider records against local records so missing work can be found even when delivery retries are exhausted. Track backlog age and unresolved items; an error counter without an owner changes nothing.

Discuss the payment and fulfilment journey you need. The useful deliverable is a tested path from payment to completed work, with recovery designed into it.