What to do when webhooks fire repeatedly: AI Agent idempotency, deduplication, and compensation design
After integrating a webhook with an AI Agent, the hardest failures to diagnose are often not missing events, but the same event being processed twice: customers receive duplicate messages, two copies of a ticket are created, inventory is deducted repeatedly, and scheduled tasks may become increasingly chaotic with each retry. Sender retries, network timeouts, and consumer crashes can all cause duplicate callbacks. The right goal is not to require webhooks never to repeat, but to ensure that duplicate events do not produce a second business outcome.
First, identify the layer where duplication occurs
Start by saving the event ID, source, receipt time, request body digest, processing status, and external operation receipt. If two requests have the same event ID, the sender has usually retried the event. If the event IDs differ but the business object is the same, the upstream system may have generated the event more than once. If only one request was received but two results were produced, investigate internal retries or concurrent execution in the consumer. Without this evidence, hastily disabling retries will merely replace “duplicate events” with “lost events.”
Use an idempotency key to block a second business write
Give priority to a stable event ID provided by the sender. If no event ID is available, create an idempotency key by combining the source, event type, business object ID, and version number. Do not simply hash the entire request body: changes to timestamps, field order, or signatures can cause the same business event to produce different hashes.
The database should enforce a unique constraint on the idempotency key. When processing begins, atomically insert a “processing” record first. If a conflict occurs, read the previous status: if processing has succeeded, return the original result; if it is still in progress, wait briefly or return a retryable status; if it has failed, retry according to the defined policy. Maintaining a deduplication set only in memory is unreliable because it will fail after a process restart or when multiple instances run concurrently.
What an idempotency record should store at a minimum
| Field | Purpose | Acceptance criterion |
|---|---|---|
| Idempotency key | Identify the same event | A unique constraint is enforced |
| Processing status | Distinguish between processing, success, and failure | Status transitions are traceable |
| Result summary | Reuse the previous result for duplicate requests | External systems are not called again |
| Expiration time | Control the deduplication window | Longer than the sender’s maximum retry period |
Separate receipt acknowledgment from actual execution
The webhook endpoint should quickly complete signature verification, idempotency registration, and queue insertion before returning an acknowledgment. Time-consuming model calls, file processing, and cross-system writes should be handled by background tasks. This reduces the likelihood that the sender will retry because of a response timeout and also makes concurrency easier to control. For the boundaries between APIs, webhooks, and skills, see the guide to integrating AI Agents with existing tools.
Prevent duplicate external operations separately
Successfully deduplicating an event does not mean subsequent actions are inherently safe. Operations such as sending messages, creating orders, and writing tickets should each have a separate operation key, and the ID returned by the target system should be saved. If a task crashes after the external operation succeeds but before the internal status is persisted, query the receipt during recovery before deciding whether to persist the missing status or execute the operation again. Rerunning the entire chain directly is the most dangerous approach.
How to divide responsibilities among retries, compensation, and dead-letter queues
Network instability, rate limiting, and temporary 5xx errors can be handled with exponential backoff retries and a maximum attempt limit. Deterministic failures such as invalid parameters or insufficient permissions should not be retried repeatedly. If an error occurs after some steps have succeeded, perform an explicit compensation action or send the event for manual verification. Events that exceed the retry budget should enter a dead-letter queue, retaining the original payload, failed step, and receipts. For a more complete recovery strategy, see the AI Agent workflow monitoring and recovery guide.
Run four fault-injection exercises before launch
- Send the same event twice in succession and confirm that only one business outcome is produced.
- Have two consumers claim the same event simultaneously and confirm that the unique constraint blocks concurrent writes.
- Force an interruption after an external operation succeeds and verify that its receipt is reused during recovery.
- Simulate rate limiting, timeouts, and permanent errors, then inspect the retry budget and dead-letter records.
Logs should connect the event ID, idempotency key, task ID, and external receipt. If the issue is still difficult to locate, continue with the AI Agent end-to-end tracing checklist.
Frequently asked questions
Will a webhook still be retried after returning 200?
It depends on the sender’s protocol. Some platforms may continue retrying because of an interrupted connection, a noncompliant response body, or internal delivery policies, so consumers must still implement idempotency.
How long should the deduplication window be?
It should cover at least the maximum retry period declared by the sender. Events involving orders, payments, or tickets usually need to be retained for longer to meet business traceability requirements.
Is Redis SETNX sufficient?
It is suitable for claiming processing rights, but you must also account for expiration times, task crashes, result persistence, and multistep external operations. Critical business processes usually require a database unique constraint as a safeguard.
Can duplicate events simply be discarded?
For successfully processed events, you can return the previous result. Events that are still being processed or have an unknown status should not be discarded blindly; query their status or send them to a verification queue.
Turn duplicate callbacks into a verifiable normal condition
Webhook reliability comes from “allow duplicates, but apply the result only once.” First establish a stable idempotency key, then separate receipt from execution and save a receipt for every external action. SmaugBrain can organize deduplication, background execution, failure recovery, and acceptance steps into a sustainably operating Agent workflow. Visit SmaugBrain and begin with a trial run of one low-risk callback.