Retries, backoff and ordering
Your endpoint will fail sometimes. A deploy might restart the process, a database might go down for a bit, or a bug might break your handler. Retries are there so a short problem doesn't make you lose an event.
Retries don't guarantee order, though. In fact, a failed event can end up arriving after newer events. That's the part to watch out for.
What counts as a failure
Any 2xx response means the delivery succeeded. Everything else is a failure
and will be retried.
| Outcome | Result |
|---|---|
2xx | Success |
3xx | Failure. Redirects aren't followed |
4xx | Failure |
5xx | Failure |
| No response within 10 seconds | Failure |
| Connection refused, DNS failure, TLS error | Failure |
We don't follow redirects because your endpoint URL is the address you gave us.
Following a 302 could send a signed payload somewhere you didn't register.
If you need to move your endpoint, update the URL in Cockpit.
A 4xx is retried just like a 5xx. If your handler returns 400 because it
doesn't understand an event, you'll get that same event seven more times.
The retry schedule
We make up to eight attempts. The wait between attempts gets longer, and the actual wait is randomized.
| After attempt | Next attempt lands within |
|---|---|
| 1 | 30 seconds |
| 2 | 50 seconds |
| 3 | 90 seconds |
| 4 | 3 minutes |
| 5 | 6 minutes |
| 6 | 11 minutes |
| 7 | 22 minutes |
| 8 | Never. The delivery is marked failed |
All eight attempts happen within 45 minutes of the first attempt.
The random delay is important. Imagine 1,000 deliveries fail at the same time. If they all retried at exactly the same time, they'd all hit your endpoint again together. Spreading them out avoids another traffic spike.
The times above are limits, not an exact schedule. A retry might arrive quickly or much later in the window. Don't rely on a specific delay or try to guess the attempt number from the delay.
Why events can arrive out of order
New events are ready to be delivered immediately. Retries have to wait for their backoff period.
For example, a card might be created and then frozen a second later:
10:00:00 card.created attempt 1 → 500 from your endpoint
10:00:01 card.frozen attempt 1 → 200
10:00:18 card.created attempt 2 → 200
Your handler sees card.frozen before card.created.
There are also four deliveries running at the same time, so even events that don't fail can arrive in a different order.
If two endpoints listen for the same events, they can also see those events in different orders.
Handling out-of-order events
Don't use arrival order to decide which event is newer.
Use created_at from the event instead. It tells you when the event happened,
not when it reached your endpoint.
Before applying an event, compare its created_at with the timestamp you
already have for that object. If the event is older, ignore it.
For example:
UPDATE cards
SET status = $1, event_at = $2
WHERE id = $3
AND event_at < $2;
If the event is older, the update changes nothing. Your handler should still
return 200.
If you need the object's current state rather than the state at the time the event was created, fetch the object from the API. The event can tell you that something changed without being the final source of truth.
When an endpoint keeps failing
After 20 consecutive failed deliveries, the endpoint is disabled automatically. This prevents one broken endpoint from using delivery capacity for everyone else.
A successful delivery resets the counter, so the failures have to happen 20 times in a row.
While an endpoint is disabled, new events don't create deliveries for it.
This is important:
Re-enabling an endpoint does not back-fill the time it was disabled. Events still exist in the event log, but no deliveries were created for them, so there's nothing to replay from the delivery list.
If an endpoint was disabled, treat the missing period as a reconciliation job. List the affected objects through the API and compare them with your own data.
Don't wait for those events to show up after re-enabling the endpoint.
Cockpit shows endpoint health over the last 24 hours, so you can spot problems before they become customer-facing.
Reading the delivery log
Every delivery attempt is recorded.
In Cockpit's webhook delivery list, you can see:
- event id and type
- attempt number
- HTTP status
- first 8 KB of the response body
- request duration
- any error that occurred
The response body is there to help you debug. We don't parse it or use it to make delivery decisions.
Common mistakes
| Mistake | What happens |
|---|---|
Returning 400 for an event you don't recognise | It gets retried 7 more times and counts toward the 20 failures that disable your endpoint. Return 200 and ignore it |
| Doing the work before responding | Slow dependencies can cause timeouts and retries. Acknowledge the event quickly |
Assuming card.created always arrives before card.activated | It usually does, but it can arrive the other way around |
| Treating a long gap as "this was attempt 6" | Retry delays are randomized, so the gap doesn't tell you the attempt number |
| Waiting for events after re-enabling an endpoint | Nothing was queued while the endpoint was disabled. Reconcile through the API |
Next
- Idempotent event handling — how to safely handle the same event more than once.
- Overview and delivery guarantees — what's guaranteed and what isn't.