Skip to main content

Idempotent event handling

You will sometimes receive the same event more than once. This usually isn't a problem, but it can be if your handler charges an account twice or sends the same email twice.

The fix is simple: dedupe using event.id, and make your handler safe to run more than once.

Where duplicates come from​

There are three common reasons.

Your 2xx response didn't reach us. Your handler processed the event, but the response was lost. We retry it because we don't know the event was already handled. This is the most common case.

Someone pressed replay. Cockpit can replay an old delivery. It creates a new delivery for the same event, with the same payload and id. This is useful when you're fixing a bug in your handler, so dedupe needs to use the event id, not the delivery.

Two endpoints listen for the same event. Each endpoint gets its own delivery. If both endpoints point to the same service, your service will see the event twice.

Deduplicating​

Every event has a unique id:

{
  "id": "evt_01M480J2CTN4T8D9E8D2TBN1KC",
  "type": "cardholder.created",
  "created_at": "2026-10-06T07:06:30Z"
}

Store the id before doing the work:

CREATE TABLE processed_events (
  event_id     TEXT PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
async function handle(event) {
  const { rowCount } = await db.query(
    'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING',
    [event.id],
  );

  if (rowCount === 0) return; // already handled

  await applyEvent(event);
}

The insert acts as the lock. If two copies of the same event arrive at the same time, only one insert succeeds. The other gets ON CONFLICT and stops.

Don't use these as the key:

  • The object id. A card can have card.created, card.activated, and card.frozen. They all have the same card_id, but they're different events.
  • A hash of the payload. Two different events can have the same payload. For example, two identical updates can still be two separate events.
  • The delivery id. A replay creates a new delivery for the same event. You want to treat it as a duplicate.

Insert first, or work first?​

Inserting the event id first has a downside. If the handler crashes halfway through, the event is already marked as handled. A retry will be ignored, even though the work wasn't finished.

If the work is a database change you control, put both the insert and the work in the same transaction:

await db.transaction(async (tx) => {
  const { rowCount } = await tx.query(
    'INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING',
    [event.id],
  );

  if (rowCount === 0) return;

  await applyEvent(tx, event);
});

If the handler crashes, both changes are rolled back. The retry can then run the event again.

This doesn't help with things the database can't roll back, like:

  • charging a card
  • sending an email
  • calling another API

For those, put the event on a durable queue as part of the transaction. Then let the worker handle idempotency for the external operation, using whatever idempotency key that service supports.

Make the work idempotent too​

Dedupe is useful, but don't rely on it completely. The table could be cleared, truncated during a migration, or a replay could happen much later.

Your handler should still be safe if the same event runs twice.

Usually, that means writing the state from the event instead of applying a change:

// Safe to run twice.
await db.query(
  'UPDATE cards SET status = $1 WHERE id = $2',
  [event.data.object.status, event.data.object.id],
);

// Not safe to run twice.
await db.query(
  'UPDATE accounts SET balance = balance - $1 WHERE id = $2',
  [event.data.object.amount, accountId],
);

data.object contains the full object, not just the changes. So for most events, you can just save the state it reports. Running the same event again won't change anything.

For things that really need to accumulate, store the event id with the row and add a unique constraint:

CREATE TABLE ledger_entries (
  id       BIGSERIAL PRIMARY KEY,
  event_id TEXT NOT NULL UNIQUE,
  amount   BIGINT NOT NULL
);

If the same event tries to create the row again, the unique constraint catches it.

Combining this with ordering​

Deduplication and ordering are two different problems.

A duplicate event should be dropped. An old event should also be dropped if it would overwrite newer state. That old event won't be in processed_events because you haven't seen it before.

Handle both:

async function handle(event) {
  if (await alreadyProcessed(event.id)) return; // duplicate
  if (await staleFor(event)) return;            // out of order

  await applyEvent(event);
}

The second check uses created_at from the event. See retries, backoff and ordering.

Not the same as Idempotency-Key​

The API's Idempotency-Key header is different.

That one prevents a request you send from being applied twice when you retry it.

Event deduplication prevents an event we send you from being applied twice.

They solve different problems and don't share any state. Having one doesn't give you the other.

How long to keep the ids​

Automatic retries stop after eight attempts within 45 minutes, so keeping ids for 24 hours is enough to cover those retries.

Replays are different. Someone might replay an event from last month while debugging. If you only keep ids for one day, that event will be processed again.

For most systems, it's easiest to keep the ids indefinitely. One small row per event is usually cheap. You can clean them up later if the table gets too large.

Check it works​

Pick a cardholder event in Cockpit's delivery list and replay it.

Your endpoint should return 200, and nothing in your database should change:

  • no new row
  • no updated timestamp
  • no second email

If something changes, your handler isn't fully idempotent yet. The replay should help you find where.

Common mistakes​

MistakeWhat happens
Checking for the id, then inserting it after the workTwo copies can both pass the check
Deduplicating on the object idReal events for the same object get dropped
Deduplicating per delivery instead of per eventReplay gets processed again
Relying only on dedupe with balance = balance - xClearing the dedupe table can cause a double debit
Returning 4xx for a duplicateThe event gets retried. Return 200 instead

Next​

Last updated on