How to Deduplicate Webhooks in n8n
Catch the copies before they cost you an execution.
- 18% of incoming webhooks are repeats — up to 3× on a busy day.
- Each duplicate is a paid execution plus a wasted model call.
- Dedup in front of n8n drops wasted executions to under 1%.
The problem
- Webhooks are delivered at least once — the same event hits your Webhook node two or three times.
- n8n has no memory it already ran, so it starts a fresh execution for every copy.
- On execution-based pricing, each duplicate is a paid run — and a wasted model call.
The solution
Give each webhook a fingerprint — the event ID, or a hash of sender + body — and remember it for a little while. Seen that fingerprint already? Drop it.
- Key on the event ID when the sender gives a stable one.
- No reliable ID? Hash sender + normalized body (trim whitespace, drop timestamps).
- Bound it with a window — 30–120 seconds — so tomorrow's real repeat still runs.
Where you put the check decides what you save: inside n8n saves the model call; in front saves the execution too.
Result: one event → one execution, however many copies arrive.
Your n8n workflow executed three times for one order last night, and every execution counted against your plan. The payload was identical each time. This is duplicate webhooks — the quietest way an n8n pipeline wastes executions, and in an AI workflow each duplicate is also a wasted model call. Here is why it happens and how to stop it.
Why n8n receives the same webhook more than once
Webhooks are delivered at least once by design. The platform firing the webhook — a messaging API, a payment provider, a CRM — retries delivery whenever it does not get a fast enough 200 back. n8n has no memory that it already processed the event, so it runs the Webhook node again, start to finish. On execution-based pricing that multiplication is money: one real event becomes N executions, and in an AI workflow N calls to your model. The waste never surfaces as an error, only as a usage graph that runs ahead of your actual traffic.
There are two ways to stop it: build the dedup yourself inside n8n, or put a gateway in front that does it for you. This post walks the full manual path first — so you understand exactly what’s involved — then lets you compare the two at the end.
The two sources of duplicates
Knowing the source tells you which field to key on when you deduplicate.
| Source | What causes it | Key on |
|---|---|---|
| Delivery retries | Sender re-sends on timeout | Event / delivery ID |
| Repeats & loops | Impatient user or misconfigured fan-out | Content hash |
Retries carry the same event ID, so they’re the easiest to catch. Repeats and integration loops arrive with fresh IDs but identical content, so an ID check misses them and you need a hash of the body.
The manual fix (Option 1)
n8n ships a node for exactly this — but building the key well, and knowing what the node can’t do, is on you.
Fix 1: key on the event ID
The Remove Duplicates node, in its Remove Items Processed in Previous Executions mode, keeps a store of keys it has already seen across runs and drops anything that matches. When the sender gives a stable delivery/event ID, point the node at it and retries are filtered before the rest of your workflow runs.
On a gateway, the seen-key store is built in — nothing to add after the Webhook node.
Fix 2: fall back to a content hash
No reliable ID? Hash sender + normalized body (a Code or Crypto node producing a SHA-256 digest) so repeats with fresh IDs still collapse onto the same key.
On a gateway, the fingerprint keys on sender + message content and ignores the volatile envelope for you.
Fix 3: bound it with a window
The same text today and tomorrow are two real events, not a copy. Deduplicate only within a window — 30 to 120 seconds catches most retry storms without suppressing legitimate repeat requests. Match it to how your users behave.
On a gateway, the window is just a setting you pick — not code you wire into every workflow.
The catch: the execution is already counted
Here is the gap the Remove Duplicates node can’t close on its own: it fires after the Webhook node has already triggered the run. It stops the expensive downstream steps — the AI node, the database writes — but n8n has already counted the execution toward your quota. Retry storms still burn executions. And you repeat the whole Remove Duplicates + hash + window setup in every workflow, patching it as senders change their retry behavior. For one workflow it’s fine; across a fleet it becomes its own chore.
Place a gateway layer (Option 2) — faster
Instead of rebuilding dedup inside every workflow, you point the sender at one layer that already does all of it — hashing, windows, idempotency — before anything reaches n8n.
- One layer, every workflow. Swap the URL the sender posts to for the gateway’s; it covers all your n8n flows and other channels too.
- The execution never starts. The duplicate is dropped at the edge, so it doesn’t trigger an execution at all — not just the downstream steps.
- Nothing to maintain. No hash store to host, no normalization to patch per workflow.
- Tune, don’t code. The window is a setting, not a Code node.
You trade a little control of the internals for a lot less work — and, on n8n specifically, for the execution count itself.
Option 1 (DIY) vs Option 2 (gateway)
Neither is wrong — one trades time for control, the other control for time. The honest side-by-side:
| Do it yourself | Use a gateway | |
|---|---|---|
| Delivery retries (same ID) | ✓Caught downstream | ✓Dropped at the edge |
| Repeats (fresh ID, same body) | ~Only if you hash & normalize | ✓Content keyed, envelope ignored |
| Execution quota | ✕Still counted | ✓Never triggered |
| Hash store & normalization | ✕You build & host | ✓Built in |
| Every workflow | ~Repeat setup each time | ✓One layer, all flows |
| Control of internals | ✓Full | ~Abstracted away |
Deduplication is one of several input-layer problems a gateway handles — see the complete guide to message quality for AI agents. The mechanics of hashing, windows and idempotency are the same whether you build them in n8n or offload them — broken down in how to stop duplicate messages to your AI agent.
If option 2 sounds more like you, here’s the short version of how to set it up: let a gateway dedup before n8n ever runs →.
Either way — go make your workflow run once per real event.
Thanks for reading. 👋
Orlando. -
Orlando Quijada
Technologist with 15+ years of experience. Developer, SaaS tester, and AI collaboration & optimization specialist. Currently contributing at portarus.com.
Written on
