Deduplicate

Analytics Flow Control

Synopsis

Drops duplicate events cluster-wide within a configurable time window by hashing a set of field values into a key and writing it atomically to a shared NATS JetStream KV bucket. The first event for a given key passes through; every subsequent event within the window is either dropped or tagged with _vmetric.dedup_seen: true.

Schema

- deduplicate:
    fields:
      - <ident>
    bucket: <string>
    ttl: <integer>
    method: <string>
    drop: <boolean>
    tag: <string>
    description: <text>
    if: <script>
    disabled: <boolean>
    ignore_failure: <boolean>
    ignore_missing: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>

Configuration

Field	Required	Default	Description
fields	Y		Ordered list of log entry field paths used to compute the dedup key. Field order is significant — [src, dst] and [dst, src] produce different keys for the same values.
bucket	N	"default"	NATS JetStream KV bucket name. Use a distinct name per dedup scope to prevent unrelated pipelines from sharing key space.
ttl	N		Dedup window in seconds. Keys expire after this duration; a new event with the same key then passes through and starts a new window. 0 or omitted creates keys with no expiry. Set at bucket creation — changing ttl only takes effect after deleting and recreating the bucket.
method	N	xxhash	Hash algorithm for key derivation. Accepted values: xxhash (default, fastest, 16-char hex), sha-1 (40-char hex), sha-256 (64-char hex). Use sha-1 or sha-256 for compliance scenarios requiring cryptographic collision resistance.
drop	N	true	When true, duplicate events are dropped. When false, duplicates pass through with _vmetric.dedup_seen: true set, allowing downstream processors to branch on the duplicate flag.
tag	N		Identifier for this processor instance.
description	N		Explanatory note.
if	N		Condition that must be true for the processor to run.
disabled	N	false	When true, the processor is skipped.
ignore_failure	N	false	Continue pipeline processing if the processor encounters an error (including NATS connectivity failures). Use for fail-open behavior during network partitions.
ignore_missing	N	false	When true, missing fields are incorporated into the hash using a marker prefix rather than causing a failure. Two events both missing the same field share a key; events missing different fields do not collide.
on_failure	N		Processors to run when this processor fails.
on_success	N		Processors to run when this processor passes an event through.

Details

The deduplicate processor uses NATS JetStream KV Create semantics for atomic first-occurrence detection. Each event's key is computed from the values of the configured fields in declaration order. kv.Create succeeds only when the key does not already exist in the bucket — giving "first occurrence wins" semantics with a single round-trip per event and no client-side coordination. Concurrent events for the same key on different nodes race to create the entry; the loser gets ErrKeyExists and is treated as a duplicate.

Output fields on passing events: _vmetric.dedup_key is set to the computed hash string on the first occurrence, making the key visible to downstream processors.

Output fields on duplicates with drop: false: Both _vmetric.dedup_key and _vmetric.dedup_seen: true are set on the log entry before it continues downstream.

Bucket lifecycle: The KV bucket is auto-created on first reference using the configured ttl. Changing ttl in the processor configuration after the bucket exists has no effect; delete the bucket with nats kv del <bucket> to apply the new TTL. All pipelines referencing the same bucket name share the same key space and the same TTL.

Field order in fields is significant: [src, dst] and [dst, src] produce different keys for the same field values. Field boundaries are demarcated internally so fields: [ab, c] and fields: [a, bc] cannot collide.

Network latency: Each event requires one NATS round-trip (~0.1–1 ms on localhost). A network partition surfaces as a processor error. Set ignore_failure: true for fail-open behavior where events pass through when the KV store is unreachable.

Relationship to suppress: The suppress processor provides equivalent in-process per-pipeline deduplication with no NATS round-trip. Use suppress for single-node or single-pipeline dedup; use deduplicate when dedup state must be shared across multiple pipelines or multiple Director nodes.

Examples

Basic Deduplication

Dropping repeated network events that share the same source, destination, and action...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert" } - deduplicate: fields: - src - dst - action ttl: 300
First occurrence passes with _vmetric.dedup_key set; identical events within 300 seconds are dropped...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert", "_vmetric": { "dedup_key": "a3f8c1d2e4b5f6a7" } }

Isolated Bucket per Pipeline Scope

Using a named bucket to keep dedup state isolated from other pipelines...	- deduplicate: fields: - host - event.id bucket: firewall-dedup ttl: 600
Only events processed by pipelines referencing firewall-dedup share this key space; other pipelines using the default bucket are unaffected...

Tag Duplicates Instead of Dropping

Passing duplicate events downstream tagged for conditional branching instead of dropping them...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert" } - deduplicate: fields: - src - dst - action ttl: 60 drop: false
Duplicate events pass through with _vmetric.dedup_seen: true set; a downstream if processor can branch on this flag...	{ "src": "10.0.0.1", "dst": "10.0.0.2", "action": "alert", "_vmetric": { "dedup_key": "a3f8c1d2e4b5f6a7", "dedup_seen": true } }

SHA-256 for Compliance Scenarios

Using SHA-256 for cryptographic collision resistance in high-cardinality key spaces...	- deduplicate: fields: - transaction.id - user.id - event.code bucket: compliance-dedup ttl: 3600 method: sha-256
Keys are 64-character hex strings derived via SHA-256; collision probability is negligible even across billions of distinct events...

Fail-Open on NATS Unavailability

Continuing pipeline processing when the NATS JetStream store is unreachable...	- deduplicate: fields: - src - dst - action ttl: 300 ignore_failure: true
When the KV bucket cannot be reached, the processor error is suppressed and the event passes through — dedup is best-effort rather than blocking...