Conditional Running

Every processor accepts an optional if field: a condition string that determines whether the processor runs at all. The same mechanism gates pipeline-level branching — a pipeline: processor with its own if skips the entire nested pipeline when the condition is false — and any processor placed inside an on_success or on_failure handler can carry its own if too, making recovery logic conditional on the same terms as ordinary processing.

Conditions are written in DataStream's native expression engine, not a third-party expression library. Each unique condition string is compiled exactly once and the compiled program is cached for the life of the process; every later event reuses that cached program, and steady-state evaluation performs zero heap allocations. Writing an if on every processor that needs one carries effectively no cost by itself — whatever cost exists comes from what the condition looks up or computes, not from the engine.

note

The cache is keyed on the exact condition string: two conditions that differ only in whitespace or clause order are distinct cache entries and each compiles separately, while a byte-for-byte identical string reused across many processors compiles only once.

A single line already shows the shape of the language:

- pipeline:
    name: cef_to_csl
    if: '_vmetric?.syslog_format == "CEF"'

This reads a value a prior syslog processor already detected, and only invokes the cef_to_csl sub-pipeline when the source format was identified as CEF. Every event that isn't CEF-formatted skips the sub-pipeline entirely, with no error and no side effect.

The same expression engine, compiled in the same mode, also evaluates route-level if filters and the condition field inside a case processor's cases list, so the field-addressing rules, case sensitivity, and language reference documented below apply equally in all three places. Error handling is the exception: a route-level if filter and a case processor's condition do not treat a compile or evaluation failure as a processor failure the way an ordinary if does — see Condition Results and Errors below for how each context actually handles a broken condition.

Accessing Fields

A condition operates on the event currently flowing through the pipeline. A handful of name forms can appear in a condition, and mixing them up is the most common reason a condition silently never matches.

Event Fields

A bare identifier is a top-level field on the event, and a dotted path walks into nested objects the same way a field: parameter does elsewhere in a pipeline:

- set:
    if: network.direction == "inbound"
    field: traffic.direction
    value: "ingress"

Field names are matched exactly as they appear on the event — case-sensitively. network.direction and Network.Direction are different fields to the engine; if the event actually holds Network.Direction, the condition above compares against a field that doesn't exist and the processor never runs. This is easy to trip over when the same logical field is NetworkDirection in an ASIM-normalized event but network.direction in an ECS one — write the condition using the casing the field actually has at that point in the pipeline, not the casing it has in a different format.

System Metadata: `_vmetric`

_vmetric addresses DataStream's own metadata namespace, which travels alongside the event rather than inside it. Parser results, routing configuration flags, and per-tenant state all live here:

- pipeline:
    name: normalize_csl
    if: _vmetric?.syslog_format == "CEF"

- reroute:
    if: _vmetric?.conf?.use_azblob == true
    destination: archive_blob

_vmetric is reachable as a plain top-level identifier — it is not nested inside $ or $env (see below).

Virtual Tables: `$table`

When a pipeline has opted into virtual tables, a $ followed by a table name addresses that table's typed columns directly, instead of the untyped event map — see Virtual Tables for the full lifecycle:

- rename:
    if: $syslog.TimeGenerated == nil
    field: "@timestamp"
    target_field: "$syslog.TimeGenerated"
    ignore_missing: true

Table names and their column names are matched case-insensitively — $syslog.TimeGenerated and $Syslog.timegenerated address the same value. A condition can reference a virtual table whether or not use_table has already run in this pipeline; the reference is presence-aware and simply evaluates to nil if the table hasn't been selected or populated yet.

The Event Root: `$` and `$env`

$ alone is the event's root object. It is mainly useful with bracket indexing, for two cases a bare dotted path can't express: a key that isn't a valid identifier, and a key that needs to be looked up as a literal string:

- set:
    if: $env["@timestamp"] != nil
    field: has_native_timestamp
    value: true

- drop:
    if: $["field-with-dash"] == "unwanted"

$env is a legacy-compatible alias for the same root object — $env["key"] and $["key"] read the identical top-level value. Either form works; new conditions typically reach for $ since it's shorter.

Legacy `ctx` Alias

ctx is also accepted as a member-scoped alias for the event root — ctx.field, ctx?.field, and ctx.containsKey("field") all resolve against the same root object a bare field reference would. A bare ctx on its own, or ctx["field"] without a leading dot, is not special in this way — it is looked up as an ordinary top-level event field literally named ctx. Prefer bare field names in new conditions; ctx. exists for pipelines written against earlier engine versions.

Migration from service.

Earlier releases exposed the event under a service. prefix (service.network.direction). That prefix no longer exists. Conditions address event fields directly by their bare name (network.direction) — rewrite any surviving service.-prefixed condition by dropping the prefix.

Handling Missing Data

Conditions check for missing data constantly, so the engine makes navigating into a missing field safe by default.

By default, every pipeline condition compiles with nil-safe navigation. Dotted-path and bracket access on a field that is missing, or on a value that turns out not to be a container, evaluates to nil instead of raising an error:

- set:
    if: properties.requestUri != nil
    field: request_path
    value: "{{{properties.requestUri}}}"

If properties doesn't exist on the event at all, properties.requestUri evaluates to nil — the comparison against nil is well-defined and the condition evaluates to false. There's no error, and no need to guard properties separately before reaching into it.

Optional Chaining

The ?. operator still works, and behaves identically to a plain . under the default navigation mode:

- set:
    if: properties?.requestUri != nil && properties.requestUri != ""
    field: request_path
    value: "{{{properties.requestUri}}}"

Note the mix in that example: ?. on the first hop, then a bare . once the preceding && clause has already established that properties isn't nil. Both spellings resolve a missing chain to nil under the default mode, so from here on ?. is a matter of style, not correctness — most examples on this page write it purely as a visual cue that a hop might be absent.

Strict Navigation Opt-Out

An operator can set the environment variable VMETRIC_EVAL_STRICT_NAV=true on the process to restore strict navigation. Under strict navigation, a bare . on a nil or missing value becomes a runtime error, and ?. is required again to navigate safely. This is a process-wide setting read once at startup — an individual pipeline or condition cannot opt into it independently. A condition written with ?. throughout works correctly in either mode.

`nil`, Not `null`

The engine treats null as an alias for nil — but not by parsing it as a separate literal. Before a condition is even parsed, every literal occurrence of the four characters null in its source text is replaced with nil, including occurrences inside a quoted string. A condition written as:

if: status == "null"

does not compare status against the three-letter string null — it is silently rewritten to status == "nil" before compiling, so it can never match a field that legitimately holds the string null, and it also can't match an actual nil (a string is never equal to nil).

caution

Never write the word null in a condition, including inside a string literal, and never compare a field against a string containing the substring null. The rewrite described above applies to the entire condition source, silently, before parsing. Always write nil.

Existence Checks

Because nil-safe navigation resolves a missing chain to nil rather than erroring, the standard existence check is a direct comparison:

- set:
    if: _vmetric?.aikido?.type != nil && _vmetric.aikido.type != ""
    field: event_category
    value: authentication

This is also why most real conditions pair a != nil guard with the check it protects, joined by && — the guard runs first, and short-circuit evaluation means the protected check never runs against a value it can't handle.

`containsKey`

containsKey checks a map for a key whose value is present and non-nil — a key explicitly set to nil reports false, the same as a key that was never set at all:

- set:
    if: containsKey("network")
    field: has_network
    value: true

Called with one string argument, containsKey checks the event root; called as a method on a nested value it checks that value instead — network.containsKey("direction"). It also accepts two free arguments to check an arbitrary map, containsKey(network, "direction"). A containsKey call against something that isn't a map at all — a string, a number, nil — returns false rather than erroring.

Condition Results and Errors

A condition must evaluate to a boolean. There is no implicit "truthiness" coercion: a condition that evaluates to a string, a number, or nil is a strict-result error, not an automatic pass or fail.

The Three Outcomes

Outcome	What happens
Evaluates to `true`	The processor runs
Evaluates to `false`	The processor is skipped silently — no error, no log entry mutation
Fails to compile, or errors during evaluation	The processor fails

An erroring if does not get caught by that same processor's own ignore_failure or on_failure — the condition is checked before those are consulted, so a broken condition fails the step outright regardless of what that processor's own failure handling says. The error instead propagates up and is handled by the enclosing pipeline's on_failure chain, or by an enclosing pipeline processor's ignore_failure/on_failure, exactly as described in Handling Failures. So put failure handling for a condition typo at the pipeline level, not on the individual processor whose if might be wrong.

Route filters and case conditions handle errors differently

The table above describes an ordinary processor's if (including a pipeline: processor's own if). Two other places compile the exact same condition language but do not surface an error the same way:

A route-level if filter resolves a compile failure, an evaluation failure, or a non-boolean result to false, silently. The record is simply treated as not matching that route — there is no distinguishable error and no on_failure/ignore_failure chain to catch anything.
A condition inside a case processor's cases list treats an error the same way: the case is skipped as though it didn't match, and evaluation continues with the next case (or falls through to default), rather than failing the case processor itself.

Because neither of these ever produces a failure, a typo in a route if or a case condition will not raise an error anywhere — it will just silently never match. Test these conditions carefully; there's no failure-handling safety net to catch a mistake.

Non-Boolean Results

A bare field reference is not itself a condition unless the field genuinely holds a boolean value:

# Wrong — some_field usually isn't a boolean; this errors unless it happens to hold true or false
- drop:
    if: some_field

# Right — check for existence explicitly
- drop:
    if: some_field != nil

The same strictness applies to &&, ||, and !: every operand they touch must already be a boolean, or the condition errors at evaluation time.

# Wrong — DvcAction is a string; && requires a boolean on both sides
- set:
    if: DvcAction && DvcAction == "block"
    field: action_kind
    value: "blocking"

# Right — guard first, so the comparison only runs once DvcAction is known not to be nil
- set:
    if: DvcAction != nil && DvcAction == "block"
    field: action_kind
    value: "blocking"

Case Sensitivity

Case sensitivity is not uniform across the language — it depends on what kind of name is being matched:

Name kind	Case sensitivity	Example
Event field names	Sensitive	`network.direction` and `Network.Direction` are different fields
Word operators and keywords (`and` `or` `not` `in` `contains` `startsWith` `endsWith` `matches` `instanceof` `true` `false` `nil` `null`)	Sensitive — canonical lowercase spelling only	`and` is the operator; `AND` is a plain field reference
Builtin function names	Insensitive	`lower(x)`, `LOWER(x)`, and `Lower(x)` are equivalent
`instanceof` type names	Insensitive	`x instanceof string` and `x instanceof STRING` are equivalent
Virtual-table names and column names	Insensitive	`$syslog.TimeGenerated` and `$Syslog.timegenerated` are equivalent
Root-alias keywords (`$env`, `$`, `ctx`)	Sensitive — lowercase only	`$env` must be lowercase; `$ENV` is treated as an ordinary field, not the event root

The keyword row is the one that catches people migrating conditions by hand: an uppercase or mixed-case spelling of an operator is not recognized as an operator at all — it's parsed as an ordinary event field name.

# AND is a plain identifier here, not the logical operator — this is a compile error
if: DvcAction == "block" AND EventResult == "Success"

# Correct — lowercase and
if: DvcAction == "block" and EventResult == "Success"

The same applies to camelCase spellings: startsWith/endsWith must keep their exact casing — s startswith 'x' (lowercase w) is a compile error, not a lenient match.

Engine Modes

The underlying engine also implements a second, relaxed compatibility mode with case-insensitive field lookup, lenient truthiness (a present field is enough to pass, without an explicit boolean), and string/number equality bridging. That mode exists inside the engine, but no user-facing condition in DataStream runs in it today: every processor if, every pipeline-level if, every route-level if filter, and every case processor condition compiles in the same elastic-compatibility mode documented on this page. Treat the relaxed mode as an internal engine capability rather than something a pipeline can select.

VMETRIC_EVAL_STRICT_NAV=true, described above under Nil-Safe Navigation, is the only condition-compilation behavior that an operator can currently switch — and it only affects navigation strictness, not case sensitivity, truthiness, or type bridging.

Language Reference

Operators

From loosest-binding to tightest:

Precedence	Operator(s)	Notes
1 (loosest)	`?:` ternary	Condition must be boolean; branches evaluate lazily
2	`??` nil-coalesce	Returns the left value unless it is `nil`, then the right; never errors
3	logical OR (`or` word or symbol form)	Short-circuiting; both operands must be boolean
4	`&&` / `and`	Logical AND, short-circuiting; both operands must be boolean
5	`!` / `not` (prefix)	Operand must be boolean; sits above comparisons, so `!Message contains "denied"` parses as `!(Message contains "denied")`
6	`==` `!=` `<` `<=` `>` `>=`, `in`, `not in`, `instanceof`, `=~`, `contains`, `startsWith`, `endsWith`, `matches`	Comparison tier, detailed below — non-associative, cannot be chained without parentheses
7	`+` `-` (binary)	Numeric addition/subtraction; `+` also concatenates two strings
8	`*` `/` `%`	Multiplication, division, modulo
9	`-` (unary)	Numeric negation
10	`.name` `?.name` `[k]` `?[k]` `[a:b]` `?[a:b]`, method-call sugar	Field navigation, indexing, slicing, and method calls
11 (tightest)	literals, identifiers, `$`, `(...)`, `[...]`	Atoms

In real pipelines, the symbol spellings of OR and AND are far more common than the or/and words — both spellings are exactly the same operator, chosen purely by style.

A few operators need more than a one-line note:

== / != never error on a type mismatch. nil == nil is true; nil compared to anything else is false; numbers compare across the int/float tower; strings compare byte-for-byte; any other mismatched pair (a string against a number, for instance) is simply false. There is no string/number bridging — see Membership and Lists below for the one place bridging does happen.
< <= > >= use the same numeric tower for numbers and byte-order for strings, but unlike ==/!= they do error on a bad pairing: comparing against nil, or a string against a number, is a runtime error rather than false. Guard with != nil first: field != nil && field > 5.
+ - * error on any non-numeric operand (+ additionally accepts two strings, which it concatenates). / requires numeric operands too, but unlike the others never errors on division by zero — it produces +Inf, -Inf, or NaN instead. % requires two integers and errors on division by zero.

warning

Comparison, membership, and matching operators are non-associative — they cannot be chained. a < b < c is a compile error, not a range check. Use parentheses and &&: a < b && b < c.

Literals and Reserved Words

Strings use single or double quotes with identical semantics: 'value' and "value" are the same string. Supported escapes are backslash, single quote, double quote, forward slash, \n, \r, \t, and \uXXXX (four hex digits).
Numbers are decimal integers, floats (a digit must follow the decimal point), scientific notation (1e5, 2.5E-3), or 0x-prefixed hexadecimal integers. A leading - is always unary negation, never part of the literal itself.
Booleans are true and false.
nil and null are the same literal — see nil, Not null above for why nil is always the right one to type.
Arrays are written [e1, e2, ...], may mix types freely, tolerate a trailing comma, and [] is a valid empty array.
Reserved words — and or not in contains startsWith endsWith matches instanceof true false nil null — cannot be used as a bare field reference. A field genuinely named one of these (unusual, but possible after a vendor-specific normalization step) is still reachable through bracket access: $["contains"]. Reserved words are unrestricted after a dot or as a free function name — x.contains(y) and contains(a, b) are both fine.

Ternary and Coalesce

The grammar supports a ternary ?: and a nil-coalesce ??, though they show up far less often than a guarded && chain.

if: '(EventResult == "Success") ? true : false'

Nil-coalesce is more commonly reached for, chiefly to supply a fallback before a further check runs:

if: (_temp.event?.type ?? "") != ""

?? never errors on its own — it only inspects whether the left side is nil. Most conditions express the same fallback-and-check idea with an explicit != nil guard instead of ??; either is valid, and ?? is worth reaching for when the fallback value itself needs to flow into a further expression, such as membership sugar (see one(...) under Membership and Lists).

`instanceof`

instanceof resolves the type name at compile time, matched case-insensitively:

Type name	Matches
`string`	String values
`boolean`	Boolean values
`integer`	All numeric values — both integers and floats
`list`, `array`	Only a concrete string array; a generic array produced by parsing JSON does not match
`map`	Only a string-keyed map
`function`	Function values only
Anything else (`float`, `number`, `nil`, an unrecognized name)	Never matches — always `false`

- set:
    if: SrcUserId instanceof string
    field: user_id_kind
    value: "string"

note

Because list/array only match a concrete string array, instanceof is rarely the right tool for "is this field a JSON array" checks on parsed event data. Use type(x) == "array" instead — type() reports array for any slice value, not only a []string.

Regex: `=~` and `matches`

Two ways to match a field against a regular expression, both using RE2 syntax:

x =~ /pattern/ requires a slash-delimited regex literal — the pattern is lexed as a regex only in this position, immediately after =~, and it compiles once, at compile time. Writing x =~ "pattern" with a quoted string instead of a /pattern/ literal is a compile error.
x matches "pattern" is the word-operator form. It accepts either a literal string pattern or one built at runtime; a dynamic pattern is compiled once and cached process-wide the first time it's seen, not recompiled per event.

- set:
    if: network?.direction != nil && network?.direction =~ /^(in|out)$/
    field: direction_kind
    value: "binary"

- set:
    if: message != nil && message matches "^[0-9]{4}-[0-9]{2}-[0-9]{2}T"
    field: has_iso_timestamp
    value: true

The two forms differ slightly on the left-hand side: =~ treats a nil subject as false and stringifies a non-string subject before matching, while matches also treats nil as false but raises a runtime error for any other non-string subject. Guard non-string fields with a nil check either way, and reach for =~ when the field's type isn't guaranteed to already be a string.

String Indexing and Slicing

Strings and arrays support bracket indexing and range slicing:

if: message[0:1] == "<"

x[i] on a string returns the byte at that position as an integer, not a one-character string — indexing into a string is rarely useful directly in a condition; slicing (x[0:1]) is the normal way to test a leading character.
x[a:b] returns a substring for a string receiver or a sub-array for an array receiver; an omitted bound defaults to 0 or the receiver's length, and bounds are clamped rather than erroring on an out-of-range value.
Unlike ordinary field navigation, slicing is not nil-safe by default: field[0:4] on a missing field is a runtime error even though field.sub on the same missing field would quietly resolve to nil. Use field?[0:4] for a nil-safe slice, or guard with field != nil && first.

Arithmetic

if: source.bytes != nil && destination.bytes != nil && (source.bytes + destination.bytes) > 1000000

Arithmetic operators show up far less often inside conditions than comparisons do — most real conditions compare an existing numeric field, or a len() result, directly against a threshold rather than computing a fresh value inline. When a condition does need arithmetic: +/-/* require numeric operands (+ also concatenates two strings), / always returns a float and never errors on division by zero, and % requires two integers and errors on division by zero.

Method-Call Sugar

Every builtin function can also be called as a method on its first argument: x.f(a) is exactly the free-function call f(x, a). This is why message.toLowerCase().contains('disconnected') and contains(lower(message), 'disconnected') are the same condition.

The method form adds one more thing the free-function form doesn't have: x?.f(a) returns nil without calling f at all when x is nil. Because lower/upper/len/int/float/split all error on a nil receiver (see Functions below), calling them through ?. is the concise way to make a chain nil-tolerant instead of writing a separate guard for every hop:

if: vendor?.requestLine?.toLowerCase()?.contains(' ') == true

If vendor or requestLine is missing, the ?. before toLowerCase() short-circuits the whole chain to nil without ever calling a function that would otherwise error on a nil input. The trailing == true matters here too: the chain can legitimately resolve to nil, and a condition must resolve to a definite boolean either way.

Functions

Function	Behavior	Nil / type behavior
`len(x)`	Byte length of a string; element count of an array or map	Errors on `nil` or any other type
`lower(x)`, `toLowerCase(x)`	Lowercases a string	Errors on `nil` or a non-string
`upper(x)`, `toUpperCase(x)`	Uppercases a string	Errors on `nil` or a non-string
`int(x)`	Converts to an integer: identity on an int, truncates a float, parses a base-10 string	Errors on `nil`, an unparsable string, or any other type
`float(x)`	Converts to a float: widens an int, identity on a float, parses a string	Errors on `nil`, an unparsable string, or any other type
`string(x)`	Renders any value as text	Never errors; `string(nil)` returns the text `<nil>`
`type(x)`	Returns the value's type name — `nil`, `bool`, `int`, `uint`, `float`, `string`, `array`, `map`, `func`, or a qualified name such as `time.Time`	Never errors
`trim(x)`	Trims leading/trailing whitespace from the stringified argument	Never errors; `trim(nil)` returns `<nil>`
`split(s, sep)`	Splits a string into an array on a separator	Both arguments must be strings, or it errors
`abs(x)`	Absolute value, preserving int or float type	Errors on a non-numeric argument
`min(...)`, `max(...)`	Minimum/maximum across any number of numeric arguments	Errors if any argument is non-numeric
`contains(a, b)`	Same as the `contains` word operator	`nil` on either side returns `false`; a non-string, non-nil pairing errors
`startsWith(a, b)`, `endsWith(a, b)`	Same as the word operators	`nil` on either side returns `false`; non-string, non-nil errors
`matches(s, pattern)`	Regex match, RE2 syntax	`nil` subject returns `false`; non-string, non-nil subject errors
`containsKey(...)`	Key-present-and-non-nil check (see Handling Missing Data)	Returns `false` for a non-map receiver, never errors
`Before(x, y)`, `After(x, y)`	Chronological comparison of two already-parsed timestamp values	Errors unless both arguments are timestamps
`InstanceOf(v, "type")`	Function form of `instanceof`, with the type name passed as a string	Same type-name table as `instanceof`
`MatchesWith(v, pattern)`	Function form behind `=~` / `matches`	`nil` subject returns `false`; non-string pattern errors; non-string subject is stringified
`ArrayCheck(v, arr)`	Fold-and-stringify membership check — the mechanism behind the `one(...)` legacy sugar	A scalar `arr` is treated as a single-element haystack; `v` itself may be an array, matching if any of its elements matches

caution

to_string(x) is accepted by the compiler for backward compatibility but always fails when actually evaluated. Use string(x) instead.

A few of these in context:

# upper() — normalize casing before comparing against a known constant
- set:
    if: complianceType != nil && upper(complianceType) == "NON_COMPLIANT"
    field: alert
    value: true

# len() — reject malformed CSV/VPC flow records by field count
- drop:
    if: _vpcflow_tokens != nil && len(_vpcflow_tokens) != 14

# trim() — never errors, so it's safe to call without a nil guard first
- set:
    if: trim(HostName) != ""
    field: has_hostname
    value: true

# split() + len() — flag deeply nested request paths
- set:
    if: url.path != nil && len(split(url.path, "/")) > 3
    field: deep_path
    value: true

# string() — deliberately bridge int/string since the engine won't do it implicitly
- set:
    if: HttpStatusCode != nil && string(HttpStatusCode) == "200"
    field: is_ok
    value: true

# int() — parse a raw string field extracted by an earlier processor
- set:
    if: _raw_status != nil && int(_raw_status) >= 500
    field: is_server_error
    value: true

# abs() / max() — flag highly asymmetric byte counters
- set:
    if: SrcBytes != nil && DstBytes != nil && abs(SrcBytes - DstBytes) > 10485760
    field: asymmetric_transfer
    value: true

# Before() / After() — assumes EventStartTime/EventEndTime were already parsed by a date processor
- set:
    if: EventEndTime != nil && EventStartTime != nil && EventEndTime.After(EventStartTime)
    field: duration_valid
    value: true

Membership and Lists

in and .contains() behave differently depending on whether the right-hand side is a literal array written inline or a field holding an array — this is the single most important asymmetry to know about in the whole language.

Literal Arrays

Membership against a literal array is case-insensitive and stringifying: both sides are converted to strings and folded to the same case before comparing.

- set:
    if: _vmetric?.event?.definition_id in [24, 25]
    field: event_category
    value: "logon"

- set:
    if: EventResult != nil && ['Success', 'Successful'].contains(EventResult)
    field: outcome
    value: "success"

Because this form stringifies and folds case, a numeric 24 matches the string "24", and "Success" matches "success" or "SUCCESS" without any explicit lower()/upper() call:

if: 24 in ["24", "25"]      # true — numeric 24 matches string "24"
if: '"SUCCESS" in ["success"]' # true — case-folded before comparing

note

A literal in-list also reads more compactly and, once it grows past about five values, evaluates faster than the equivalent || chain — prefer _vmetric?.event?.definition_id in [22, 23, 24, 25] over x == 22 || x == 23 || x == 24 || x == 25.

Dynamic Arrays

Membership against a field that holds an array uses ordinary typed, case-sensitive equality instead — no stringifying, no case-folding.

if: >-
  _vmetric?.conf?.allowed_asim_tables == nil ||
  "ASimDnsActivityLogs" in _vmetric?.conf?.allowed_asim_tables

Here the table name must match an entry in _vmetric?.conf?.allowed_asim_tables by exact, case-sensitive equality — and the == nil clause in front is a short-circuiting "no allow-list configured, so allow everything" guard. Guard with == nil rather than len(...): len errors on a missing field, while the == nil check is always safe.

caution

Moving a hardcoded list from a literal (x in ['A', 'B']) into a shared configuration field for reuse (x in $env["allowed_values"]) silently changes the matching rules — from case-insensitive and stringifying to case-sensitive and typed. A value that matched the literal list by case-folding or by number/string bridging can stop matching once the same list lives in a field.

Legacy Sugar: `notcontains` and `one()`

Two negated-membership spellings are accepted as legacy sugar, both rewritten to a negated literal-array membership check before compiling: [...].notcontains(x) and [...].!contains(x).

- set:
    if: "['Reject', 'Block', 'Prevent'].notcontains(DispositionAction)"
    field: allow_status
    value: true

New conditions should prefer the direct form — !['Reject', 'Block', 'Prevent'].contains(DispositionAction) — since notcontains exists purely for compatibility with pipelines written before that spelling existed.

A second legacy idiom, one(x, string(#) in [...]), is likewise accepted and rewritten automatically into the equivalent membership check:

if: one(_temp.event?.type ?? "", string(#) in ['denied', 'start', 'end'])

is equivalent to, and should be written as:

if: "['denied', 'start', 'end'].contains(_temp.event?.type ?? \"\")"

Both legacy forms keep working without modification; write new conditions with in or .contains() directly.

Scenario Cookbook

Realistic conditions grouped by intent. Every example below uses field names and processors that exist in shipped content today.

Severity-Based Filtering

Keep only higher-severity events once a normalized numeric severity is available:

- set:
    if: LogSeverityNumber >= 0 && LogSeverityNumber <= 3
    field: severity_tier
    value: "critical"

- drop:
    if: LogSeverityNumber != nil && LogSeverityNumber > 5

Vendor and Product Routing

Branch to a vendor-specific sub-pipeline once the event's vendor and product are known:

- pipeline:
    name: normalize_checkpoint
    if: EventVendor == "Check Point" && EventProduct == "Security Gateway"

Format Branching on Parser Output

Gate normalization sub-pipelines on the format an earlier syslog processor detected:

- pipeline:
    name: cef_to_csl
    if: _vmetric?.syslog_format == "CEF"

- pipeline:
    name: leef_to_csl
    if: _vmetric?.syslog_format == "LEEF"

Network Direction Filtering

Classify or drop traffic based on a normalized direction field:

- set:
    if: network?.direction != nil && network?.direction =~ /^(in|out)$/
    field: direction_kind
    value: "binary"

- drop:
    if: network.direction == "internal"

Threshold Alerts on Numeric Fields

Compare a numeric field against a fixed range, and page a notification processor on a confidence score:

- set:
    if: HttpStatusCode != nil && HttpStatusCode >= 400 && HttpStatusCode < 500
    field: EventResultDetails
    value: "Client Error"

- slack:
    if: _anomali_threatscore != nil && _anomali_threatscore >= 80
    title: "High-Confidence Threat Indicator"
    message: "Threat score exceeds alert threshold"

The slack processor's if is evaluated the same way as any other processor's — a false result simply skips the notification.

IP Prefix Checks

Filter internal traffic before it reaches egress enrichment:

- drop:
    if: SrcIpAddr != nil && SrcIpAddr startsWith "10."
    description: "Drop internal 10.0.0.0/8 traffic before egress enrichment"

Regex Extraction Guards

Only run an extraction processor once the message shape has been confirmed, avoiding wasted work on messages that can't match:

- grok:
    if: message != nil && message matches "^[0-9]{4}-[0-9]{2}-[0-9]{2}T"
    field: message
    patterns:
      - "%{TIMESTAMP_ISO8601:log_timestamp} %{GREEDYDATA:log_body}"

Timestamp Presence Guards

Only convert a raw timestamp when the normalized target field isn't already populated:

- date:
    if: TimeGenerated == nil && $env["@timestamp"] != nil
    field: "@timestamp"
    target_field: TimeGenerated
    formats:
      - ISO8601

Guarding Expensive Enrichment

Skip network lookups when the target field is already populated, or the source field to enrich from is missing:

- geoip:
    if: source.ip != nil && source.geo == nil
    field: source.ip
    target_field: source.geo

- dns_lookup:
    if: destination.ip != nil && destination.domain == nil
    type: "forward"
    field: destination.ip
    target_field: destination.domain

warning

dns_lookup's type field (forward or reverse) has no default. Leave it unset and every event for which the if evaluates true fails the processor outright, since the underlying dispatch has no fallback branch — this is not a silent skip. This differs from the geoip example above, whose database_file defaults to GeoLite2-City.mmdb when left unset. Always set type explicitly on dns_lookup.

Pipeline Branching

Send an event down one of several nested pipelines based on which fields are already present:

- pipeline:
    name: network_traffic
    if: source?.packets != nil && destination?.packets != nil

- pipeline:
    name: threat_detection
    if: threat?.indicator?.confidence > 50

Fail-Fast Validation

Raise an explicit, descriptive failure when mandatory fields are still missing after all normalization has run:

- fail:
    if: >-
      EventCount == nil || EventStartTime == nil || EventEndTime == nil ||
      EventType == nil || EventResult == nil || EventProduct == nil ||
      EventVendor == nil || EventSchema == nil || EventSchemaVersion == nil ||
      Dvc == nil
    message: "One or more mandatory fields are missing after all processing steps"

- fail:
    if: DvcDomain != nil && DvcDomainType == nil
    message: "DvcDomainType is required whenever DvcDomain is present"

Dropping Noise by List Membership

Drop known-noisy event classes and disposition values by list membership:

- drop:
    if: _vmetric?.event?.definition_id in [22]

- drop:
    if: "['Drop', 'Reject', 'Block', 'Prevent'].contains(DispositionAction)"

Virtual-Table Guards

Backfill required columns once a virtual table has been created but a column is still unset. See Virtual Tables for the full create_table/use_table lifecycle these conditions assume:

- rename:
    if: $syslog.TimeGenerated == nil
    field: "@timestamp"
    target_field: "$syslog.TimeGenerated"
    ignore_missing: true

- set:
    if: $commonsecuritylog.DeviceVendor == nil
    field: $commonsecuritylog.DeviceVendor
    value: "Unknown"

Multi-Line Conditions for Readability

Long &&/|| chains read better split across lines. YAML's | block scalar keeps line breaks (handy for one clause per line), while >- folds everything into a single line and strips the trailing newline (handy once parentheses already carry the grouping):

- reroute:
    if: |
      _vmetric?.conf?.use_smart_routing == true &&
      _vmetric?.conf?.use_object_storage == true
    destination: object_storage

- reroute:
    if: >-
      _vmetric?.table == nil && (
        network?.protocol == "dns" ||
        dns?.question?.name != nil
      )
    destination: dns_events

Both styles compile to the exact same condition text once YAML has finished parsing — pick whichever reads more clearly for the condition at hand.

Migration from the Legacy Engine

Pipelines written for the previous expression engine keep working essentially unchanged: bare field names, ?. navigation, and in/.contains() membership are all identical constructs on the current engine, which runs in the same elastic-compatibility mode for legacy parity. Three things did change:

The service. prefix has been removed — event fields are addressed directly by name (see Accessing Fields above).
?. optional chaining is no longer required to reach into a possibly-missing field. Nil-safe navigation is the default, so a bare . now behaves the way ?. used to. ?. remains valid, and many existing conditions keep using it for clarity.
one(x, string(#) in [...]), [...].notcontains(x), and [...].!contains(x) are still accepted and rewritten automatically before compiling — no existing pipeline needs to be rewritten to keep working, though new conditions should prefer the direct in/.contains() forms.

Accessing Fields​

Event Fields​

System Metadata: _vmetric​

Virtual Tables: $table​

The Event Root: $ and $env​

Legacy ctx Alias​

Handling Missing Data​

Nil-Safe Navigation​

Optional Chaining​

nil, Not null​

Existence Checks​

containsKey​

Condition Results and Errors​

The Three Outcomes​

Non-Boolean Results​

Case Sensitivity​

Engine Modes​

Language Reference​

Operators​

Literals and Reserved Words​

Ternary and Coalesce​

instanceof​

Regex: =~ and matches​

String Indexing and Slicing​

Arithmetic​

Method-Call Sugar​

Functions​

Membership and Lists​

Literal Arrays​

Dynamic Arrays​

Legacy Sugar: notcontains and one()​

Scenario Cookbook​

Severity-Based Filtering​

Vendor and Product Routing​

Format Branching on Parser Output​

Network Direction Filtering​

Threshold Alerts on Numeric Fields​

IP Prefix Checks​

Regex Extraction Guards​

Timestamp Presence Guards​

Guarding Expensive Enrichment​

Pipeline Branching​

Fail-Fast Validation​

Dropping Noise by List Membership​

Virtual-Table Guards​

Multi-Line Conditions for Readability​

Migration from the Legacy Engine​

Accessing Fields

Event Fields

System Metadata: `_vmetric`

Virtual Tables: `$table`

The Event Root: `$` and `$env`

Legacy `ctx` Alias

Handling Missing Data

Nil-Safe Navigation

Optional Chaining

`nil`, Not `null`

Existence Checks

`containsKey`

Condition Results and Errors

The Three Outcomes

Non-Boolean Results

Case Sensitivity

Engine Modes

Language Reference

Operators

Literals and Reserved Words

Ternary and Coalesce

`instanceof`

Regex: `=~` and `matches`

String Indexing and Slicing

Arithmetic

Method-Call Sugar

Functions

Membership and Lists

Literal Arrays

Dynamic Arrays

Legacy Sugar: `notcontains` and `one()`

Scenario Cookbook

Severity-Based Filtering

Vendor and Product Routing

Format Branching on Parser Output

Network Direction Filtering

Threshold Alerts on Numeric Fields

IP Prefix Checks

Regex Extraction Guards

Timestamp Presence Guards

Guarding Expensive Enrichment

Pipeline Branching

Fail-Fast Validation

Dropping Noise by List Membership

Virtual-Table Guards

Multi-Line Conditions for Readability

Migration from the Legacy Engine