Skip to main content

Virtual Tables

A pipeline entry normally carries its fields in an untyped map — flexible, but every read, write, and marshal pays for that flexibility with hashing, boxing, and reflection. Virtual tables give a pipeline branch an alternative: a fixed, strongly typed struct that a handful of processors write into explicitly, and that the entry can marshal from instead of the map. On a hot fan-out leaf — parse once, set a known set of fields, ship to several destinations — switching from map writes to a virtual table measures ~1.6x faster end to end and ~3.2x faster at the marshal step alone. Marshal allocations drop 12x (2 vs 24 per event). Timing ratios move with machine load. The allocation counts are exact and reproduce identically on any machine, because they come from removing reflection and hash-map growth, not CPU speed.

Virtual tables are strictly opt-in, and nothing about them is automatic. There is no migration from map to table in either direction. There is no "active schema" that bare field names silently start routing into once a table exists. There is no switch that quietly discards a table's contents behind the pipeline author's back. The engine does exactly what the pipeline writes, and nothing else:

  • A table exists on an entry only after a create_table processor runs on it.
  • A field lands in a table only when a processor writes it there through an explicit $<table>.Field path.
  • An entry marshals from a table only after a use_table processor selects it as the output source.

Adopting virtual tables in one branch of one pipeline never affects any other branch, any other pipeline, or the untyped map fields already in use everywhere else, because every one of those three steps is explicit.

The Model

Four rules describe the whole feature:

  1. Tables don't exist until created. create_table allocates a table's backing struct on the entry. Before that, $<table>.Field paths targeting it are the same as any other uncreated resource — reads resolve to nothing, writes fail loudly.
  2. Fields land only through explicit $<table>.Field writes. A bare field name — HostName, log.syslog.hostname — always targets the untyped map, exactly as it did before virtual tables existed, whether or not a table happens to be alive on the entry and whether or not the table has a same-named column.
  3. Output switches only through use_table. Nothing marshals from a table until a use_table processor says so. Selecting a table doesn't touch the map's contents — it only changes what the entry serializes at output.
  4. The map always coexists. Creating, writing to, and even selecting a table never clears the map. Multiple tables can also be alive on the same entry simultaneously (rare, but supported) — each addressed independently by its own $<table> prefix.

The lifecycle in one table:

StageTriggered byEffect
Createdcreate_tableBacking struct allocated on the entry; every column starts absent
Writtenset / rename / syslog with table: / etc. targeting $table.FieldValue coerced into the column's type and stored; presence bit set
Selecteduse_tableThe entry's MarshalJSON (and reroute's staged payload) switches to serializing this table
Clearedtruncate_tableEvery column value and presence bit reset; the table stays allocated for further writes
Removeddrop_tableBacking struct discarded; if it was selected, output reverts to the map

A table addressed in an if: condition or a get-style read does not need to be the selected output table — it only needs to have been created. Selection (use_table) is purely about what ships at marshal time, not about what is readable.

note

Because a leading $ in a field path is reserved for table addressing, a literal $-prefixed key in the map is reachable only through the condition engine's bracket syntax ($env["$weird_key"]), never through a plain dotted field path.

Supported Tables

Twelve tables are registered, all mirroring Microsoft Sentinel schemas — the Syslog and CommonSecurityLog ingestion tables, plus ten Advanced Security Information Model (ASIM) normalized tables. These are the same normalized shapes discussed in Normalization and monitored by Schema Drift Detection — virtual tables give you a typed, high-throughput way to produce payloads in those shapes, on top of the untyped normalization path that already exists.

TableColumns
Syslog13
CommonSecurityLog156
ASimAuditEventLogs123
ASimAuthenticationEventLogs126
ASimDhcpEventLogs95
ASimDnsActivityLogs135
ASimFileEventLogs122
ASimNetworkSessionLogs142
ASimProcessEventLogs136
ASimRegistryEventLogs81
ASimUserManagementActivityLogs108
ASimWebSessionLogs145
note

Table names and column names are both case-insensitive everywhere they're referenced — in create_table/use_table/drop_table/truncate_table's name: field, in $table.Field write paths, and in if: conditions. $Syslog.computer, $syslog.Computer, and $SYSLOG.COMPUTER all resolve to the same column. Exact spelling is matched first, with a case-folded lookup as the fallback, so pick one spelling and use it consistently for readability even though the engine doesn't require it.

Syslog is the smallest table and the one used throughout this page as the worked example. Its 13 columns:

ColumnType
Computerstring
EventTimestring
Facilitystring
HostIPstring
HostNamestring
ManagementGroupNamestring
ProcessIDint
ProcessNamestring
SeverityLevelstring
SourceSystemstring
SyslogMessagestring
TimeCollectedstring
TimeGeneratedstring

ProcessID is the only non-string column on Syslog; every other field, including the two timestamp-shaped columns EventTime and TimeGenerated, is stored as a string. CommonSecurityLog and the ASIM tables follow the same shape — a fixed set of typed columns, one storage kind per column — just with far more of them.

The Four Processors

Four processors form the entire lifecycle. All four share the same scaffold — name, description, disabled, if, ignore_failure, on_failure, on_success, tag — and none of them has an ignore_missing field, since there's no notion of a "missing" table separate from an unknown one.

Create Table

Allocates the named table's backing struct on the entry so $<table>.Field paths can address it.

- create_table:
name: <ident>
description: <text>
disabled: <boolean>
if: <script>
ignore_failure: <boolean>
on_failure: <processor[]>
on_success: <processor[]>
tag: <string>
FieldRequiredDefaultDescription
nameY-Table to allocate; one of the Supported Tables (case-insensitive); templated
descriptionN-Explanatory note
disabledNfalseDisable the processor without removing it from the pipeline
ifN-Condition to run
ignore_failureNfalseSee Handling Failures
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier
- create_table:
name: Syslog
  • Idempotent. Running it again against a table that's already allocated is a no-op — existing values and presence bits are left completely untouched.
  • Failure mode. If name isn't one of the 12 registered tables, the processor fails with unknown virtual table: "<name>". Add ignore_failure: true to treat an unrecognized name as a soft skip instead.
  • name is templated, so it can be built from the event, for example name: "{{{target_table}}}".

Use Table

Selects which table, if any, the entry marshals from at output. This is the switch — nothing an entry has written into a table reaches a target until use_table runs.

- use_table:
name: <ident>
description: <text>
disabled: <boolean>
if: <script>
ignore_failure: <boolean>
on_failure: <processor[]>
on_success: <processor[]>
tag: <string>
FieldRequiredDefaultDescription
nameY-Table to select as the output source; templated
descriptionN-Explanatory note
disabledNfalseDisable the processor without removing it from the pipeline
ifN-Condition to run
ignore_failureNfalseSee Handling Failures
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier
- use_table:
name: Syslog
  • name: "", name: "map", and name: "none" (case-insensitive, whitespace-trimmed) all mean "marshal from the map" — the default state before any use_table has run, and a valid way to switch back to the map mid-pipeline.
  • Failure mode. Any other name must resolve to a table that was already create_table-d on this entry, or the processor fails with cannot use virtual table "<name>": unknown or not created. There is no silent fallback to the map for a name that looks like a table but wasn't created — that would defeat the purpose of an explicit switch.
  • Selecting a table doesn't clear or otherwise touch the map; it only changes what the entry's output marshal reads from. Calling use_table again later — with a different table, or with map/none — switches the output source again.

Truncate Table

Clears every value and presence bit on the named table while keeping it allocated for further writes.

- truncate_table:
name: <ident>
description: <text>
disabled: <boolean>
if: <script>
ignore_failure: <boolean>
on_failure: <processor[]>
on_success: <processor[]>
tag: <string>
FieldRequiredDefaultDescription
nameY-Table to clear; templated
descriptionN-Explanatory note
disabledNfalseDisable the processor without removing it from the pipeline
ifN-Condition to run
ignore_failureNfalseSee Handling Failures
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier
- truncate_table:
name: Syslog
  • Equivalent to zeroing the underlying struct and its presence bitmap in one step — every column goes back to absent, not just to its zero value. A truncated ProcessID reads as not-present, not as 0.
  • Truncating a table that's a known table name but was never create_table-d is a no-op that still succeeds.
  • Failure mode. An unrecognized name fails with unknown virtual table: "<name>", same as create_table.
  • Useful inside a loop (for example a foreach) where the same table shape needs to be rebuilt cleanly for each iteration without a drop_table/create_table round trip.

Drop Table

Removes the named table from the entry entirely.

- drop_table:
name: <ident>
description: <text>
disabled: <boolean>
if: <script>
ignore_failure: <boolean>
on_failure: <processor[]>
on_success: <processor[]>
tag: <string>
FieldRequiredDefaultDescription
nameY-Table to remove; templated
descriptionN-Explanatory note
disabledNfalseDisable the processor without removing it from the pipeline
ifN-Condition to run
ignore_failureNfalseSee Handling Failures
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier
- drop_table:
name: Syslog
  • If the dropped table was the entry's currently selected output table, output reverts to the map automatically — the same as running use_table with name: "map".
  • Dropping a table that was never created is a no-op that still succeeds; only an unrecognized name is a failure.
  • Failure mode. An unrecognized name fails with unknown virtual table: "<name>", same as create_table and truncate_table.

Addressing Fields: $table.Field

Any processor whose field path goes through the shared field resolver understands $<table>.Field paths — that covers set, rename (a get, set, and delete under the hood), remove, date's target_field, and if: conditions. The routing lives once, in the shared resolver, not in each processor individually — so if a new processor gains field-path support in the future, $table.Field addressing comes with it for free.

- set:
field: $syslog.Facility
value: "{{{log.syslog.facility.name}}}"
ignore_missing: true

- rename:
if: "$syslog.TimeGenerated == nil"
field: "@timestamp"
target_field: $syslog.TimeGenerated
ignore_missing: true

- remove:
field: $syslog.ProcessName

Four rules govern this addressing, and all of them are intentional rather than incidental:

  • Bare names are pure map. Facility and $syslog.Facility are two completely independent locations. Writing one never becomes readable through the other, even while both a table and the map are alive on the same entry.
  • The $ namespace never falls back to the map. A set/rename/remove against $<table>.Field fails the processor — subject to that processor's own ignore_failure/ignore_missing — instead of silently landing in the map, in every one of these cases:
    • the named table was never create_table-d (virtual table "<name>" not created),
    • the field name isn't a column of that table, or the value can't coerce into the column's type (field "<name>" is not a valid column of virtual table "<table>" — both failure reasons produce the same error text, since a column-name miss and a coercion miss are indistinguishable from the caller's side),
    • the path nests deeper than $table.Field — tables are flat, so $syslog.a.b is always an error, never a partial write into a nested structure.
  • Reads are quiet. Getting $table.Field, or checking it in an if: condition, when the table doesn't exist or the field was never set, resolves to not-found (nil) — exactly like reading a missing map key. Only writes and deletes are loud. This asymmetry is deliberate: it lets guard conditions like if: "$syslog.TimeGenerated == nil" be written safely before the table even exists, while still catching a genuine typo in a set target immediately.
  • $table alone resolves to the table object. With no field segment, $syslog on its own is readable (useful in debugging or an if: $syslog != nil existence check) but is not a valid set or remove target — there's no single value a whole table coerces down to.

Using Tables in Conditions

Every pipeline if: condition compiles against VirtualMetric's native condition engine, in Elastic-compatible mode with nil-safe navigation — see Conditional Running for the full condition language. Two consequences matter specifically for tables:

  • Event field names are case-sensitive, but $table names and their columns fold case, as covered above.
  • Navigating through a missing or nil value with a plain dot doesn't error — it resolves to nil — so $syslog.TimeGenerated == nil is safe to write whether Syslog has been created yet or not.

Reading a table's field in a condition is presence-aware: an unset column reports absent (nil) before any struct field is even examined, and a present empty string is correctly not nil. That distinction matters — a plain map-shaped nil check can't tell "never set" apart from "explicitly set to empty", but a table's presence bitmap can:

- set:
if: "$syslog.TimeGenerated == nil"
field: $syslog.TimeGenerated
value: "{{{parsed_timestamp}}}"

This runs only when TimeGenerated has genuinely never been set — an earlier explicit write, even to an empty-looking value, is respected and never silently overwritten.

A table is readable in a condition as soon as it's created — it doesn't need to also be the selected output table. use_table only controls what marshals to a target; it has no bearing on what a subsequent if: can see. Both of these resolve identically for an unset TimeGenerated, whether or not use_table has run yet:

if: "$syslog.TimeGenerated == nil"
if: "$syslog?.TimeGenerated == nil"

Optional chaining (?.) remains valid on a $table reference and behaves identically to a plain dot under the default nil-safe navigation — it's optional, not required. If a table was never created at all, $syslog.Computer (with or without ?.) still resolves to nil rather than erroring, exactly the same way navigating into a missing map field does.

tip

Guard a table write with a presence check the same way you'd guard a map write: if: "$syslog.HostName == nil" before a set that shouldn't overwrite a value a parser already populated.

Type Coercion

Every column has a fixed storage kind, decided once when the table was generated. Writing to a column coerces the incoming value into that kind and fails the write when it can't:

KindAcceptsRejects
stringstring, bool, every signed/unsigned int width, both float widths, and time.Time (formatted as RFC3339Nano, byte-identical to how encoding/json marshals a time.Time)maps, slices, structs, nil
intany numeric type (floats truncate, no rounding) and numeric stringsnon-numeric strings, nil, other types; uint/uint64 values above the maximum signed 64-bit value
int64same as int, widened to 64 bitssame as int
float64any numeric type and numeric stringsnon-numeric strings, nil, other types
boolbool and parseable strings ("true", "false", "1", "0", "t", "f", and similar)numbers — a 1 or 0 value does not coerce to true/false

The bool rule is deliberately the strictest of the five: numbers never coerce to bool, so a stray integer landing in a boolean column fails loudly instead of being silently reinterpreted as truthy or falsy. Every other kind is comparatively permissive about numeric-shaped input.

The time.Time special case on string columns is what lets a parsed-timestamp value hand itself to a table column without an intermediate formatting step: if a value arriving at a string column happens to be a native time.Time (rather than an already-formatted string), it renders with the same RFC3339Nano layout the map path's own JSON marshal would have produced for that value — so the table and map paths agree on timestamp formatting even when the write route differs.

note

Setting a string column to an empty string clears its presence bit — an empty string is treated as "absent," not "present but empty." This is specific to string columns; a set of 0 on an int/int64/float64 column, or false on a bool column, still marks that column present. For example, set: { field: $syslog.HostName, value: "" } doesn't create a present-but-empty HostName — it leaves (or makes) the field absent, so it's omitted from the marshaled output and reads back as nil in an if: guard.

Presence itself is the other half of this model: Get (and therefore every if: read) only returns a value when the column's presence bit is set. A never-written column is never readable as its Go zero value — an unset ProcessID reads as absent, not as 0; an unset HostName reads as absent, not as "". This is what makes if: "$syslog.TimeGenerated == nil" a reliable "has this been set yet" check rather than a coincidental match on a default value.

Parsing Straight into a Table: the syslog Processor's table Parameter

The syslog processor accepts an optional table: parameter. When it's absent, the processor's behavior is unchanged from ordinary map mode — it parses into the intermediate map object at target_field (default log.syslog), exactly as documented for that processor.

When table: is set, the parser writes straight into the named table's columns instead of building an intermediate map. The table must already exist — create_table has to run first, or the processor fails with unknown virtual table "<name>" (unregistered name) or virtual table "<name>" not created (registered but never allocated on this entry).

- create_table:
name: Syslog

- syslog:
field: message
table: Syslog

The field mapping, with the gating condition for each:

Table fieldSourceCondition
FacilityParsed facility nameAlways set
SeverityLevelParsed severity nameAlways set
HostNameParsed hostnameOnly if non-empty
ProcessNameParsed application nameOnly if non-empty
ProcessIDParsed process IDOnly if non-zero
SyslogMessageParsed message body, falling back to the raw line if the body is emptyAlways set (one or the other)
TimeGeneratedReparsed date from the message, formatted as RFC3339NanoOnly if TimeGenerated isn't already present

That last row mirrors the same if: "$syslog.TimeGenerated == nil" guard a map-mode date/rename chain would apply by hand — so an earlier explicit set of TimeGenerated always wins over the parser's own reparse. Columns the parser doesn't populate at all — Computer, HostIP, EventTime, SourceSystem, ManagementGroupName — are the pipeline author's job through ordinary set/rename steps targeting $syslog.*. That mirrors how a set/rename chain reads log.syslog.* in map mode.

The parser also still publishes the detected envelope format to _vmetric.syslog_format — this always lands in system metadata, independent of map or table mode, so downstream routing like if: "_vmetric?.syslog_format == 'CEF'" keeps working with no intermediate map to read it from.

warning

Any value the parser tries to write that fails to coerce into its target column fails the whole syslog step: virtual table "<table>" rejected <field>=<value> (unknown column or non-coercible value). This is rare, since the parser writes its own well-formed values, but it can surface if a table's generated schema and the parser's expected columns ever diverge.

What Gets Sent to Targets

Every output path an entry can take — the final pipeline result, reroute's staged payload, debug output — reads from the exact same switch: has a table been selected for this entry, or not?

  • No use_table has run (or the selected table was later dropped, or use_table selected map/none/""): the payload is the map, exactly as it was before virtual tables existed. Any table's contents — even a fully populated one — are simply invisible to this output.
  • use_table selected a table: the payload is only that table's present columns, serialized by its own generated, reflection-free marshal function.
warning

When a table is selected, the map is dropped in its entirety — not merged, not filtered, dropped. Any field left in the map ships nowhere: an enrichment key a reader added upstream, an intermediate field a step forgot to move, _vmetric itself. If a field needs to reach the target after use_table has run, it must be written through $<table>.Field before that point, not left as a bare-name map write. This is the single most common mistake when converting a branch to virtual tables — a field that worked fine in map mode silently disappears from the output the moment use_table starts pointing elsewhere.

Fan-Out with Clones and Reroute

Two different mechanisms both use the word "clone" here, and they behave differently with respect to tables:

  • Entry cloning — the pipeline processor's own clone: true option, and other internal entry-carrying paths — copies every table the source entry has allocated by value into a genuinely new, independent entry, and repoints that new entry's output-table selection at its own copy if the source had one selected. From that point on, the clone has its own independent copy of every table: mutating a clone's table never touches the source's, and the clone is free to create_table, write, or use_table a completely different selection afterward with zero effect on the source.
  • reroute's own clone: true is a lighter-weight, different thing: it does not create a new entry or copy any table. It marshals whatever the entry's current selected table (or map) is at that moment into a parked payload for the given destination, without persisting that reroute's routing overrides onto the entry itself — so a later step, or a later reroute, still sees the entry exactly as it was before that clone-mode reroute ran.

This pairing is what makes per-branch table selection a clean fan-out pattern. A single map-mode parse can stay untyped at the parent level while spawning one typed entry clone that builds and selects its own table shape, then fans that one already-selected table out to several destinations with a sequence of reroute steps:

pipelines:
- name: parse_syslog
processors:
# Parent stays in map mode
- syslog:
field: message
target_field: log.syslog

- pipeline:
name: typed_fanout
clone: true

- name: typed_fanout
processors:
- create_table:
name: Syslog
- set:
field: $syslog.Facility
value: "{{{log.syslog.facility.name}}}"
ignore_missing: true
- set:
field: $syslog.HostName
value: "{{{log.syslog.hostname}}}"
ignore_missing: true
- set:
field: $syslog.SyslogMessage
value: "{{{log.syslog.message}}}"
ignore_missing: true
- use_table:
name: Syslog

# Every reroute downstream of use_table marshals the typed struct
- reroute:
destination: archive
clone: true
- reroute:
destination: sentinel

Because typed_fanout selected Syslog with use_table before its reroute steps run, every reroute in that pipeline — clone-mode or not — checks the same outputTable switch at send time and serializes the typed struct for its destination. No per-destination change is needed once use_table has run, and the parent pipeline's own map-mode entry is never touched by any of it. The reroute steps themselves are completely ordinary; see reroute and Multi-Tier Pipelines for the staging and commit mechanics that pattern builds on.

Converting a Pipeline: Recipe

Converting an existing map-mode branch to a virtual table is a mechanical, five-step rewrite, plus verification:

  1. Confirm the branch is worth converting. Virtual tables pay off on a branch that writes a small, fixed, known set of fields and then marshals — a fan-out reroute leaf is the classic shape. A branch that does open-ended, dynamic-key enrichment gains nothing, since there's no table column for an arbitrary key the branch doesn't know about ahead of time.

  2. Pick the table that matches the destination shape and create_table it at the top of the branch:

    - create_table:
    name: Syslog
  3. If the branch might receive an entry that hasn't been parsed yet, feed the parser straight into the table with table: instead of building the intermediate map:

    - syslog:
    if: "log == nil"
    field: message
    table: Syslog
  4. Rewrite every set/rename/date step that wrote a bare field into the equivalent $<table>.Field path, keeping the same value expressions:

    # Before (map mode)
    - set:
    field: Facility
    value: "{{{log.syslog.facility.name}}}"
    ignore_missing: true

    # After (typed)
    - set:
    field: $syslog.Facility
    value: "{{{log.syslog.facility.name}}}"
    ignore_missing: true

    Update every if: guard the same way: if: "TimeGenerated == nil" becomes if: "$syslog.TimeGenerated == nil".

  5. Select the table once every field is written, right before the fan-out:

    - use_table:
    name: Syslog

    The reroute steps themselves need no changes — a clone: true fan-out already picks up the selected table automatically once use_table has run.

Before treating the conversion as done, verify byte-for-byte parity against the old map-mode output: run the pipeline's existing tests, and diff a captured payload from the new typed branch against the old map-mode payload for the same input. A field-by-field comparison catches a missed rename or a coercion mismatch that a simple byte-count check would not. Converting one branch doesn't obligate converting anything else — a sibling branch of the same pipeline, or the shared parser pipeline other branches also call, can stay in map mode indefinitely.

Naming Collision: reroute's table and schema Fields

caution

The reroute processor has its own, long-standing table: and schema: fields, and the pipeline-wide _vmetric.table metadata field — these are destination-routing metadata (for example, "which Azure Data Explorer table should this land in"), set as ordinary strings, with no connection whatsoever to create_table/use_table/$table.Field. A reroute step with table: "Syslog" is not creating or referencing the Syslog virtual table — it's telling the destination target what table name to write into on its end. The two features happen to share the word "table" because both predate the other's naming, not because they interact.

This distinction matters because the two can appear in the same pipeline, addressing genuinely different things:

- create_table:
name: Syslog # virtual-table engine: allocates a typed struct

- set:
field: $syslog.HostName
value: "{{{hostname}}}"

- use_table:
name: Syslog # virtual-table engine: selects the typed struct for marshal

- reroute:
destination: sentinel
table: "Syslog" # destination metadata: unrelated string, read by the target

If a pipeline needs both — a typed table selected for its own marshal efficiency, and a destination-side table name for the target to route by — both fields are set independently, with no implied relationship between their values. They can even use different casing or entirely different names without causing any conflict, precisely because neither one reads the other.

Performance

The headline numbers come from VirtualMetric's benchmark suite, comparing the same field mappings on the same input with only the storage and marshal path changed — map writes and reflection-based JSON marshaling on one side, a Syslog virtual table and its generated marshal function on the other:

ComparisonTyped vs. map
End-to-end branch (parse, set fields, marshal)~1.6x faster, 54 vs 69 allocations per event
Marshal step alone~3.2x faster, 2 vs 24 allocations per event

Marshal is where the gap is widest: a table's generated marshal function walks its own presence bitmap and appends bytes directly, with no reflection over an untyped map. End to end, the intermediate map object and its later cleanup steps are simply gone — use_table drops the map at output for free.

note

Exact timing ratios vary with hardware and machine load; the allocation counts do not — they measure heap behavior rather than clock speed, and reproduce identically everywhere. On a high-volume branch, the allocation savings are the number that compounds.

When to Use Virtual Tables

SignalUse a virtual tableStay with the map
Field setSmall, fixed, known ahead of timeOpen-ended, dynamic keys added by enrichment
Branch shapeHot fan-out leaf: parse, set fields, marshal, shipExploratory or one-off processing
Destination schemaMatches one of the 12 registered tablesDoesn't correspond to any registered schema
VolumeHigh — allocation savings compound per eventLow — the fixed cost of adopting the pattern outweighs the gain