Skip to main content

Commit

Flow Control

Synopsis​

Finalizes staged routes created by the reroute processor with staging: true, moving events from the staging area to their designated destinations.

Schema​

- commit:
    destination: <string>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration​

FieldRequiredDefaultDescription
destinationN-Specific destination to commit. If omitted, commits all staged routes.
descriptionN-Explanatory note
ifN-Condition to run
ignore_failureNfalseSee Handling Failures
on_failureN-See Handling Failures
on_successN-See Handling Success
tagN-Identifier

Details​

The commit processor works with the reroute processor to enable multi-tier pipeline architectures. When reroute uses staging: true, events are held in a staging area instead of being delivered immediately. The commit processor finalizes those staged routes.

Staged Route Behavior:

  • Staged routes are stored in a map keyed by destination name
  • Multiple staged routes to the same destination overwrite previous versions
  • Only the final staged version for each destination is committed
  • Without commit, staged routes are discarded at pipeline end

Commit Modes:

  • Commit all: When destination is omitted, all staged routes are committed
  • Commit specific: When destination is specified, only that destination is committed

Error Handling:

  • Returns error if specified destination is not found in staged routes
  • Use ignore_failure: true to continue processing when destination not found
  • Empty staging area with no destination specified succeeds silently

This pattern enables progressive normalization where data passes through multiple transformation stages, with only the final, most-normalized version delivered to each destination. See Multi-Tier Pipelines for detailed patterns.

Examples​

Commit All Staged Routes​

Committing all staged routes after multi-tier processing...

processors:
  # Stage to archive
  - reroute:
      destination: "archive"
      table: "Syslog"
      staging: true

  # Normalize and stage to SIEM
  - normalize:
      target_format: csl
  - reroute:
      destination: "sentinel"
      table: "CommonSecurityLog"
      staging: true

  # Commit all destinations
  - commit:

Both archive and sentinel receive their respective formats...

Commit Specific Destination​

Committing only a specific destination...

processors:
  - reroute:
      destination: "sentinel"
      table: "Syslog"
      staging: true

  - reroute:
      destination: "lake"
      staging: true

  # Commit only sentinel
  - commit:
      destination: "sentinel"

  # Continue processing for lake
  - enrich:
      field: "additional_data"

  # Commit remaining
  - commit:

Sentinel receives data immediately, lake receives enriched data later...

Staged Route Override​

Progressive normalization with automatic override...

processors:
  # Stage Syslog format
  - reroute:
      destination: "sentinel"
      table: "Syslog"
      staging: true

  # Normalize to CSL and stage (overwrites)
  - normalize:
      target_format: csl
  - reroute:
      destination: "sentinel"
      table: "CommonSecurityLog"
      staging: true

  # Normalize to ASIM and stage (overwrites again)
  - normalize:
      target_format: asim
  - reroute:
      destination: "sentinel"
      table: "ASimNetworkSession"
      staging: true

  # Only ASIM version is committed
  - commit:

Only the final ASIM version reaches sentinel...

Conditional Commit​

Committing based on validation results...

processors:
  - normalize:
      target_format: asim
  - check_schema:
      schema: "ASimNetworkSessionLogs"
      target_field: "schema_check"
  - reroute:
      destination: "sentinel"
      staging: true

  # Only commit if schema is valid
  - commit:
      destination: "sentinel"
      if: "schema_check.is_valid == true"

  # Route invalid events elsewhere
  - reroute:
      if: "schema_check.is_valid == false"
      destination: "quarantine"

Valid events go to sentinel, invalid to quarantine...

Error Handling​

Handling missing destination gracefully...

processors:
  - commit:
      destination: "nonexistent"
      ignore_failure: true
      on_failure:
        - set:
            field: "commit_error"
            value: "destination not found"

Processing continues with error captured...

{
  "commit_error": "destination not found"
}