Routes: Overview

Routes define how incoming data is filtered, processed, and directed to specific destinations. They act as traffic controllers, determining what data goes through which pipelines and ultimately reaches which targets.

Definitions

Routes are made up of filtering expressions and sequences of processors that shape the format and flow of data streams. They curate data through an evaluation process and forward it to downstream consumers.

Their behavior is characterized by three key elements:

Devices - One or more sources to ingest data from
Pipelines - Optional processing schemes applied to selected events
Targets - One or more destinations to forward the processed data

In addition to these, there is one more consideration that involves the binary relations between devices and pipelines, pipelines and targets, or directly devices and targets: the optional filter expressions that match specific events.

Configuration

All routes share the following base configuration fields:

Field	Required	Default	Description
`name`	Y		Unique identifier for the route
`description`	N		Optional explanation of the route's purpose
`devices`	N		List of source IDs to collect events from
`if`	N		Expression determining if the route applies to an event
`pipelines`	N		List of pipelines to process matching events (processed in parallel)
`targets`	N		List of targets to deliver processed events to
`status`	N	`true`	Enable/disable the route

warning

Do not confuse field names like pipelines or targets with the component entries at the first level. As the highest level of a telemetry configuration, routes orchestrate the stream by specifying the components that constitute the traffic.

note

Enlisted devices, targets, and pipelines must be configured in advance to be usable by routes. Use their ids or names to refer to them.

Example:

routes:
  - name: syslog_to_backup
    devices: 
      - name: syslog_*
    pipelines: 
      - name: normalize
      - name: enrich
    targets: 
      - name: backup

This route receives data from devices with the prefix syslog_ in their names, transforms it using the normalize and enrich pipelines, and forwards it to the backup target.

Deployment

Forwarding data involves several design patterns (use of pipelines is optional in each case):

Single source to single target

Syslog → Route → [Normalize] → Storage
Single source to multiple targets

Syslog → Route → [Enrich] → Monitoring + Storage + Backup
Multiple sources to single target

Syslog + Apache → Route [Normalize] → Central Analytics
Multiple sources to multiple targets

Syslog + Winlog + Apache → Route [Normalize ∘ Enrich] → Cloud Upload + Central Analytics

Routes can include zero or more pipelines. Multiple pipelines process data in parallel. A pipeline's processors can override routing using the reroute processor, and the final processor terminates execution.

Route Types

Routes can be categorized based on their function in the data flow:

Primary Routes - These routes handle critical data paths with high priority:
- Specific filtering, targeted processing, and specialized destinations
Filtering Routes - These routes select specific subsets of data:
- Data segregation based on categories, security levels, or compliance requirements
Processing Routes - These routes apply transformations to data:
- Apply enrichment, normalization, or specialized processing
Distribution Routes - These routes focus on delivering data to targets:
- Fan-out delivery, redundant storage, or multi-system integration
Unconditional Routes - These routes process all events regardless of content:
- No if condition means every event is processed by this route

Use Cases

Some scenarios demonstrating the uses of routes:

Security Event Workflow:

routes:
  - name: security_critical
    if: "event.category == 'security' && event.severity == 'critical'"
    pipelines:
      - name: security_enrichment
      - name: threat_intelligence
    targets:
      - name: siem_platform
      - name: security_team_notification
  
  - name: security_standard
    if: "event.category == 'security'"
    pipelines:
      - name: security_enrichment
    targets:
      - name: security_elasticsearch

Data Segregation:

routes:
  - name: pci_data
    if: "tags.contains('pci')"
    pipelines:
      - name: pci_compliance
      - name: data_masking
    targets:
      - name: pci_storage
  
  - name: hipaa_data
    if: "tags.contains('hipaa')"
    pipelines:
      - name: hipaa_compliance
      - name: phi_protection
    targets:
      - name: hipaa_storage

Environment Separation:

routes:
  - name: production_logs
    if: "kubernetes.namespace == 'production'"
    pipelines:
      - name: production_enrichment
    targets:
      - name: production_elasticsearch
  
  - name: development_logs
    if: "kubernetes.namespace == 'development'"
    pipelines:
      - name: dev_enrichment
    targets:
      - name: development_elasticsearch

Implementation Strategies

The following aspects of your telemetry stream must be considered in the implementation of routes:

Conditional Routing:

Selecting events involves evaluating boolean expressions such as:

device.type == 'syslog'
dataset.name == 'firewall_logs'
log.severity > 4
!(source.ip == '10.0.0.1'||source.ip == '10.0.0.2')

Key points about route evaluation:
- All routes are evaluated for every event in parallel
- Each route independently filters events using its if condition
- Events matching multiple route conditions are processed by all matching routes
- Route conditions are evaluated before pipelines begin their work
- Multiple targets within a route receive identical copies of the same data
Examples of conditional routing:
```
routes:
  - name: critical_alerts
    if: "event.severity == 'critical'||event.severity == 'high'"
    pipelines:
      - name: alert_enrichment
    targets:
      - name: alerts_webhook
      - name: security_team_email
  
  - name: normal_logs
    if: "event.severity == 'low'||event.severity == 'medium'"
    pipelines:
      - name: standard_processing
    targets:
      - name: logs_elasticsearch
```

Parallel Route Processing:

All routes are processed concurrently. Use distinct if conditions to control which events each route handles:

routes:
  - name: urgent_alerts
    if: "event.severity == 'critical'"
    targets:
      - name: alerting_system

  - name: security_events
    if: "event.category == 'security'"
    targets:
      - name: security_platform

  - name: general_logs
    targets:
      - name: log_storage

In this example, a critical security event would be processed by all three routes simultaneously.

Device Selection:

Routes can collect data from specific devices:

routes:
  - name: network_traffic
    devices:
      - name: firewall_logs
      - name: router_logs
      - name: switch_logs
    pipelines:
      - name: network_enrichment
    targets:
      - name: network_monitoring

Multiple Pipelines:

Routes can apply multiple pipelines, which are processed in parallel:

routes:
  - name: web_security
    devices:
      - web_server_logs
    if: "event.category == 'web' && event.type == 'attack'"
    pipelines:
      - common_enrichment
      - attack_detection
      - threat_scoring
    targets:
      - security_elasticsearch

Each pipeline processes the event data independently and concurrently.

Multi-Target Delivery:

Routes can send processed data to multiple destinations:

routes:
  - name: compliance_events
    if: "tags.contains('compliance')"
    pipelines:
      - name: compliance_formatting
    targets:
      - name: compliance_elasticsearch  # For searching and analytics
      - name: compliance_s3             # For long-term retention
      - name: auditor_webhook           # For real-time notification

Unconditional Routes:

Routes without an if condition process all events:

routes:
  - name: primary_security
    if: "event.category == 'security'"
    targets:
      - name: security_elasticsearch

  - name: all_events
    # No if condition - processes every event
    targets:
      - name: general_elasticsearch

Since routes run in parallel, events matching primary_security will also be processed by all_events.

Route Templates:

Use environment variables for flexible configurations.

routes:
  - name: dynamic_routing
    if: "${ROUTING_CONDITION}"
    pipelines:
      - ${PRIMARY_PIPELINE}
      - ${SECONDARY_PIPELINE}
    targets:
      - ${TARGET_SYSTEM}

Definitions​

Configuration​

Deployment​

Route Types​

Use Cases​

Implementation Strategies​