DataStream Stats

Pull

Synopsis

The device subscribes to the Director's internal NATS JetStream stats stream and republishes route, target, device, pipeline, device-resource, and queue metrics through standard pipelines and routes. Each device instance uses its own durable consumer with an independent read position. Output field naming is selectable between ECS, Cribl, and Prometheus conventions.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: stats
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    poll_interval: <numeric>
    workers: <numeric>
    field_format: <string>

Configuration

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be stats
tags	N	-	Array of labels for categorization
pipelines	N	-	Array of preprocessing pipeline references
status	N	true	Enable/disable the device

Polling

Field	Required	Default	Description
poll_interval	N	60	How often to scan for completed partitions, in seconds. Must be greater than 0.
workers	N	1	Number of worker goroutines that drain the partition queue

Output

Field	Required	Default	Description
field_format	N	cribl	Output field naming convention. One of ecs, cribl, prometheus.

Details

What Gets Emitted

Each polling cycle emits records in one of six input_type categories:

• route — per-route counters including events and bytes in/out, latency, errors, and drops
• target — per-target counters with the same shape as route records, plus a target identifier
• device — per-device counters tracking events and bytes processed by each device
• pipeline — per-pipeline execution_count and execution_time aggregates
• device_resource — host resource usage with resource_type of volume, cpu, or memory, plus identifier and total/utilized sizes
• queue — queue depth and discard counters including pending_files, pending_bytes, discarded_files, and discarded_bytes

Output Formats

Three field naming conventions are supported:

• ecs — flat snake_case internal representation using ECS-aligned field names such as route_name, target_name, latency_ns, and error_count
• cribl — Cribl Stream internal-metrics naming with cribl_* system fields (cribl_route, cribl_output, cribl_host) and underscore-prefixed custom fields (_time, _type, _events_in, _error_count)
• prometheus — Prometheus base-unit naming with _total, _nanoseconds, and _bytes suffixes (events_in_total, latency_nanoseconds, bytes_in_total)

Polling Cadence

poll_interval must be greater than 0; the device rejects any configuration where this constraint is violated.

Lifecycle

The device has no external network connectivity. On start, it validates poll_interval configuration and, if valid, reports ConnectionStateConnected and subscribes to the JetStream stats stream with a per-device durable consumer (role device-<id>). If poll_interval is invalid, it reports ConnectionStateErrorConfiguration. If 120 seconds pass without an internal heartbeat, the device reports ConnectionStateErrorHeartbeat and stops the collector.

Restart Triggers

Changes to poll_interval, workers, or field_format trigger a clean collector restart. Changes to other configuration fields do not restart the collector.

Examples

Basic Configuration

Creating a basic statistics publisher with default polling and Cribl output naming...	- id: 1 name: stats-publisher type: stats
Device polls every 60 seconds and emits route records with Cribl field naming...	{ "_time": 1746009600, "_input_type": "route", "_type": "in", "cribl_route": "main-route", "cribl_host": "director-01", "_events_in": 14820, "_events_out": 14820, "bytes_in": 9437184, "bytes_out": 9437184, "_latency_ns": 412000, "_error_count": 0, "_dropped_count": 0 }

ECS Format

Configuring flat ECS field naming for the emitted records...	- id: 2 name: stats-ecs type: stats properties: field_format: ecs
Route records use ECS-aligned snake_case field names...	{ "timestamp": 1746009600, "input_type": "route", "type": "in", "route_name": "main-route", "director_name": "director-01", "events_in": 14820, "events_out": 14820, "bytes_in": 9437184, "bytes_out": 9437184, "latency_ns": 412000, "error_count": 0, "dropped_count": 0 }

Cribl Format

Configuring Cribl Stream internal-metrics field naming explicitly...	- id: 3 name: stats-cribl type: stats properties: field_format: cribl
Route records use Cribl system fields with underscore-prefixed custom fields...	{ "_time": 1746009600, "_input_type": "route", "_type": "in", "cribl_route": "main-route", "cribl_host": "director-01", "_events_in": 14820, "_events_out": 14820, "bytes_in": 9437184, "bytes_out": 9437184, "_latency_ns": 412000, "_error_count": 0, "_dropped_count": 0 }

Prometheus Format

Configuring Prometheus base-unit field naming for the emitted records...	- id: 4 name: stats-prometheus type: stats properties: field_format: prometheus
Route records use Prometheus naming conventions with _total, _nanoseconds, and _bytes suffixes...	{ "timestamp_ms": 1746009600000, "input_type": "route", "direction": "in", "route": "main-route", "instance": "director-01", "events_in_total": 14820, "events_out_total": 14820, "bytes_in_total": 9437184, "bytes_out_total": 9437184, "latency_nanoseconds": 412000, "errors_total": 0, "dropped_total": 0 }

High-Volume Polling

Configuring explicit poll interval, workers, and ECS output for stats collection...	- id: 5 name: stats-highvolume type: stats properties: poll_interval: 30 workers: 4 field_format: ecs
Device subscribes to the stats stream with ECS output; changes to poll_interval, workers, or field_format trigger a collector restart...