Skip to main content

Reference

This reference covers the configuration files and fields available for self-managed Director deployments. Managed Directors receive their configuration from the DataStream platform automatically; the fields documented here are relevant when you manage Director infrastructure directly.

For Director concepts and GUI-based setup, see Overview. For installation procedures, see Deployment.

Configuration Files​

A Director reads two categories of configuration at startup:

FileLocationPurpose
Service configurationvmetric.yml in the Director working directoryCore service settings (identity, network, queue, processing defaults)
Environment configurationconfig/environments/*.ymlTopology, clustering, ACL, and TLS

Supported formats: YAML (.yml / .yaml), TOML (.toml), and VMF (.vmf). Maximum file size: 1 MB.

Config file lookup order (first match wins):

  1. vmetric.toml
  2. vmetric.yml
  3. vmetric.yaml
Debug Mode

When the virtualmetric=debug environment variable is set, the service loads vmetric-local.yml (or .toml / .yaml) instead of the standard file. This is intended for local development and troubleshooting only -- do not use in production.

Service Configuration (vmetric.yml)​

Director Identity​

FieldYAML PathTypeDefaultDescription
IDdirector.idint64--Unique Director instance identifier
Statusdirector.statusbooltrueEnable or disable the Director service
Streamdirector.streamstring--NATS stream name for Director replies

Network​

FieldYAML PathTypeDefaultDescription
Addresslisten.addressstring0.0.0.0Bind address for all listeners
External Addresslisten.external_addressstring--Advertised address for remote connections (comma-separated for multiple)
Portlisten.portint8890Main collector listening port
WebSocket Portlisten.ws_portint9080VMMQ WebSocket port
HTTP Config Portlisten.httpconfigportint80 / 443HTTP configuration server port

Queue​

FieldYAML PathTypeDefaultDescription
Limitqueue.limitint172800 (48 h)Queue size limit in seconds
Parallelismqueue.parallelismintCPU countQueue processing parallelism
Non-Existing Target Limitqueue.non_existing_target_limitint86400 (24 h)TTL for queued data to non-existing targets (seconds)
Cleanup Intervalqueue.cleanup_intervalint900 (15 min)Cleanup interval for non-existing target queues (seconds)
DeQueue Intervalqueue.dequeue_intervalint1DeQueue processing interval in seconds

Debug and Logging​

FieldYAML PathTypeDefaultDescription
Leveldebug.levelint0Debug verbosity: 0 = off, 1--3 = increasing detail
Console Statusdebug.console.statusboolfalseEnable console debug output
Log Statusdebug.log.statusboolfalseEnable file-based debug logging
Remotedebug.remoteboolfalseEnable remote debug mode (disables device tasks when true)

Fleet​

Fleet settings control how a Director connects to the DataStream cloud platform. Self-managed Directors that operate independently (no fleet connection) set self_managed: true and omit the fleet block.

FieldYAML PathTypeDefaultDescription
Self-Managedself_managedbool--If true, Director runs independently without fleet connection
Fleet Typefleet.typestring--Connection type (e.g., httpclient)
Fleet Instancefleet.instancestring--Fleet environment instance name
Fleet Addressfleet.addressstring--Fleet server WebSocket URL
Fleet Tokenfleet.tokenstring--Authentication token for fleet

Data Processing Defaults​

These defaults apply to all data processing unless overridden at the device or target level.

FieldYAML PathTypeDefaultDescription
CompressioncompressionbooltrueEnable data compression
Compression Levelcompression_levelstring--Compression level (algorithm-specific)
ReusereusebooltrueEnable connection reuse
NormalizationnormalizationbooltrueEnable data normalization
Smart Data Enginesmart_data_enginebooltrueEnable SDxL processing
No Bufferno_bufferboolfalseDisable buffering
Batch Sizebatch_sizeint6410000Read batch size
Write Batch Sizewrite_batch_sizeint641000Write batch size
WorkersworkersintCPU countWorker count
Chunk Sizechunk_sizeint4194304 (4 MB)Data chunk size in bytes
Min Chunk Countmin_chunk_countint10Minimum chunk count
Stats Enabledstats_enabledboolfalseEnable per-device, per-route, and per-target metrics collection. Fleet-managed Directors override this to true automatically.
Stats Drop On Fullstats_drop_on_fullbooltrueWhen the stats channel buffer is full, drop new metrics rather than blocking the record producer.
Disable Directdisable_directboolfalseDisable the in-process listener-to-target fast path. When false, the direct path engages automatically when eligible.

Collector Resource Management​

Controls resource allocation and scheduling for device collectors.

FieldYAML PathTypeDefaultDescription
Retry Intervalretry_intervalint30Device retry delay in seconds
Start Delaystart_delayint1Collector startup delay in seconds
Windows Startwindows_startint524288000 (500 MB)Memory threshold for Windows collectors (bytes)
Windows Start Bufferwindows_start_bufferbool--Buffer mode for Windows collectors
Monitor Intervalmonitor_intervalint30Monitor interval in seconds
Flush Intervalflush_intervalint1Log flush interval in seconds
ModulesmodulesbooltrueEnable or disable module loading

Profiler​

FieldYAML PathTypeDefaultDescription
Profile Portprofiler.director_profile_portint6061pprof profiler port
Timeoutprofiler.timeoutint60Profiler and metrics collection timeout in seconds

Load Balancer​

FieldYAML PathTypeDefaultDescription
Modeloadbalancer.modestring--Set to dynamic for dynamic load balancing across cluster nodes

Persistent Storage​

Controls whether pipeline data survives a restart, and how far upstream that durability extends. The two flags combine into four modes:

persistent_storage.statuscrash_resistanceEffect
false (default, standalone)forced falseIn-memory VMFL/VMF and JetStream MemoryStorage. Maximum throughput; records buffered in process memory are lost on a crash.
truefalseJetStream FileStorage plus the NATS Object Store for payloads. VMFL/VMF writers stay in-memory; durability is provided by NATS.
truetrueEnterprise: local VMFL/VMF queue writers also go to disk, flushing each record to the OS page cache -- process-crash durable end-to-end.
falsetrueIgnored -- crash_resistance is forced false and the mode collapses to the first row.
FieldYAML PathTypeDefaultDescription
Statuspersistent_storage.statusboolfalseEnable the durable NATS JetStream pipeline layer (JetStream FileStorage + NATS Object Store). Defaults to true on fleet-managed Directors.
Crash Resistancepersistent_storage.crash_resistanceboolfalseAlso write the local VMFL/VMF queue files to disk, flushing each record to the OS page cache so the upstream pipeline hops survive a process crash (SIGKILL, OOM, panic). Enterprise deployments; only effective when persistent_storage.status: true (forced false otherwise).
note

crash_resistance targets process-crash durability, not power loss. With persistent_storage.status: true, power-loss durability is provided by NATS -- best-effort on a single node (default 2-minute sync interval) or per-write via RAFT quorum on a cluster. For power-loss-sensitive workloads, run a cluster.

In clustered or environment-file deployments, place the persistent_storage block -- including its nested payload and pipeline_bus -- under a node's or the cluster's properties: block rather than at the top level of vmetric.yml (see the persistent-storage examples at the end of this page). These settings must be uniform across all nodes of a cluster.

Payload​

Configures the payload backend that carries log record bytes between pipeline stages. The payload and pipeline_bus blocks nest inside persistent_storage; the full key paths are persistent_storage.payload.* and persistent_storage.pipeline_bus.*.

FieldYAML PathTypeDefaultDescription
Typepersistent_storage.payload.typestring""Payload backend: inline (bytes carried in the pipeline message), object_store (NATS Object Store), memory (in-process store), or "" (automatic — memory when persistent storage is off, object_store when on).
Max Sizepersistent_storage.payload.max_sizestring""Maximum payload size per record. Accepts a plain integer (bytes) or a size suffix such as 1mb or 512kb. When the payload type is inline defaults to 1 MB to fit within Kafka and Event Hubs message limits; unlimited for other backends.

Pipeline Bus​

Configures the transport that moves stage-to-stage pipeline notifications between the ingester, router, and sender workers.

FieldYAML PathTypeDefaultDescription
Typepersistent_storage.pipeline_bus.typestringnatsStage-to-stage transport: nats (embedded VMMQ, default) or kafka (external Kafka broker including Azure Event Hubs' Kafka endpoint).
Addresspersistent_storage.pipeline_bus.addressstring--Comma-separated Kafka broker addresses. Required when the pipeline bus type is kafka.
Topic Prefixpersistent_storage.pipeline_bus.topic_prefixstringvm-pipelinePrefix for stage topics. Topics are named <prefix>.<stage> (e.g., vm-pipeline.sender).
Algorithmpersistent_storage.pipeline_bus.algorithmstring--Kafka SASL algorithm: plain, scram-sha-256, scram-sha-512, or gssapi.
Usernamepersistent_storage.pipeline_bus.usernamestring--Kafka SASL username.
Passwordpersistent_storage.pipeline_bus.passwordstring--Kafka SASL password.
TLS Statuspersistent_storage.pipeline_bus.tls.statusboolfalseEnable TLS for the Kafka broker connection.
note

pipeline_bus.type: kafka requires payload.type: inline. If payload.type is not inline, the Director logs a warning and reverts to the nats transport.

Environment Configuration​

Environment files define Director topology, clustering, and access control. Located in config/environments/. Files containing .example. or .sample. in their name are templates only and are not loaded by the system.

Standalone Director​

Minimal configuration for a single Director instance. The node name must match the director.id from vmetric.yml.

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true

Clustered Director​

Clustered configurations define a cluster with multiple nodes for high availability. See Clusters for operational details.

cluster: "{cluster-id}"
environments:
  - name: "{cluster-id}"
    description: "Production Cluster"
    status: true
    clusters:
      - name: "{cluster-id}"
        description: "Primary Cluster"
        status: true
        type: director
        properties:
          address: "10.0.0.1"
          port: 6222
          authentication:
            username: ""
            password: ""
            token: ""
          tls:
            status: false
            cert_name: "cert.pem"
            key_name: "key.pem"
            ca_name: ""
            verify: false
        nodes:
          - name: "{node-1-id}"
            status: true
            properties:
              address: "10.0.0.1"
              port: 4222
              cluster:
                address: "10.0.0.1"
                port: 6222
                authentication:
                  token: ""
                tls:
                  status: false

ACL​

ACL rules can be applied at the cluster or node level within environment files.

FieldYAML PathTypeDefaultDescription
Allowed IPsacl.allowed_ips[]string--IP allowlist (individual IPs, ranges "ip1 - ip2", or CIDR notation)
Access Tokensacl.allowed_tokens[]string--Valid access tokens for API authentication
Rate Limit Maxacl.rate_limit_max_requestsint100Maximum requests per rate limit window
Rate Limit Windowacl.rate_limit_windowint60Rate limit window in seconds
Update Intervalacl.update_intervalint20ACL configuration refresh interval in seconds

TLS​

TLS settings for inter-node and cluster communication.

FieldYAML PathTypeDefaultDescription
Statustls.statusboolfalseEnable TLS
Cert Nametls.cert_namestringcert.pemTLS certificate filename
Key Nametls.key_namestringkey.pemTLS key filename
CA Nametls.ca_namestring--CA certificate filename
Verifytls.verifyboolfalseEnable certificate verification

Node Properties​

FieldYAML PathTypeDefaultDescription
Addressproperties.addressstring--Node bind address
Portproperties.portint4222Node NATS port
Cluster Addressproperties.cluster.addressstring--Cluster communication address
Cluster Portproperties.cluster.portint6222Cluster communication port

Health Check Configuration​

Health checks monitor device collector resources. A resource must exceed its threshold for the configured number of consecutive iterations before action is taken.

Check Toggles​

FieldYAML PathTypeDefaultDescription
Check CPU Usagehealth.check_cpu_usageboolfalseMonitor CPU usage
Check Memory Usagehealth.check_memory_usageboolfalseMonitor memory usage
Check Log File Usagehealth.check_log_file_usagebooltrueMonitor log file sizes
Check TMP Usagehealth.check_tmp_usageboolfalseMonitor temp directory
Check Report Usagehealth.check_report_usagebooltrueMonitor report files
Check Simulator Usagehealth.check_simulator_usagebooltrueMonitor simulator
Check Hanged Processeshealth.check_hanged_processesbooltrueDetect hung processes
Check Stats Usagehealth.check_stats_usagebooltrueMonitor stats collection

Thresholds​

FieldDefaultDescription
CPU Limit80%CPU usage threshold percentage
Memory Limit1024 MBMemory usage threshold
Max Log Size10 MBMaximum log file size
Stats Retention24 hoursStats retention period

Threshold Iterations​

All threshold iteration defaults are 10. A resource must exceed its threshold for this many consecutive check iterations before the health system takes action.

VIP (Virtual IP) Configuration​

VIP enables high-availability failover for clustered Directors. Configured within cluster properties in the environment file.

FieldYAML PathTypeDefaultDescription
Use VIPvip.use_vipboolfalseEnable VIP management
Bind VIPvip.bind_vipboolfalseBind VIP on startup
VIP Addressvip.vip_addressstring--Virtual IP address
VIP Netmaskvip.vip_netmaskstring--VIP subnet mask
VIP Gatewayvip.vip_gatewaystring--VIP gateway address
VIP Interfacevip.vip_interfacestring--Network interface for VIP
VIP Monitor Intervalvip.vip_monitor_intervalint--Health check interval in seconds
Is NATS Clustervip.is_nats_clusterboolfalseWhether this is a NATS cluster (affects VIP binding logic)

Environment Variables​

VariableValueEffect
virtualmetricdebugLoads vmetric-local.yml instead of vmetric.yml

Debug config file lookup order:

  1. {serviceName}-local.toml
  2. {serviceName}-local.yml
  3. {serviceName}-local.yaml

Internal Timing Constants​

These values are hardcoded and not user-configurable. They are documented here for operational awareness.

Manager Intervals​

ConstantValuePurpose
Heartbeat Update Interval5 sDevice heartbeat publishing frequency
Leader Check Interval1 sLeadership election check frequency
KV Sync Interval5 sJetStream KV synchronization
Config Sync Interval1 sConfiguration change polling
Heartbeat Check Interval15 sDevice health check and node offline threshold

Cluster Update Intervals​

ConstantValuePurpose
Cluster Update Check Interval10 sPoll for pending updates
Node Timeout30 minMaximum wait for a single node update
Health Check Wait30 sPost-update health check delay
Leader Stability30 sRequired leader stability before proceeding
Command Retry Interval30 sBetween retry attempts
Max Retries3Maximum update retries per node
Max Offline Retries5Maximum retries for offline nodes

Configuration Limits​

ConstantValuePurpose
Config File Max Size1 MBMaximum configuration file size
Retention Interval~3 monthsDefault data retention period

Examples​

Minimal Standalone Director​

vmetric.yml:

director:
  id: 1
listen:
  address: "0.0.0.0"
  port: 8080
debug:
  level: 1
  console:
    status: true

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true

Production Standalone with ACL​

vmetric.yml:

director:
  id: 1
listen:
  address: "0.0.0.0"
  external_address: "10.0.1.100"
  port: 8890
queue:
  limit: 900000000
debug:
  level: 1
  console:
    status: false
  log:
    status: true

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true
        properties:
          acl:
            allowed_ips:
              - "10.0.0.0/24"
              - "192.168.1.50"
              - "172.16.0.1 - 172.16.0.254"
            access_tokens:
              - "prod-token-abc123"
            rate_limit_max_requests: 200
            rate_limit_window: 60

Three-Node HA Cluster with VIP​

vmetric.yml (same on all nodes, only director.id differs):

director:
  id: 1  # Change to 2, 3 on other nodes
listen:
  address: "0.0.0.0"
  external_address: "10.0.1.101"
  port: 8890
queue:
  limit: 900000000
debug:
  level: 2
  console:
    status: true
  log:
    status: true
loadbalancer:
  mode: dynamic

config/environments/cluster.yml:

cluster: "prod-cluster"
environments:
  - name: "prod-cluster"
    description: "Production HA Cluster"
    status: true
    clusters:
      - name: "prod-cluster"
        description: "3-node director cluster"
        status: true
        type: director
        properties:
          address: "10.0.1.101"
          port: 6222
          authentication:
            token: "cluster-secret-token"
          tls:
            status: true
            cert_name: "cluster-cert.pem"
            key_name: "cluster-key.pem"
            ca_name: "ca.pem"
            verify: true
          acl:
            allowed_ips:
              - "10.0.0.0/16"
            allowed_tokens:
              - "api-token-xyz"
            rate_limit_max_requests: 500
            rate_limit_window: 60
          vip:
            use_vip: true
            bind_vip: true
            vip_address: "10.0.1.100"
            vip_netmask: "255.255.255.0"
            vip_gateway: "10.0.1.1"
            vip_interface: "eth0"
            vip_monitor_interval: 5
            is_nats_cluster: true
          update:
            mode: auto
            method: cron
            cron: "0 3 * * 0"
            version: "latest"
        nodes:
          - name: "1"
            description: "Node 1 - 10.0.1.101"
            status: true
            properties:
              address: "10.0.1.101"
              port: 4222
              cluster:
                address: "10.0.1.101"
                port: 6222
                authentication:
                  token: "cluster-secret-token"
                tls:
                  status: true
                  cert_name: "cluster-cert.pem"
                  key_name: "cluster-key.pem"
                  ca_name: "ca.pem"
                  verify: true
          - name: "2"
            description: "Node 2 - 10.0.1.102"
            status: true
            properties:
              address: "10.0.1.102"
              port: 4222
              cluster:
                address: "10.0.1.102"
                port: 6222
                authentication:
                  token: "cluster-secret-token"
                tls:
                  status: true
                  cert_name: "cluster-cert.pem"
                  key_name: "cluster-key.pem"
                  ca_name: "ca.pem"
                  verify: true
          - name: "3"
            description: "Node 3 - 10.0.1.103"
            status: true
            properties:
              address: "10.0.1.103"
              port: 4222
              cluster:
                address: "10.0.1.103"
                port: 6222
                authentication:
                  token: "cluster-secret-token"
                tls:
                  status: true
                  cert_name: "cluster-cert.pem"
                  key_name: "cluster-key.pem"
                  ca_name: "ca.pem"
                  verify: true

Fleet-Managed Director​

vmetric.yml:

self_managed: false
fleet:
  type: httpclient
  instance: production
  address: wss://fleet.virtualmetric.io/ws
  token: "fleet-auth-token-abc123"
director:
  id: 2028574672450228224
listen:
  address: "0.0.0.0"
  port: 8890
debug:
  level: 1
  console:
    status: true

Persistent Storage (Embedded NATS)​

Durable pipeline on the embedded NATS backend -- the default when no payload or pipeline_bus is set. JetStream switches to FileStorage and payloads move to the NATS Object Store; VMFL/VMF writers stay in-memory. In a cluster, put these settings on the cluster's properties (they must be uniform across all nodes), not per-node.

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true
        properties:
          persistent_storage:
            status: true

For the Enterprise crash-resistant mode, add crash_resistance: true -- the local VMFL/VMF queue writers also go to disk, flushed per record:

persistent_storage:
  status: true
  crash_resistance: true

External Kafka / Redpanda​

Routes the router-to-sender hop over an external Kafka-compatible broker. Requires payload.type: inline -- with any other payload type the bus silently falls back to nats. The topic (vm-pipeline.sender) and consumer group (<director-name>-sender) auto-create where the broker allows it.

config/environments/env.yml:

environments:
  - name: "1"
    status: true
    nodes:
      - name: "1"
        status: true
        properties:
          persistent_storage:
            status: true
            payload:
              type: inline
              max_size: 512kb
            pipeline_bus:
              type: kafka
              address: "{broker-1}:9092,{broker-2}:9092"
              topic_prefix: vm-pipeline
              # Optional SASL auth -- omit the block below for a plaintext broker.
              # algorithm: plain            # plain | scram-sha-256 | scram-sha-512 | gssapi
              # username: "{username}"
              # password: "{password}"
              # tls:
              #   status: true

The inline payload rides base64-encoded inside the bus message, so max_size caps the raw payload before encoding (~4/3 on the wire). 512kb (~683 KB encoded) stays under a default broker's ~1 MB message.max.bytes; raise it only in step with the broker limit. For crash resistance, add crash_resistance: true under persistent_storage (the shipped example also raises max_size to 1mb).

Azure Event Hubs (Kafka endpoint)​

Uses the Event Hubs Kafka endpoint as the pipeline bus -- the durable-broker option for ephemeral deployments (e.g. Azure Container Apps) with no durable local disk, where in-flight data survives a container restart inside the managed broker. SASL PLAIN over TLS on port 9093.

config/environments/env.yml:

environments:
  - name: "{director-id}"
    status: true
    nodes:
      - name: "{director-id}"
        status: true
        properties:
          persistent_storage:
            status: true
            payload:
              type: inline
              max_size: 512kb
            pipeline_bus:
              type: kafka
              address: "{namespace}.servicebus.windows.net:9093"
              topic_prefix: vm-pipeline
              algorithm: plain
              username: "$ConnectionString"
              password: "Endpoint=sb://{namespace}.servicebus.windows.net/;SharedAccessKeyName={policy};SharedAccessKey={key}"
              tls:
                status: true

Event Hubs does not auto-create topics: pre-create the event hub vm-pipeline.sender and the consumer group <director-name>-sender before starting the Director -- a missing topic drops 100% of pipeline traffic. Keep payload.max_size at or below the namespace tier's message cap (Basic 256 KB, Standard/Premium 1 MB, Dedicated 20 MB), measured on the raw payload before base64. Add crash_resistance: true under persistent_storage for the Enterprise crash-resistant mode.