Event Hubs

Microsoft Azure

Synopsis

Creates a collector that connects to Azure Event Hubs and consumes messages from specified event hubs. Supports multiple authentication methods, TLS encryption, and multiple workers for high-throughput scenarios.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: eventhubs
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    event_hub: <string>
    client_connection_string: <string>
    tenant_id: <string>
    client_id: <string>
    client_secret: <string>
    namespace: <string>
    consumer_group: <string>
    container_connection_string: <string>
    container_url: <string>
    container_name: <string>
    reuse: <boolean>
    workers: <numeric>
    tls:
      status: <boolean>
      cert_name: <string>
      key_name: <string>
      insecure_skip_verify: <boolean>

Configuration

The following fields are used to define the device:

Device

Field	Required	Default	Description
`id`	Y		Unique identifier
`name`	Y		Device name
`description`	N	-	Optional description
`type`	Y		Must be `eventhubs`
`tags`	N	-	Optional tags
`pipelines`	N	-	Optional pre-processor pipelines
`status`	N	`true`	Enable/disable the device

Connection

Event Hubs supports two authentication methods:

Method 1: Connection String Authentication

Field	Required	Default	Description
`client_connection_string`	Y*		Event Hubs connection string (required if not using service principal)
`event_hub`	Y		Event hub name to consume from

Method 2: Service Principal Authentication

Field	Required	Description
`tenant_id`	Y*	Azure tenant ID (required if not using connection string)
`client_id`	Y*	Azure service principal client ID
`client_secret`	Y*	Azure service principal client secret
`namespace`	Y*	Event Hubs namespace (required if not using connection string)
`event_hub`	Y	Event hub name to consume from

Consumer Configuration

Field	Required	Default	Description
`consumer_group`	N	`"$Default"`	Consumer group name

Storage Configuration

EventHubs requires checkpoint storage. Choose one method:

Method 1: Storage Account Connection String

Field	Required	Default	Description
`container_connection_string`	Y*		Azure Storage connection string
`container_name`	Y*		Blob container name for checkpoints

Method 2: Storage Account URL

Field	Required	Default	Description
`container_url`	Y*		Azure Storage container URL

* = Conditionally required (see authentication and storage methods above)

Performance

Field	Required	Default	Description
`reuse`	N	`true`	Enable multi-worker mode
`workers`	N	`4`	Number of worker processes when reuse enabled

TLS

Field	Required	Default	Description
`tls.status`	N	`false`	Enable TLS encryption
`tls.cert_name`	N*		TLS certificate file name (required if TLS enabled)
`tls.key_name`	N*		TLS private key file name (required if TLS enabled)
`tls.insecure_skip_verify`	N	`false`	Skip server certificate verification

* = Conditionally required (only when tls.status: true)

Details

IAM Permissions

When using service principal authentication, the following Azure RBAC roles are required:

Azure Role	Scope	Purpose
`Azure Event Hubs Data Receiver`	Event Hubs Namespace or Event Hub	Consume events and read hub properties
`Storage Blob Data Contributor`	Storage Account or Container	Read, write, and list checkpoint blobs

The checkpoint storage requires Contributor (not just Reader) because the device writes checkpoint state and manages ownership blobs for partition load balancing.

When using connection string authentication, Azure RBAC roles are not needed for Event Hubs access. The Shared Access Policy embedded in the connection string governs access (typically Listen claim for consumers). Checkpoint storage still requires either a connection string or RBAC role assignment.

Multiple Workers

When reuse is enabled, the collector uses multiple workers. Each worker maintains its own Event Hubs consumer and processes messages independently, automatically balancing message volumes.

Messages

The collector supports automatic checkpoint management, consumer group load balancing, multiple Event Hub subscriptions, TLS-encrypted connections, both connection string and service principal authentication, and custom message-processing pipelines.

Examples

Basic with Connection String

Creating a simple EventHubs consumer with connection string...

- id: 1
  name: basic_eventhubs
  type: eventhubs
  properties:
    client_connection_string: "Endpoint=sb://mynamespace.servicebus.windows.net/;SharedAccessKeyName=mykey;SharedAccessKey=myvalue"
    event_hub: "logs"
    container_connection_string: "DefaultEndpointsProtocol=https;AccountName=mystorage;AccountKey=mykey"
    container_name: "checkpoints"

Service Principal Authentication

Connecting with service principal authentication...

- id: 2
  name: sp_eventhubs
  type: eventhubs
  properties:
    tenant_id: "12345678-1234-1234-1234-123456789012"
    client_id: "87654321-4321-4321-4321-210987654321"
    client_secret: "${AZURE_CLIENT_SECRET}"
    namespace: "mynamespace"
    event_hub: "security-logs"
    consumer_group: "datastream-group"
    container_url: "https://mystorage.blob.core.windows.net/checkpoints"

High-Volume Processing

Optimizing for throughput with multiple workers...

- id: 3
  name: performant_eventhubs
  type: eventhubs
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "high-volume-logs"
    consumer_group: "processing-group"
    container_connection_string: "${STORAGE_CONNECTION_STRING}"
    container_name: "checkpoints"
    reuse: true
    workers: 8

Secure Connection

Secure EventHubs connection with TLS...

- id: 4
  name: secure_eventhubs
  type: eventhubs
  properties:
    tenant_id: "${AZURE_TENANT_ID}"
    client_id: "${AZURE_CLIENT_ID}"
    client_secret: "${AZURE_CLIENT_SECRET}"
    namespace: "secure-namespace"
    event_hub: "secure-logs"
    consumer_group: "secure-group"
    container_url: "${STORAGE_CONTAINER_URL}"
    tls:
      status: true
      cert_name: "eventhubs.crt"
      key_name: "eventhubs.key"

Pipeline Processing

Applying custom processing to EventHubs messages...

- id: 5
  name: pipeline_eventhubs
  type: eventhubs
  pipelines:
    - json_parser
    - field_extractor
    - normalize_timestamps
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "application-logs"
    consumer_group: "processing-group"
    container_connection_string: "${STORAGE_CONNECTION_STRING}"
    container_name: "checkpoints"

Multiple Consumer Groups

Configuring consumer groups for load distribution...

- id: 6
  name: distributed_eventhubs
  type: eventhubs
  properties:
    client_connection_string: "${EVENTHUBS_CONNECTION_STRING}"
    event_hub: "distributed-logs"
    consumer_group: "instance-1"
    container_connection_string: "${STORAGE_CONNECTION_STRING}"
    container_name: "checkpoints-instance1"
    reuse: true
    workers: 4

Synopsis​

Schema​

Configuration​

Device​

Connection​

Consumer Configuration​

Storage Configuration​

Performance​

TLS​

Details​

IAM Permissions​

Multiple Workers​

Messages​

Examples​

Basic with Connection String​

Service Principal Authentication​

High-Volume Processing​

Secure Connection​

Pipeline Processing​

Multiple Consumer Groups​