IBM Event Streams

IBM Cloud Message Queue

Synopsis

The IBM Event Streams target writes log messages to IBM's managed Kafka service on IBM Cloud with full Kafka API compatibility. IBM Event Streams provides enterprise-grade messaging with automatic scaling, high availability, and IBM Cloud integration. Configuration follows Apache Kafka patterns with IBM Cloud-specific authentication.

Schema

- name: <string>
  description: <string>
  type: ibmeventstreams
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    address: <string>
    port: <integer>
    client_id: <string>
    topic: <string>
    algorithm: <string>
    username: <string>
    password: <string>
    compression: <string>
    compression_level: <string>
    acknowledgments: <string>
    allow_auto_topic_creation: <boolean>
    disable_idempotent_write: <boolean>
    max_bytes: <integer>
    max_events: <integer>
    batch_mode: <string>
    batch_separator: <string>
    field_format: <string>
    tls:
      status: <boolean>
      insecure_skip_verify: <boolean>
      min_tls_version: <string>
      max_tls_version: <string>
      cert_name: <string>
      key_name: <string>
      passphrase: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be ibmeventstreams
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

IBM Event Streams Connection

Field	Required	Default	Description
address	Y	-	IBM Event Streams broker address (from service credentials). Supports a single address or a comma-separated list for multiple seed brokers. Each entry may include a port; if omitted, the port field value is appended.
port	N	9093	IBM Event Streams broker port (SASL_SSL)
client_id	N	-	Client identifier for connection tracking
topic	Y	-	Kafka topic name for message delivery

Authentication

Field	Required	Default	Description
algorithm	N	scram-sha-512	Authentication mechanism (IBM Event Streams uses SASL/SCRAM-SHA-512)
username	Y	-	IBM Event Streams username (from service credentials)
password	Y	-	IBM Event Streams password (from service credentials)

Producer Settings

Field	Required	Default	Description
compression	N	"none"	Message compression: none, gzip, snappy, lz4, zstd
compression_level	N	-	Compression level (algorithm-specific)
acknowledgments	N	"leader"	Acknowledgment level: none, leader, all
allow_auto_topic_creation	N	false	Allow automatic topic creation if topic doesn't exist
disable_idempotent_write	N	false	Disable idempotent producer

Batch Configuration

Field	Required	Default	Description
max_bytes	N	1048576	Maximum batch size in bytes (0 = unlimited, max: 104857600)
max_events	N	1000	Maximum number of events per batch
batch_mode	N	individual	Output format: individual (one message per event), json_array (array of JSON objects), jsonl (JSON Lines with separator)
batch_separator	N	,	Separator between messages when using jsonl batch mode
field_format	N	-	Data normalization format. See applicable Normalization section

TLS Configuration

Field	Required	Default	Description
tls.status	N	true	Enable TLS/SSL encryption (required for IBM Cloud connections)
tls.insecure_skip_verify	N	false	Skip TLS certificate verification (not recommended)
tls.min_tls_version	N	"tls1.2"	Minimum TLS version: tls1.0, tls1.1, tls1.2, tls1.3
tls.max_tls_version	N	"tls1.2"	Maximum TLS version: tls1.0, tls1.1, tls1.2, tls1.3
tls.cert_name	N	"cert.pem"	Client certificate file name for mTLS
tls.key_name	N	"key.pem"	Private key file name for mTLS
tls.passphrase	N	-	Passphrase for encrypted private key

Scheduling

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Authentication

SASL/SCRAM-SHA-512:

• IBM Event Streams uses SASL/SCRAM-SHA-512 authentication
• Set algorithm: scram-sha-512 (default for IBM Event Streams)
• Username and password from IBM Cloud service credentials
• TLS encryption required for authentication

Service Credentials:

• Obtain credentials from IBM Cloud console
• Navigate to Event Streams instance � Service Credentials
• Create new credentials with appropriate permissions
• Extract kafka_brokers_sasl, user, and password values

Service Credential Format

IBM Event Streams service credentials include:

• kafka_brokers_sasl: Array of broker addresses
• user: SASL username for authentication
• password: SASL password for authentication
• Use any broker address from the array

Connection Configuration

Broker Addresses:

• IBM Event Streams provides multiple broker endpoints
• Format: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com:9093
• Use any broker from service credentials
• Default port: 9093 (SASL_SSL)

TLS Requirements:

• TLS encryption mandatory for IBM Cloud connections
• Set tls.status: true (default)
• IBM Event Streams uses valid certificates
• No need for custom CA certificates

Topic Management

Topic Creation:

• Pre-create topics in IBM Cloud console
• Configure allow_auto_topic_creation: true for automatic creation (not recommended)
• Topic configuration managed through IBM Cloud UI or CLI

Topic Permissions:

• Service credentials grant topic-level permissions
• Writer role required for producing messages
• Configure permissions in IBM Cloud console

Performance Optimization

Batch Configuration:

• Larger batches improve throughput
• Balance batch size against latency requirements
• IBM Event Streams handles high-throughput workloads

Compression:

• Enable compression to reduce bandwidth costs
• Recommended: snappy (fast) or zstd (high compression)
• Compression reduces network transfer and storage

Connection Pooling:

• Maintains persistent connection to IBM Event Streams
• Automatic reconnection on connection loss
• Configurable client ID for connection tracking

IBM Cloud Pricing

IBM Event Streams charges based on throughput and storage. Enable compression and tune batch sizes to optimize costs.

Kafka API Compatibility

DataStream uses the standard Kafka Producer API with support for:

• Idempotent writes
• Batch compression
• SASL authentication
• TLS encryption

Examples

Basic Configuration

Sending logs to IBM Event Streams using SASL/SCRAM authentication...

targets: - name: ibm-event-streams type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: application-logs algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

With Compression

Enabling Snappy compression for bandwidth efficiency...

targets: - name: ibm-event-streams-compressed type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: telemetry-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: snappy tls: status: true

High-Volume Configuration

Optimizing for high-volume ingestion with larger batches and compression...

targets: - name: ibm-event-streams-high-volume type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: metrics-stream algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: zstd compression_level: "3" max_events: 1000 max_bytes: 1048576 acknowledgments: all tls: status: true

With Client Identification

Using client ID for connection tracking and monitoring...

targets: - name: ibm-event-streams-identified type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 client_id: datastream-director-01 topic: security-logs algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

Multi-Topic Publishing

Publishing different event types to separate topics...

targets: - name: ibm-es-security type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: security-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true - name: ibm-es-application type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: application-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" tls: status: true

With Normalization

Applying ECS normalization before sending to IBM Event Streams...

targets: - name: ibm-event-streams-normalized type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 topic: normalized-events algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" field_format: ECS compression: zstd tls: status: true

Production Configuration

Production-ready IBM Event Streams configuration with compression, batching, TLS, and full acknowledgments...

targets: - name: ibm-event-streams-production type: ibmeventstreams properties: address: broker-0.kafka.svc01.us-south.eventstreams.cloud.ibm.com port: 9093 client_id: datastream-production-01 topic: production-telemetry algorithm: scram-sha-512 username: "${IBM_ES_USERNAME}" password: "${IBM_ES_PASSWORD}" compression: zstd compression_level: "3" acknowledgments: all max_events: 1000 max_bytes: 1048576 field_format: ASIM tls: status: true min_tls_version: tls1.2