IBM Cloud Logs

IBM Cloud Target

Synopsis

The IBM Cloud Logs target forwards telemetry events to IBM Cloud Logs using the Logs API singles endpoint. Events are batched and sent with configurable application context, subsystem categorization, and severity levels.

Schema

- name: <string>
  description: <string>
  type: ibmcloudlogs
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    instance_id: <string>
    region: <string>
    authentication_method: <string>
    iam_token: <string>
    iam_token_secret: <string>
    application_name: <string>
    subsystem_name: <string>
    computer_name: <string>
    default_severity: <integer>
    use_timestamp: <boolean>
    use_hires_timestamp: <boolean>
    batch_size: <integer>
    timeout: <integer>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be ibmcloudlogs
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

IBM Cloud Logs Connection

Field	Required	Default	Description
instance_id	Y	-	IBM Cloud Logs instance ID
region	Y	-	IBM Cloud region. See Valid Regions below
authentication_method	N	token	Authentication method: token, secret
iam_token	Y*	-	IBM Cloud IAM Bearer token
iam_token_secret	Y*	-	Environment variable name containing IAM token

* = Conditionally required. iam_token when authentication_method: token; iam_token_secret when authentication_method: secret.

Log Configuration

Field	Required	Default	Description
application_name	Y	-	Application name for log categorization
subsystem_name	Y	-	Subsystem name for log categorization
computer_name	N	-	Computer/host name for log source identification
default_severity	N	1	Default severity level (1-6). 1 = Debug. See Severity Levels below
use_timestamp	N	false	Use event timestamp instead of current time
use_hires_timestamp	N	false	Use high-resolution (nanosecond) timestamp

Batch Configuration

Field	Required	Default	Description
batch_size	N	1000	Maximum events per batch
timeout	N	30	Request timeout in seconds

Processing

Field	Required	Default	Description
field_format	N	-	Data normalization format. See applicable Normalization section

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Valid Regions

IBM Cloud Logs is available in the following regions:

Region Code	Region Name
us-south	US South (Dallas)
us-east	US East (Washington DC)
eu-gb	United Kingdom (London)
eu-de	Germany (Frankfurt)
eu-es	Spain (Madrid)
jp-tok	Japan (Tokyo)
jp-osa	Japan (Osaka)
au-syd	Australia (Sydney)
ca-tor	Canada (Toronto)
br-sao	Brazil (São Paulo)

Severity Levels

IBM Cloud Logs uses numeric severity levels:

Level	Name	Description
1	Debug	Debug or trace information
2	Verbose	Verbose informational messages
3	Info	Informational messages
4	Warn	Warning events
5	Error	Error events
6	Critical	Critical events requiring immediate action

Severity Handling:

• Events with severity field use that value if valid (1-6)
• Events without severity field use default_severity
• Invalid severity values default to default_severity

Authentication Methods

Token (Default):

• Use iam_token field with Bearer token directly in configuration
• Token automatically prefixed with "Bearer " if not already present
• Simpler for development and testing

Secret:

• Use iam_token_secret field with environment variable name
• More secure for production deployments
• Environment variable must be set before starting DataStream

IAM Token Requirements

IBM Cloud Logs requires a valid IAM Bearer token for authentication. Generate tokens using IBM Cloud CLI or API. Tokens expire and must be refreshed periodically.

IAM Permissions

The IAM identity (service ID or user) used to generate the Bearer token requires the following IBM Cloud IAM role:

IBM IAM Role	Service	IAM Action	Purpose
Sender	IBM Cloud Logs	logs.data.send	Ingest logs via the /logs/v1/singles REST endpoint

The Sender role (underlying IAM action: logs.data.send) is the minimum required role. No read, management, or administrative roles are needed since the target only performs log ingestion (write-only).

note

IBM Cloud Logs accepts up to 2 MB per request, which is approximately 3,000 medium-sized log entries.

Endpoint Construction

Automatic Endpoint Building:

• Endpoint format: https://{instance_id}.ingress.{region}.logs.cloud.ibm.com/logs/v1/singles
• Example: https://abc123.ingress.us-south.logs.cloud.ibm.com/logs/v1/singles
• Instance ID and region are validated during configuration

Event Structure

JSON Parsing:

• Events with valid JSON message are parsed and sent as structured data
• Non-JSON messages are sent as {"text": "message"} objects
• Supports nested JSON structures and complex data types

Application Context:

• applicationName: Required field for log categorization
• subsystemName: Required field for subsystem identification
• computerName: Optional field for source host identification

Timestamp Handling:

• Default: Current timestamp when event is sent
• use_timestamp: true: Use event's original timestamp
• use_hires_timestamp: true: Include nanosecond precision

Performance Considerations

Batch Processing:

• Events are buffered until batch_size is reached
• Flush occurs on batch limit or during finalization
• Larger batches reduce API calls but increase latency
• Maximum recommended batch size: 1000 events

Connection Pooling:

• HTTP client maintains connection pool
• Maximum 100 idle connections total
• Maximum 10 idle connections per host
• 90-second idle connection timeout

Retry Logic:

• Failed sends are retried based on sender configuration
• HTTP errors include response body for troubleshooting
• Check IBM Cloud Logs service status for API issues

Batch Size Limits

IBM Cloud Logs API has limits on batch size and request payload. Configure batch_size appropriately for your event size to avoid API rejections.

Error Handling

Authentication Failures:

• HTTP 401: Invalid or expired IAM token
• Refresh IAM token and restart DataStream
• Check token format (must include "Bearer " prefix)

API Errors:

• HTTP 400: Malformed request or invalid event structure
• HTTP 500: IBM Cloud Logs service error
• Error responses include detailed message for troubleshooting

Validation Errors:

• Invalid region codes are rejected during configuration validation
• Invalid severity levels default to default_severity
• Missing required fields (instance_id, application_name, subsystem_name) prevent target initialization

Examples

Basic Configuration

Sending logs to IBM Cloud Logs using token authentication...

targets: - name: ibm-logs type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: datastream subsystem_name: telemetry

With Secret Authentication

Using environment variable for secure IAM token storage...

targets: - name: ibm-logs-secure type: ibmcloudlogs properties: instance_id: xyz789abc123 region: eu-gb authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: security subsystem_name: audit computer_name: production-server

With Custom Severity

Setting default severity to Warning for important events...

targets: - name: ibm-logs-warnings type: ibmcloudlogs properties: instance_id: abc123def456 region: us-east iam_token: "${IBM_IAM_TOKEN}" application_name: monitoring subsystem_name: alerts default_severity: 4 use_timestamp: true

High-Volume Configuration

Optimizing for high-volume log ingestion with larger batches...

targets: - name: ibm-logs-high-volume type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: streaming subsystem_name: events batch_size: 1000 timeout: 30 use_timestamp: true use_hires_timestamp: true

Multi-Region Configuration

Sending logs to different IBM Cloud regions for geographic distribution...

targets: - name: ibm-logs-us type: ibmcloudlogs properties: instance_id: us123abc456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: us-region - name: ibm-logs-eu type: ibmcloudlogs properties: instance_id: eu456def789 region: eu-de iam_token: "${IBM_IAM_TOKEN}" application_name: global-app subsystem_name: eu-region

With Normalization

Applying ECS normalization before sending to IBM Cloud Logs...

targets: - name: ibm-logs-normalized type: ibmcloudlogs properties: instance_id: abc123def456 region: us-south iam_token: "${IBM_IAM_TOKEN}" application_name: security subsystem_name: normalized field_format: ECS default_severity: 3

Production Configuration

Production-ready configuration with secret authentication, batch optimization, and high-resolution timestamps...

targets: - name: ibm-logs-production type: ibmcloudlogs properties: instance_id: prod123abc456 region: us-south authentication_method: secret iam_token_secret: IBM_CLOUD_LOGS_TOKEN application_name: production-datastream subsystem_name: telemetry-processing computer_name: datastream-director-01 default_severity: 3 use_timestamp: true use_hires_timestamp: true batch_size: 1000 timeout: 30 field_format: ASIM