Skip to main content

Google Cloud Logging

GCP Logging Target

Synopsis​

The Google Cloud Logging target forwards events to Google Cloud Logging (formerly Stackdriver Logging) with configurable severity levels, labels, and authentication methods.

Schema​

- name: <string>
  description: <string>
  type: gcplogging
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    project_id: <string>
    log_name: <string>
    authentication:
      method: <auto|manual|secret>
    credentials: <string>
    severity: <string>
    labels: <map>
    batch_size: <integer>
    timeout: <integer>
    max_retries: <integer>
    retry_delay: <integer>
    field_format: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration​

The following fields are used to define the target:

FieldRequiredDefaultDescription
nameYTarget name
descriptionN-Optional description
typeYMust be gcplogging
pipelinesN-Optional post-processor pipelines
statusNtrueEnable/disable the target

Connection​

FieldRequiredDefaultDescription
project_idY-Google Cloud project ID
log_nameY-Log name within the project

Authentication​

FieldRequiredDefaultDescription
authentication.methodNautoAuthentication method: auto, manual, secret
credentialsN*-JSON service account credentials (inline string or environment variable expansion)

* = Conditionally required when authentication.method is manual or secret.

Log Settings​

FieldRequiredDefaultDescription
severityNDEFAULTDefault log severity level. See Severity Levels below
labelsN-Map of custom labels to attach to all log entries

Batch Configuration​

FieldRequiredDefaultDescription
batch_sizeN1000Maximum log entries per batch
timeoutN30Request timeout in seconds
max_retriesN0Maximum retry attempts for failed sends (0 = no retries)
retry_delayN1Delay between retries in seconds

Processing​

FieldRequiredDefaultDescription
field_formatN-Data normalization format. See applicable Normalization section

Scheduling​

See Scheduling and Pool Behavior for interval and cron fields shared by all targets.

Debug Options​

FieldRequiredDefaultDescription
debug.statusNfalseEnable debug logging
debug.dont_send_logsNfalseProcess logs but don't send to target (testing)

Details​

Severity Levels​

Valid severity levels for Google Cloud Logging:

SeverityDescription
DEFAULTDefault severity (no specific level)
DEBUGDebug or trace information
INFOInformational messages
NOTICENormal but significant events
WARNINGWarning events
ERRORError events
CRITICALCritical events requiring immediate action
ALERTAlert requiring immediate notification
EMERGENCYEmergency requiring immediate response

Authentication Methods​

Auto (Default):

  • Uses Application Default Credentials (ADC)
  • Checks GOOGLE_APPLICATION_CREDENTIALS environment variable
  • Falls back to compute metadata service for GCE/GKE

Manual:

  • Inline JSON service account credentials
  • Credentials embedded directly in configuration

Secret:

  • Service account credentials from environment variable
  • More secure than inline credentials for production

IAM Permissions​

The service account requires the following IAM role:

IAM RoleRole IDPurpose
Logs Writerroles/logging.logWriterWrite log entries to Cloud Logging

Minimum permissions: logging.logEntries.create

Log Entry Structure​

Each log entry sent to Google Cloud Logging includes:

  • Timestamp: Event timestamp from pipeline
  • Severity: Configured or default severity level
  • Payload: Event message content
  • Labels: Custom labels for filtering and organization

Labels for Log Organization​

Labels enable efficient log filtering and organization:

  • Resource labels: Identify the source resource
  • User labels: Custom categorization
  • System labels: Automatic GCP-assigned labels

Labels are key-value pairs attached to every log entry.

Performance Considerations​

Batch Processing:

  • Events are buffered until batch_size is reached
  • Flush occurs on batch limit or service shutdown
  • Larger batches reduce API calls but increase latency

Retry Logic:

  • Failed sends are retried up to max_retries times
  • Exponential backoff between retries using retry_delay
  • Permanent failures are logged but not re-queued

Examples​

Basic Configuration​

Sending logs to Google Cloud Logging using auto authentication from Application Default Credentials...

targets:
  - name: gcp-logs
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: application-logs
      authentication:
        method: auto

With Service Account​

Using explicit service account credentials for authentication...

targets:
  - name: gcp-logs-manual
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: security-logs
      authentication:
        method: manual
        credentials: |
          {
            "type": "service_account",
            "project_id": "my-project-id",
            "private_key_id": "key-id",
            "private_key": "-----BEGIN PRIVATE KEY-----\n...",
            "client_email": "logging@my-project.iam.gserviceaccount.com",
            "client_id": "123456789",
            "auth_uri": "https://accounts.google.com/o/oauth2/auth",
            "token_uri": "https://oauth2.googleapis.com/token"
          }

With Secret Authentication​

Loading service account credentials from environment variable for better security...

targets:
  - name: gcp-logs-secret
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: audit-logs
      authentication:
        method: secret
        credentials: "${GCP_LOGGING_CREDENTIALS}"

With Severity and Labels​

Configuring specific severity level and custom labels for log organization...

targets:
  - name: gcp-logs-labeled
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: firewall-logs
      severity: WARNING
      labels:
        environment: production
        application: firewall
        datacenter: us-central1
      authentication:
        method: auto

High-Volume Configuration​

Optimizing for high-volume log ingestion with larger batches and retry configuration...

targets:
  - name: gcp-logs-high-volume
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: access-logs
      batch_size: 1000
      timeout: 30
      max_retries: 3
      retry_delay: 1
      authentication:
        method: secret
        credentials: "${GCP_LOGGING_CREDENTIALS}"

Error Severity​

Forwarding error logs with ERROR severity for immediate visibility...

targets:
  - name: gcp-error-logs
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: application-errors
      severity: ERROR
      labels:
        log_type: error
        alert: true
      authentication:
        method: auto

Debug Logs​

Sending debug-level logs for development and troubleshooting...

targets:
  - name: gcp-debug-logs
    type: gcplogging
    properties:
      project_id: my-project-id
      log_name: debug-logs
      severity: DEBUG
      labels:
        environment: development
        purpose: debugging
      authentication:
        method: auto

Production Configuration​

Production-ready configuration with performance tuning, retry logic, and comprehensive labels...

targets:
  - name: gcp-logs-production
    type: gcplogging
    properties:
      project_id: production-project
      log_name: production-logs
      severity: INFO
      batch_size: 1000
      timeout: 30
      max_retries: 3
      retry_delay: 2
      labels:
        environment: production
        application: datastream
        region: us-central1
        team: platform
        compliance: required
      authentication:
        method: secret
        credentials: "${GCP_LOGGING_CREDENTIALS}"