Proofpoint On Demand

Pull

Synopsis

Director polls the source system's API at configured intervals to retrieve log and telemetry data rather than receiving pushed events. The device establishes a persistent WebSocket connection to Proofpoint's On Demand (POD) log stream service and receives email security event data. Supports both message and maillog data types with secure token authentication.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: proofpoint
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    endpoint: <string>
    cluster: <string>
    token: <string>
    type: <string>
    secure: <boolean>
    workers: <numeric>
    reuse: <boolean>

Configuration

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be proofpoint
tags	N	-	Array of labels for categorization
pipelines	N	-	Array of preprocessing pipeline references
status	N	true	Enable/disable the device

Connection

Field	Required	Default	Description
endpoint	N	"wss://logstream.proofpoint.com:443/v1/stream"	Proofpoint WebSocket endpoint URL
cluster	Y	-	Proofpoint cluster identifier
token	Y	-	Authentication token for Proofpoint API (resolvable via ${ENV_VAR} or $secret{...})
type	N	"message"	Data type to consume (message or maillog)
secure	N	false	When true, the token is decrypted with the service shared key

Performance

Field	Required	Default	Description
workers	N	1	Number of worker processes
reuse	N	true	Enable multi-worker mode

Details

WebSocket Connection

The device establishes a persistent WebSocket connection to Proofpoint's On Demand log stream service. The connection URL includes query parameters for cluster ID and data type. Bearer token authentication is used in the Authorization header.

Data Types

Proofpoint supports two log data types:

• message: Email processing logs including connection metadata, envelope details, message headers, and filter module results
• maillog: Mail transfer agent logs including SMTP transactions and delivery status

Token Security

When secure is set to true, the token is encrypted in the YAML configuration using the service shared key. The token is decrypted at runtime before authentication. Alternatively, use ${ENV_VAR} or $secret{...} token resolution to keep the token out of the configuration file entirely.

Connection Management

The device handles WebSocket connection lifecycle including automatic reconnection on failure. Query parameters are encoded in the connection URL including cluster ID (cid) and data type (type).

Examples

Basic Configuration

Creating a basic Proofpoint On Demand consumer for email processing logs...	- id: 1 name: proofpoint-pod type: proofpoint properties: cluster: "your-cluster-id" token: "your-api-token" type: "message"
Device receives Proofpoint email events in real-time...	{ "guid": "abc123-def456", "ts": "2025-01-02T10:30:00.000000", "connection": { "ip": "192.0.2.10", "host": "mail.example.com" }, "envelope": { "from": "sender@example.com", "rcpts": ["user@company.com"] }, "msg": { "header": { "subject": "Quarterly Report", "from": ["sender@example.com"] } }, "filter": { "modules": { "spam": { "verdict": "notspam" } } } }

Secure Token Storage

Storing the token encrypted with the service shared key...

- id: 2 name: proofpoint-secure type: proofpoint properties: cluster: "your-cluster-id" token: "encrypted-token-string" type: "message" secure: true

note

The token must be encrypted using the service shared key before setting secure: true.

Maillog Collection

Collecting mail routing and delivery logs...	- id: 3 name: proofpoint-maillog type: proofpoint properties: cluster: "your-cluster-id" token: "your-api-token" type: "maillog"
Device receives mail transfer agent log events including delivery status...	{ "ts": "2025-01-02T10:30:00.000000", "data": "smtp;250 2.0.0 OK", "sm": { "from": "sender@example.com", "to": ["dest@company.com"] }, "pps": { "cid": "cluster01", "agent": "m0042.ppops.net" }, "id": "msg-20250102-001" }

High-Volume Processing

Multi-worker mode for high message rates...

- id: 4 name: proofpoint-highvolume type: proofpoint properties: cluster: "your-cluster-id" token: "your-api-token" type: "message" reuse: true workers: 4

Custom Endpoint

Connecting to a regional or custom endpoint...

- id: 5 name: proofpoint-custom type: proofpoint properties: endpoint: "wss://logstream-eu.proofpoint.com:443/v1/stream" cluster: "eu-cluster-id" token: "your-api-token" type: "message"

Pipeline Processing

Applying custom processing to email security events...

- id: 6 name: proofpoint-pipeline type: proofpoint pipelines: - email_enrichment - geo_lookup properties: cluster: "your-cluster-id" token: "your-api-token" type: "message"

note

Pipelines are processed sequentially and can modify or drop events before ingestion.