Codeless Connector Framework (CCF)

Pull

Synopsis

Director polls the source system's API at configured intervals to retrieve log and telemetry data rather than receiving pushed events. The poller is driven by a Microsoft Sentinel Codeless Connector Framework (CCF) PollingConfig.json file — either a single connector object or the as-published array of connectors, in which case each connector runs as its own poller. Secrets and environment-specific values referenced as {{placeholder}} tokens are supplied as separate device properties, keeping credentials out of the polling config. Credentials shared by every connector can be defined once at the device level via ccf_auth.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: ccf
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    ccf_config: <string>
    ccf_auth: <string>
    <placeholder_key>: <string>
    tls:
      insecure_skip_verify: <boolean>
    poll_interval: <numeric>
    ingestion_delay: <numeric>
    workers: <numeric>
    reuse: <boolean>

Configuration

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be ccf
tags	N	-	Optional tags
pipelines	N	-	Optional preprocessing pipelines
status	N	true	Enable/disable the device

CCF Config

Field	Required	Description
ccf_config	Y	Contents of the connector's PollingConfig.json — a single connector object, or a multi-connector array pasted verbatim from a Sentinel solution. In the array form every connector with a request.apiEndpoint runs as its own poller; sibling resources bundled in the same array (DCRs, workspaces) are skipped automatically. Any {{placeholder}} token is resolved from a sibling device property with the same name.
ccf_auth	N	A single CCF auth block (same JSON shape as a connector's properties.auth, {{placeholder}} tokens allowed) used as the default credentials for every connector in ccf_config that does not carry its own auth. A connector-level auth block always takes precedence, so verbatim Sentinel files — where auth is repeated per poller — import unchanged.
<placeholder_key>	N*	Any property whose name matches a {{placeholder}} token in ccf_config or ccf_auth. Typically used for secrets such as API keys and passwords.

* Required for each {{placeholder}} token present in ccf_config or ccf_auth.

note

A CCF connector package contains multiple files. Only the PollingConfig.json is used here — the connector definition, DCR, and workbook files are not required.

The PollingConfig.json top-level fields connectorDefinitionName, dataType, and dcrConfig are ignored. auth, request, paging, and response drive the poller; the resource name labels the connector in logs and poller state, and disabled turns individual connectors off.

[[concat(...)]] ARM template expressions found in some polling configs are not evaluated. Replace them with {{placeholder}} tokens and supply the resolved value as a sibling property.

TLS

Field	Required	Default	Description
tls.insecure_skip_verify	N	false	Skip TLS certificate verification (useful for dev/test environments or APIs with self-signed certificates)

Performance

These fields are device-level controls and are not part of the connector JSON.

Field	Required	Default	Description
poll_interval	N	10	Polling interval in seconds
ingestion_delay	N	0	Shifts every poll window's end back from now by this many seconds. Use for APIs that make events queryable only some time after they occur (ingestion lag), so late-arriving events are not skipped when the window advances. 0 keeps windows ending at the current time.
workers	N	1	Number of concurrent queue processing workers
reuse	N	true	Enable multi-worker mode

Details

Placeholder Resolution

Placeholder resolution runs on the raw JSON string before parsing. If any placeholder cannot be resolved the device fails to start and reports all missing keys in the error message. The same resolution applies to ccf_auth.

Multi-Connector Configs

Sentinel solutions often publish several pollers in one PollingConfig.json array (for example 1Password ships signinattempts, auditevents, and itemusages connectors in a single file). Pasting that array into ccf_config runs one poller per connector under the same device. Each connector polls independently with its own HTTP client, retry/rate-limit settings, and cursor state, keyed by the connector's resource name (falling back to its array position).

Individual connectors can be switched off without editing them out of the file by adding "disabled": true at the connector's top level or inside its properties. A config in which every connector is disabled is a configuration error.

Shared Authentication

When all connectors of a device authenticate the same way, define the credentials once in ccf_auth instead of repeating the auth block per connector. Connectors without their own auth inherit it; a connector-level auth block always wins.

For refreshable auth types (OAuth2, JwtToken, Session), the acquired token is cached per device and shared by all of its connectors: the first poller fetches the token, the others reuse it, and an expiry or invalidation triggered by any connector refreshes it for all of them. Static credentials (Basic, APIKey) involve no token exchange.

Poller State

Each connector's poll cursor (and persistent paging token or link, where the paging type uses one) is stored in Director's cluster-replicated persistent store. After a restart the poller resumes from its stored cursor — records already collected are not re-fetched, and events that occurred during the downtime are picked up by the catch-up window. On a first start (no stored cursor) the poller looks back one queryWindowInMin, not the full history.

Ingestion Delay

Poll windows normally end at the current time. Some APIs ingest events with a delay — an event with timestamp T only becomes queryable at T + lag — which can cause a window that has already advanced past T to miss it permanently. Setting ingestion_delay to a value larger than the source's worst-case lag (for example 120 for the 1Password Events API) shifts every window back by that amount so late-arriving events are always still inside a future window. The trade-off is that delivery lags real time by the configured delay.

Defaults

When optional numeric fields such as retryCount, timeoutInSeconds, queryWindowInMin, and pageSize are omitted from the connector JSON, the following defaults are applied:

Field	Default
queryWindowInMin	5
retryCount	3
timeoutInSeconds	20
pageSize	100

If these fields are present in the connector JSON they are used as-is. An explicit value of 0 or negative causes the device to silently fall back to the default value listed above.

rateLimitQPS and maxPages are exceptions: 0 is a valid value for both. A rateLimitQPS of 0 disables rate limiting; a maxPages of 0 allows unlimited pages.

Examples

Basic Auth (Confluence Audit Logs)

Polling Confluence audit logs using Basic authentication with offset-based pagination...

- id: 1 name: confluence_audit type: ccf properties: userid: "alice@example.com" apikey: "my-secret-key" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "Basic", "UserName": "{{userid}}", "Password": "{{apikey}}" }, "request": { "apiEndpoint": "https://myorg.atlassian.net/wiki/rest/api/audit", "httpMethod": "GET", "retryCount": 3, "timeoutInSeconds": 60, "queryTimeFormat": "UnixTimestampInMills", "startTimeAttributeName": "startDate", "endTimeAttributeName": "endDate", "headers": { "Accept": "application/json" } }, "paging": { "pagingType": "Offset", "offsetParaName": "start", "pageSizeParaName": "limit", "pageSize": 1000 }, "response": { "eventsJsonPaths": ["$.results"], "format": "json" } } }

OAuth2 (Auth0)

Polling Auth0 logs using OAuth2 client credentials with persistent token pagination across poll cycles...

- id: 2 name: auth0_logs type: ccf properties: ClientId: "my-client-id" ClientSecret: "my-client-secret" Domain: "https://myorg.auth0.com" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "OAuth2", "ClientId": "{{ClientId}}", "ClientSecret": "{{ClientSecret}}", "GrantType": "client_credentials", "TokenEndpoint": "{{Domain}}/oauth/token", "TokenEndpointQueryParameters": { "audience": "{{Domain}}/api/v2/" } }, "request": { "apiEndpoint": "{{Domain}}/api/v2/logs", "httpMethod": "GET", "headers": { "Accept": "application/json" } }, "Paging": { "pagingType": "PersistentToken", "nextPageParaName": "from", "nextPageTokenJsonPath": "$.[-1:].log_id", "PageSizeParameterName": "take", "PageSize": 100 }, "response": { "eventsJsonPaths": ["$"], "format": "json" } } }

API Key (Illumio)

Polling Illumio Insights using an API key header with a rate limit and extended query window...

- id: 3 name: illumio_insights type: ccf properties: apiToken: "my-api-token" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "APIKey", "ApiKey": "{{apiToken}}", "ApiKeyName": "X-api-secret" }, "request": { "apiEndpoint": "https://gw.console.illum.io/api/v1/insights-summary", "httpMethod": "GET", "rateLimitQPS": 10, "queryWindowInMin": 360, "retryCount": 3, "timeoutInSeconds": 60, "headers": { "Accept": "application/json" }, "queryParameters": { "persona": "threat_hunter", "report_interval": "daily" } }, "response": { "eventsJsonPaths": ["$"], "format": "json" } } }

Multi-Connector with Shared Auth (1Password)

Polling all three 1Password Events API connectors from one device: the verbatim Sentinel array runs one poller per connector, credentials are defined once in ccf_auth and inherited by every connector, the item-usage stream is switched off with disabled, and ingestion_delay compensates for the API's ingestion lag...

- id: 5 name: onepassword_events type: ccf properties: ApiToken: "my-events-api-bearer-token" BaseUrl: "https://events.1password.com" poll_interval: 10 ingestion_delay: 120 ccf_auth: | { "type": "APIKey", "ApiKey": "{{ApiToken}}", "ApiKeyName": "Authorization", "ApiKeyIdentifier": "Bearer" } ccf_config: | [ { "name": "OnePasswordSignInEvents", "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/signinattempts", "httpMethod": "Post", "queryWindowInMin": 5, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } }, { "name": "OnePasswordAuditEvents", "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/auditevents", "httpMethod": "Post", "queryWindowInMin": 5, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } }, { "name": "OnePasswordItemUsageEvents", "disabled": true, "kind": "RestApiPoller", "properties": { "request": { "apiEndpoint": "{{BaseUrl}}/api/v1/itemusages", "httpMethod": "Post", "queryWindowInMin": 1, "queryTimeFormat": "yyyy-MM-ddTHH:mm:ssZ", "queryParametersTemplate": "{\"limit\": 1000, \"start_time\": \"{_QueryWindowStartTime}\", \"end_time\": \"{_QueryWindowEndTime}\" }", "isPostPayloadJson": true }, "response": { "eventsJsonPaths": ["$.items"] }, "paging": { "pagingType": "NextPageToken", "nextPageParaName": "cursor", "nextPageTokenJsonPath": "$.cursor", "hasNextFlagJsonPath": "$.has_more" } } } ]

With Pipeline Processing

Attaching preprocessing pipelines to normalize timestamps and enrich fields before events reach the route...

- id: 4 name: ccf_with_pipeline type: ccf pipelines: - timestamp_normalizer - field_enrichment properties: apiKey: "my-key" ccf_config: | { "kind": "RestApiPoller", "properties": { "auth": { "type": "APIKey", "ApiKey": "{{apiKey}}", "ApiKeyName": "Authorization", "ApiKeyIdentifier": "Bearer" }, "request": { "apiEndpoint": "https://api.example.com/v1/events", "httpMethod": "GET" }, "response": { "eventsJsonPaths": ["$.events"], "format": "json" } } } poll_interval: 30 workers: 4