Check Schema

Flow Control

Synopsis

Validates event data against schemas defined in Parquet or Avro format — including ASIM, OCSF, UDM, and custom schemas — detecting schema drift by identifying missing fields, extra fields, and type mismatches.

Custom schemas can be created and managed in the Library.

Schema

- check_schema:
    schema: <string>
    target_field: <string>
    schema_type: <string>
    requirement_filter: <string>
    check_mode: <string>
    validate_recommended: <boolean>
    validate_optional: <boolean>
    disabled: <boolean>
    on_missing: <processor[]>
    on_extra: <processor[]>
    on_type_mismatch: <processor[]>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
schema	Y		Schema name to validate against (e.g., ASimNetworkSessionLogs, ASimAuthenticationEventLogs, ocsf_network_activity)
target_field	Y		Field name to store validation results
schema_type	N	parquet	Schema format: parquet or avro
requirement_filter	N	all	Filter for schema field requirements
check_mode	N	missing	Validation mode: missing, extra, or both
validate_recommended	N	false	Include recommended fields in validity check
validate_optional	N	false	Include optional fields in validity check
disabled	N	false	Disable the processor without removing it from the pipeline
on_missing	N		Processors to execute when missing fields are detected
on_extra	N		Processors to execute when extra fields are detected
on_type_mismatch	N		Processors to execute when type mismatches are detected
description	N		Explanatory note
if	N		Condition to run
ignore_failure	N	false	See Handling Failures
on_failure	N		See Handling Failures
on_success	N		See Handling Success
tag	N		Identifier

Details

The check_schema processor validates events against schema definitions, detecting schema drift that occurs when vendor log formats change unexpectedly. Schemas are loaded as Parquet or Avro definitions (selected by schema_type), which covers ASIM, OCSF, UDM, and custom schemas placed in the package or user schema directories. The JSON definition syntax for each format is documented under Avro and Parquet.

Validation Result Structure: Results are written to the target_field as a structured object:

{
  "is_valid": false,
  "missing_required_fields": ["EventSchema", "EventVendor"],
  "missing_recommended_fields": ["DvcAction", "EventSeverity"],
  "missing_optional_fields": ["SrcNatIpAddr"],
  "extra_fields": ["CustomField1"],
  "type_mismatches": [
    {
      "field": "EventCount",
      "expected_type": "INT32",
      "actual_type": "STRING"
    }
  ]
}

Validation Levels:

• Required fields: Always checked when check_mode includes missing. Missing required fields make is_valid false.
• Recommended fields: Reported but don't affect validity unless validate_recommended: true.
• Optional fields: Reported but don't affect validity unless validate_optional: true.
• Extra fields: Detected when check_mode includes extra. Never affect validity (informational only).
• Type mismatches: Checked for present fields. Impact follows the field's requirement level.

Check Modes:

• missing: Only detect missing fields
• extra: Only detect extra fields not in schema
• both: Detect both missing and extra fields

Metadata Fields: Fields prefixed with @ (like @timestamp, @metadata) are automatically ignored during validation.

For integration patterns, see Schema Drift Detection and Multi-Tier Pipelines.

Examples

Basic ASIM Validation

Validating network session event against ASIM schema...

{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventSchema": "NetworkSession", "EventSchemaVersion": "0.2.6", "EventStartTime": "2024-01-01T12:00:00Z", "EventEndTime": "2024-01-01T12:00:01Z", "Dvc": "firewall01", "EventType": "NetworkSession", "EventResult": "Success", "EventProduct": "Firewall", "EventVendor": "Microsoft" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both"

All required fields present, validation passes...

{ "schema_check": { "is_valid": true, "missing_required_fields": [], "missing_recommended_fields": ["DvcAction", "EventSeverity"], "missing_optional_fields": [], "extra_fields": [], "type_mismatches": [] } }

Detecting Missing Fields

Event missing required fields triggers validation failure...	{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventType": "NetworkSession" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "missing"
Missing required fields listed in result...	{ "schema_check": { "is_valid": false, "missing_required_fields": [ "EventSchema", "EventSchemaVersion", "EventEndTime", "Dvc", "EventResult", "EventProduct", "EventVendor" ], "type_mismatches": [] } }

Detecting Extra Fields

Detecting fields not defined in schema...	{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "ExtraField1": "unexpected", "CustomData": {"key": "value"}, "UnknownMetric": 999.99 } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "extra"
Extra fields detected but don't affect validity...	{ "schema_check": { "is_valid": true, "extra_fields": [ "ExtraField1", "CustomData", "UnknownMetric" ] } }

Conditional Processing Chains

Triggering alerts when schema drift is detected...	- check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both" on_missing: - set: field: "drift_type" value: "missing_fields" - set: field: "alert_required" value: true on_extra: - set: field: "drift_type" value: "extra_fields" on_type_mismatch: - set: field: "drift_type" value: "type_mismatch"
Conditional chains execute based on drift type...	{ "drift_type": "missing_fields", "alert_required": true, "schema_check": { "is_valid": false, "missing_required_fields": ["EventSchema"] } }

Strict Validation

Including recommended fields in validity check...

{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventSchema": "NetworkSession", "EventSchemaVersion": "0.2.6", "EventStartTime": "2024-01-01T12:00:00Z", "EventEndTime": "2024-01-01T12:00:01Z", "Dvc": "firewall01", "EventType": "NetworkSession", "EventResult": "Success", "EventProduct": "Firewall", "EventVendor": "Microsoft" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both" validate_recommended: true

Missing recommended fields now affect validity...

{ "schema_check": { "is_valid": false, "missing_required_fields": [], "missing_recommended_fields": ["DvcAction", "EventSeverity"] } }