Check Schema

Flow Control

Synopsis

Validates event data against ASIM (Advanced Security Information Model) or OCSF (Open Cybersecurity Schema Framework) schema definitions, detecting schema drift by identifying missing fields, extra fields, and type mismatches.

Schema

- check_schema:
    schema: <string>
    target_field: <string>
    check_mode: <string>
    validate_recommended: <boolean>
    validate_optional: <boolean>
    on_missing: <processor[]>
    on_extra: <processor[]>
    on_type_mismatch: <processor[]>
    description: <text>
    if: <script>
    ignore_failure: <boolean>
    on_failure: <processor[]>
    on_success: <processor[]>
    tag: <string>

Configuration

Field	Required	Default	Description
schema	Y	-	Schema name to validate against (e.g., ASimNetworkSessionLogs, ASimAuthenticationEventLogs)
target_field	Y	-	Field name to store validation results
check_mode	N	both	Validation mode: missing, extra, or both
validate_recommended	N	false	Include recommended fields in validity check
validate_optional	N	false	Include optional fields in validity check
on_missing	N	-	Processors to execute when missing fields are detected
on_extra	N	-	Processors to execute when extra fields are detected
on_type_mismatch	N	-	Processors to execute when type mismatches are detected
description	N	-	Explanatory note
if	N	-	Condition to run
ignore_failure	N	false	See Handling Failures
on_failure	N	-	See Handling Failures
on_success	N	-	See Handling Success
tag	N	-	Identifier

Details

The check_schema processor validates events against official schema definitions, detecting schema drift that occurs when vendor log formats change unexpectedly. It supports Microsoft Sentinel's ASIM schemas and OCSF schemas.

Validation Result Structure: Results are written to the target_field as a structured object:

{
  "is_valid": false,
  "missing_required_fields": ["EventSchema", "EventVendor"],
  "missing_recommended_fields": ["DvcAction", "EventSeverity"],
  "missing_optional_fields": ["SrcNatIpAddr"],
  "extra_fields": ["CustomField1"],
  "type_mismatches": [
    {
      "field": "EventCount",
      "expected_type": "INT32",
      "actual_type": "STRING"
    }
  ]
}

Validation Levels:

• Required fields: Always checked when check_mode includes missing. Missing required fields make is_valid false.
• Recommended fields: Reported but don't affect validity unless validate_recommended: true.
• Optional fields: Reported but don't affect validity unless validate_optional: true.
• Extra fields: Detected when check_mode includes extra. Never affect validity (informational only).
• Type mismatches: Checked for present fields. Impact follows the field's requirement level.

Check Modes:

• missing: Only detect missing fields
• extra: Only detect extra fields not in schema
• both: Detect both missing and extra fields

Metadata Fields: Fields prefixed with @ (like @timestamp, @metadata) are automatically ignored during validation.

For integration patterns, see Schema Drift Detection and Multi-Tier Pipelines.

Examples

Basic ASIM Validation

Validating network session event against ASIM schema...

{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventSchema": "NetworkSession", "EventSchemaVersion": "0.2.6", "EventStartTime": "2024-01-01T12:00:00Z", "EventEndTime": "2024-01-01T12:00:01Z", "Dvc": "firewall01", "EventType": "NetworkSession", "EventResult": "Success", "EventProduct": "Firewall", "EventVendor": "Microsoft" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both"

All required fields present, validation passes...

{ "schema_check": { "is_valid": true, "missing_required_fields": [], "missing_recommended_fields": ["DvcAction", "EventSeverity"], "missing_optional_fields": [], "extra_fields": [], "type_mismatches": [] } }

Detecting Missing Fields

Event missing required fields triggers validation failure...	{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventType": "NetworkSession" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "missing"
Missing required fields listed in result...	{ "schema_check": { "is_valid": false, "missing_required_fields": [ "EventSchema", "EventSchemaVersion", "EventEndTime", "Dvc", "EventResult", "EventProduct", "EventVendor" ], "type_mismatches": [] } }

Detecting Extra Fields

Detecting fields not defined in schema...	{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "ExtraField1": "unexpected", "CustomData": {"key": "value"}, "UnknownMetric": 999.99 } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "extra"
Extra fields detected but don't affect validity...	{ "schema_check": { "is_valid": true, "extra_fields": [ "ExtraField1", "CustomData", "UnknownMetric" ] } }

Conditional Processing Chains

Triggering alerts when schema drift is detected...	- check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both" on_missing: - set: field: "drift_type" value: "missing_fields" - set: field: "alert_required" value: true on_extra: - set: field: "drift_type" value: "extra_fields" on_type_mismatch: - set: field: "drift_type" value: "type_mismatch"
Conditional chains execute based on drift type...	{ "drift_type": "missing_fields", "alert_required": true, "schema_check": { "is_valid": false, "missing_required_fields": ["EventSchema"] } }

Strict Validation

Including recommended fields in validity check...

{ "TimeGenerated": "2024-01-01T12:00:00Z", "EventCount": 1, "EventSchema": "NetworkSession", "EventSchemaVersion": "0.2.6", "EventStartTime": "2024-01-01T12:00:00Z", "EventEndTime": "2024-01-01T12:00:01Z", "Dvc": "firewall01", "EventType": "NetworkSession", "EventResult": "Success", "EventProduct": "Firewall", "EventVendor": "Microsoft" } - check_schema: schema: "ASimNetworkSessionLogs" target_field: "schema_check" check_mode: "both" validate_recommended: true

Missing recommended fields now affect validity...

{ "schema_check": { "is_valid": false, "missing_required_fields": [], "missing_recommended_fields": ["DvcAction", "EventSeverity"] } }