Avro

Apache Avro is a data serialization system that provides rich data structures and a compact, fast, binary data format. Originally developed within the Apache Hadoop ecosystem, Avro is designed for schema evolution and language-neutral data exchange.

Binary Layout

Section	Internal Name	Description	Possible Values / Format
File Header	magic	4-byte magic number identifying Avro files	ASCII: Obj followed by 1 byte (hex: 4F 62 6A 01)
	meta	Metadata map storing key-value pairs (e.g., schema, codec)	Map of string keys to byte values (e.g., "avro.schema" → JSON schema string)
	sync	16-byte random sync marker used between blocks	16 random bytes (unique per file)
Data Block	blockCount	Number of records in the block	Long (variable-length zigzag encoding)
	blockSize	Size in bytes of the serialized records (after compression, if any)	Long
	blockData	Serialized records (optionally compressed)	Binary-encoded data per schema
	sync	Sync marker repeated after each block	Same 16-byte value as in header

Schema Types (Stored in Metadata)

Type	Internal Name	Description	Example / Format
Primitive	null, boolean, int, long, float, double, bytes, string	Basic types	`"type": "string"
Record	record	Named collection of fields	{ "type": "record", "name": "Person", "fields": [...] }
Enum	enum	Named set of symbols	{ "type": "enum", "name": "Suit", "symbols": ["SPADES", "HEARTS"] }
Array	array	Ordered list of items	{ "type": "array", "items": "string" }
Map	map	Key-value pairs with string keys	{ "type": "map", "values": "int" }
Union	JSON array	Multiple possible types	[ "null", "string" ]
Fixed	fixed	Fixed-size byte array	{ "type": "fixed", "name": "md5", "size": 16 }

JSON Schema Definition

Avro schemas are themselves JSON. A schema authored in this format drives both .avro serialization and VirtualMetric's per-field type coercion before writing. Any schema conforming to the Apache Avro 1.11 specification serializes correctly; the tables below describe which parts VirtualMetric actively understands for validation.

A schema is a single JSON object representing an Avro record:

{
  "type": "record",
  "name": "MyEvent",
  "namespace": "org.example.events",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "created", "type": ["null", { "type": "long", "logicalType": "timestamp-millis" }], "default": null }
  ]
}

Property	Required	Description
type	Y	Must be record at the top level
name	Y	Record name (used by Avro for type identity)
namespace	N	Dotted namespace, e.g. org.example.events
fields	Y	Array of field definitions

Field Definition

Each entry in fields is an object:

Property	Required	Description
name	Y	Field name; used as the key in the message produced by VirtualMetric
type	Y	Field type — a string, a JSON object, or an array (union)
default	N	Default value when the field is absent; honored by the encoder

Other Avro field properties (doc, order, aliases) are accepted by the encoder and ignored by VirtualMetric.

Primitive Types

A primitive type is declared as a string, e.g. { "name": "user_id", "type": "string" }. Each maps to a VirtualMetric internal type:

Avro type	Internal type	Description
string	STRING	UTF-8 string
boolean	BOOLEAN	Boolean
int	INT32	32-bit signed integer
long	INT64	64-bit signed integer
float	FLOAT	32-bit floating point
double	DOUBLE	64-bit floating point
bytes	BYTE_ARRAY	Variable-length byte string
null	—	Used inside unions only

Nullable Fields

Declare an optional field as a union with null:

{ "name": "email", "type": ["null", "string"], "default": null }

VirtualMetric reads the union, skips the null branch, and treats the field as the non-null type. Member order does not matter for the validator, though encoders typically expect null first when default is null.

Logical Types

A logical type annotates a primitive with extra semantics, declared as a nested object:

{ "name": "created_at", "type": { "type": "long", "logicalType": "timestamp-millis" } }

VirtualMetric recognizes these logicalType values:

logicalType	Backing type	Internal type	Description
timestamp-millis	long	DATETIME	Milliseconds since the Unix epoch
timestamp-micros	long	DATETIME	Microseconds since the Unix epoch
date	int	DATETIME	Days since the Unix epoch
decimal	bytes / fixed	DECIMAL	Fixed-point decimal

Other Avro logical types (uuid, time-millis, time-micros, local-timestamp-millis, duration) are accepted by the encoder but not coerced — they pass through as the underlying primitive. A DATETIME field accepts RFC 3339 timestamps, 2006-01-02 15:04:05 (UTC), or a numeric epoch (seconds, milliseconds, microseconds, or nanoseconds, auto-detected).

Validation Scope

The schema is consumed by two layers. The validator extracts the top-level field-name-to-type mapping and coerces each top-level field into its declared type before encoding (so a string "42" on an int field becomes a real integer). The encoder consumes the full Avro 1.11 specification. Automatic coercion applies only to top-level scalars, top-level nullable unions of primitives, and the four logical types above; nested record, array, map, enum, and fixed types serialize correctly but their values pass through without per-element coercion. Keep fields you want coerced at the top level.

Reference Example

{
  "type": "record",
  "name": "AuditLog",
  "namespace": "org.example.audit",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "received_at", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    { "name": "actor_id", "type": ["null", "string"], "default": null },
    { "name": "severity", "type": "int" },
    { "name": "success", "type": "boolean" },
    { "name": "amount", "type": ["null", { "type": "bytes", "logicalType": "decimal", "precision": 18, "scale": 4 }], "default": null },
    { "name": "tags", "type": ["null", { "type": "array", "items": "string" }], "default": null }
  ]
}

Schemas in this format can be registered as reusable Library entries and referenced by name from the Enforce Schema and Check Schema processors, or supplied inline.

Metadata Keys (in meta)

Key	Description	Example Value
avro.schema	JSON-encoded schema	JSON string defining the schema
avro.codec	Compression codec used (optional)	"null" (default), "deflate", "snappy", "bzip2", "xz", "zstandard"

Compression Codecs

Required Codecs

Codec	Description	Best For
null	No compression applied	Small files or testing
deflate	Standard ZIP compression (RFC 1951)	General-purpose compression

Optional Codecs

Codec	Description	Best For
snappy	Fast compression/decompression with CRC32 checksum	Real-time streaming applications
bzip2	High compression ratio	Storage-constrained environments
xz	XZ compression library	Maximum compression efficiency
zstandard	Facebook's Zstandard compression with configurable levels	Best balance of speed and compression ratio