Parquet

Apache Parquet is a column-oriented binary storage format optimized for analytical workloads. Originally developed within the Apache Hadoop ecosystem, Parquet provides efficient compression and encoding schemes for large-scale data processing.

Binary Layout

Section	Internal Name	Description	Possible Values / Format
File Header	magic	4-byte magic number identifying Parquet files	ASCII: PAR1 (hex: 50 41 52 31)
Row Group	row_group_metadata	Metadata for each row group	Contains column chunk metadata and statistics
	column_chunk	Data for each column in the row group	Compressed and encoded column data
File Footer	metadata	File-level metadata including schema and row groups	Thrift-encoded metadata structure
	metadata_length	Length of metadata section	4-byte little-endian integer
	magic	Footer magic number	ASCII: PAR1 (hex: 50 41 52 31)

Column Storage Example

Row-based storage (traditional):

id,name,last_name,age
1,John,Buck,35
2,Jane,Doe,27
3,Joe,Dane,42

Column-based Storage (Parquet):

id: [1, 2, 3]
name: [John, Jane, Joe]
last_name: [Buck, Doe, Dane]
age: [35, 27, 42]

JSON Schema Definition

A Parquet schema is declared in VirtualMetric as a single JSON object whose keys are field names and whose values are field definitions. Definitions nest via the fields property to describe records and lists of records. The schema is parsed into a Parquet schema tree and used by the writer when producing .parquet files.

{
  "user_id": { "type": "STRING", "requirement": "required" },
  "score": { "type": "INT", "bitWidth": 32, "signed": true },
  "metadata": {
    "type": "GROUP",
    "fields": {
      "region": { "type": "STRING" },
      "tier": { "type": "STRING" }
    }
  }
}

The top level is always an object — there is no wrapping fields, name, or schema key.

Field Definition Properties

Property	Type	Applies to	Description
type	string	every field	Logical type (see below); required for every leaf field
fields	object	groups, complex lists	Nested fields; presence promotes the field to a record/group
requirement	string	every field	required, optional, or recommended. Default: optional
repeated	boolean	every field	If true, the field is repeated (array-like); equivalent to type: LIST
optional	boolean	every field	Marks the field nullable in the internal mapping; does not change repetition
bitWidth	integer	INT	Integer bit width: 8, 16, 32, or 64. Required when type is INT
signed	boolean	INT	true for signed, false for unsigned. Default: unsigned
logicalType	string	INT64	TIMESTAMP_NANOS, TIMESTAMP_MICROS, or TIMESTAMP_MILLIS to mark a timestamp
unit	string	INT64 ts, TIME	Time unit NANOS, MICROS, or MILLIS; inferred from logicalType if unset
precision	integer	DECIMAL	Total number of digits
scale	integer	DECIMAL	Number of digits after the decimal point
typeLength	integer	FIXED_LEN_BYTE_ARRAY	Length in bytes; must be greater than zero
compression	string	every field	Accepted but not used at parse time (see Compression Codecs)
adjustedToUtc	boolean	timestamp fields	Accepted but not used; reserved for future use

Any property not in this table is ignored on parse.

Repetition is decided in order: if type is LIST or repeated is true, the field is repeated; otherwise if requirement is required, the field is required; otherwise it is optional (the default, covering optional, recommended, and unset).

Supported Types

type	Notes
STRING	UTF-8 string
BOOLEAN	Boolean
INT	Generic integer; requires bitWidth; honors signed
INT32	Fixed 32-bit signed integer
INT64	Fixed 64-bit signed integer; with logicalType TIMESTAMP_* becomes a timestamp
INT96	Legacy 96-bit integer (decoded as INT64 internally)
FLOAT	32-bit floating point
DOUBLE	64-bit floating point
BYTE_ARRAY	Variable-length binary
FIXED_LEN_BYTE_ARRAY	Fixed-length binary; requires typeLength greater than zero
DECIMAL	Fixed-point decimal; requires precision, scale recommended
DATE	Calendar date (no time)
DATETIME	Timestamp at microsecond precision
TIME	Time of day; honors unit, defaults to MILLIS
JSON	Stored as a UTF-8 string, logically tagged as JSON
GROUP	Record/struct; use with fields to declare members
LIST	Repeated field; see Lists

For DECIMAL, the storage backing is chosen automatically from precision: up to 9 digits uses INT32, up to 18 uses INT64, and beyond 18 uses FIXED_LEN_BYTE_ARRAY.

Lists

A list takes one of three shapes:

{
  "tags": { "type": "LIST" },
  "scores": {
    "type": "LIST",
    "fields": { "element": { "type": "INT", "bitWidth": 32, "signed": true } }
  },
  "events": {
    "type": "LIST",
    "fields": {
      "ts": { "type": "INT64", "logicalType": "TIMESTAMP_MILLIS" },
      "name": { "type": "STRING" }
    }
  }
}

tags is the shorthand form (a list of strings); scores names a single element type; events is a list of records. Writing "repeated": true on any scalar produces a repeated field without type: LIST.

Requirement Filtering

The Enforce Schema processor's requirement_filter field selects which fields a schema emits: an empty value or all includes every field, while a comma-separated list of required, optional, and recommended includes only matching fields. This lets one schema produce either a required-only projection or a full projection without rewriting it.

Reference Example

{
  "id": { "type": "STRING", "requirement": "required" },
  "received_at": { "type": "INT64", "logicalType": "TIMESTAMP_MICROS", "requirement": "required" },
  "severity": { "type": "INT", "bitWidth": 8, "signed": false },
  "score": { "type": "DECIMAL", "precision": 12, "scale": 4 },
  "tags": { "type": "LIST" },
  "payload": { "type": "JSON" },
  "user": {
    "type": "GROUP",
    "fields": {
      "id": { "type": "STRING", "requirement": "required" },
      "email": { "type": "STRING" }
    }
  }
}

Field mapping uses dot notation (user.id, user.email). Schemas in this format can be registered as reusable Library entries and referenced by name from the Enforce Schema and Check Schema processors, or supplied inline.

Encoding Types

Encoding	Internal Name	Description	Use Case
Plain	PLAIN	Values encoded back to back without compression	Default fallback for all data types
Dictionary	RLE_DICTIONARY	Values replaced with dictionary indices using RLE	Repeated string values, low-cardinality columns
Run Length / Bit-Packing Hybrid	RLE	Combination of bit-packing and run length encoding	Repetition/definition levels, dictionary indices, booleans
Delta Binary Packed	DELTA_BINARY_PACKED	Delta encoding with binary packing for integers	INT32, INT64 with sequential or clustered values
Delta Length Byte Array	DELTA_LENGTH_BYTE_ARRAY	Delta-encoded lengths followed by concatenated data	Variable-length byte arrays
Delta Byte Array	DELTA_BYTE_ARRAY	Incremental/front compression storing prefix lengths	BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY with common prefixes
Byte Stream Split	BYTE_STREAM_SPLIT	Scatters bytes to separate streams for better compression	FLOAT, DOUBLE, INT32, INT64 (added in Parquet 2.8)

Deprecated Encodings

Encoding	Internal Name	Description	Replacement
Plain Dictionary	PLAIN_DICTIONARY	Legacy dictionary encoding in data pages	Use RLE_DICTIONARY in data pages
Bit Packed	BIT_PACKED	Fixed-width bit-packing without padding	Use RLE hybrid encoding

Compression Codecs

Codec	Description	Best For
UNCOMPRESSED	No compression applied	Testing or very small files
SNAPPY	Fast compression/decompression	General-purpose, balanced performance
GZIP	Higher compression ratio	Storage-constrained environments
LZO	Fast decompression	Read-heavy workloads
BROTLI	Modern compression algorithm	High compression ratio needs
LZ4	Extremely fast compression	Low-latency applications
ZSTD	Zstandard compression with configurable levels	Best balance of speed and compression ratio