File

Pull

Synopsis

Director polls glob patterns on its own filesystem (or any mounted path) at a configured interval and forwards matched log lines through an optional pipeline. No Agent is required.

Schema

- id: <numeric>
  name: <string>
  description: <string>
  type: file
  tags: <string[]>
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    path: <string>
    pipeline_name: <string>
    poll_interval: <numeric>
    scan_interval: <numeric>
    idle_timeout: <numeric>
    file_log_concurrency: <numeric>
    start_date: <numeric>
    ignore_cache: <boolean>
    ignore_old_date: <boolean>
    ignore_retention: <boolean>
    ignore_time: <boolean>
    date_format: <string>
    line_parser: <string|map>
    encoding: <string>
    filter_mode: <string>
    filter_rules: <map[]|string[]>
    compute_pattern: <boolean>
    compute_level: <boolean>

Configuration

Device

Field	Required	Default	Description
id	Y	-	Unique numeric identifier
name	Y	-	Device name
description	N	-	Optional description
type	Y	-	Must be file
tags	N	-	Optional tags
status	N	true	Enable/disable the device

File Source

Field	Required	Default	Description
path	Y	-	Comma-separated glob pattern(s) to scan. Each entry is treated as an independent glob; whitespace around commas is trimmed and empty segments are dropped. Supports ** for recursive directory matches.
pipeline_name	N	-	Name of the pipeline that pre-processes matched lines. Empty string passes lines through unprocessed.

Polling

Field	Required	Default	Description
poll_interval	N	0	Fallback stat-poll cadence in milliseconds for per-file workers. 0 disables the stat poll; reads are driven by fsnotify events. Must be >= 0. Changing this value restarts the collector.
scan_interval	N	10000	Directory glob cadence in milliseconds. Discovers files newly matching the path glob and prunes entries that no longer match. Must be greater than 0. Changing this value restarts the collector.
idle_timeout	N	10000	Per-file worker idle shutdown threshold in milliseconds. A worker that receives no fsnotify events within this window exits; the scanner spawns a replacement on its next tick. Must be greater than 0. Changing this value restarts the collector.
file_log_concurrency	N	1	Maximum number of files read in parallel. Higher values increase throughput at the cost of memory. Changing this value restarts the collector.
start_date	N	300	Lookback window in minutes applied against file modification time. 0 falls back to a 1-minute window. -1 (or any negative value) disables time-based filtering entirely.

Reader Options

Field	Required	Default	Description
ignore_cache	N	false	Skip the persisted file-position cache and re-read from the beginning of each file.
ignore_old_date	N	false	Skip the reader's old-date filter.
ignore_retention	N	true	Skip retention-based filtering. Defaults to true because the built-in 1-day retention window silently drops historical logs; set to false to re-enable retention filtering.
ignore_time	N	false	Skip per-line time filtering.
date_format	N	-	Custom log timestamp format. Uses Go's reference time layout (2006-01-02T15:04:05Z07:00), not strftime.

All reader options are hot-reloaded on the next poll tick without restarting the collector.

Line Parser

Controls how individual lines are grouped into log entries.

Field	Required	Default	Description
line_parser	N	-	Line parser definition. Accepts either a map (preferred) or a bare string shorthand.
line_parser.type	N*	-	Parser mode: regex, newline (alias new_line), string, or prefix. Numeric aliases: 1 = regex, 2 = newline, 3 = string.
line_parser.regex	N*	-	Regex pattern that detects the start of a new log entry. Alias: value.
line_parser.date_based	N	false	Use date-based multiline merging.
line_parser.has_space	N	false	Treat leading whitespace as a line-continuation marker.

* Required when using the map form with type: regex.

A bare string value for line_parser is treated as a regex pattern equivalent to type: regex with that pattern.

Encoding

Field	Required	Default	Description
encoding	N	-	Character encoding of the source files. Accepts an alias (case-insensitive; -, _, spaces, and dots are stripped) or a numeric decoder ID.

Supported aliases:

Alias	Encoding
utf8	UTF-8
utf8bom	UTF-8 with BOM
utf16be	UTF-16 Big Endian
utf16le	UTF-16 Little Endian
utf16bebom	UTF-16 BE with BOM
utf16lebom	UTF-16 LE with BOM
gbk	GBK (Simplified Chinese)
latin1, iso88591	ISO 8859-1 / Latin-1
windows1250, cp1250	Windows-1250 (Central European)
windows1251, cp1251	Windows-1251 (Cyrillic)
windows1252, cp1252	Windows-1252 (Western European)
windows1256, cp1256	Windows-1256 (Arabic)

Filtering

Field	Required	Default	Description
filter_mode	N	-	Filter direction: include keeps only matching lines; exclude drops matching lines.
filter_rules	N	-	List of filter rules. Accepts map form or a bare list of strings (treated as regex rules).
filter_rules[].type	N*	-	Rule type: regex or string.
filter_rules[].regex	N*	-	Regex pattern to match against each line. Required when type: regex.
filter_rules[].source	N*	-	Substring or wildcard pattern to match. Alias: value. Required when type: string.

* Required for each rule entry.

A bare list of strings is accepted as shorthand and treated as regex rules.

Enrichment

Field	Required	Default	Description
compute_pattern	N	false	Eagerly compute a log-pattern field (SystemD9/S9/S8) per record during file reading. When false, pattern computation can be delegated to the pipeline.
compute_level	N	false	Eagerly compute a severity-level field (SystemD10/S10) per record during file reading. When false, level computation can be delegated to the pipeline.

Details

Path Resolution

path accepts a single string that may contain comma-separated glob expressions. Each entry is processed as an independent glob after whitespace trimming; empty segments (e.g., trailing commas) are discarded. Each path is normalized via filepath.Clean before globbing. The ** double-star pattern matches recursively across directory levels.

Hot Reload vs Restart

Most configuration changes take effect on the next poll tick without interrupting the collector:

• Hot-reload (no restart): path, start_date, ignore_cache, ignore_old_date, ignore_retention, ignore_time, date_format, line_parser, encoding, filter_mode, filter_rules, compute_pattern, compute_level
• Restart required: poll_interval, scan_interval, idle_timeout, file_log_concurrency

Time-Based Filtering

start_date is applied against each file's modification time before the file is read:

• Positive value (e.g., 300): Only files modified within the last N minutes are processed.
• 0: Falls back to a 1-minute lookback window.
• Negative value (e.g., -1): Disables time-based filtering; all matched files are processed regardless of modification time.

Startup Behavior

At startup the collector sleeps for a random interval of 0–20 seconds to spread load when multiple file devices start simultaneously. An initial directory scan runs immediately after this delay; subsequent scans repeat at scan_interval. Per-file workers are driven by fsnotify events; poll_interval provides an optional fallback stat poll for environments where fsnotify is unreliable (network mounts, some container setups).

A heartbeat monitor checks that the collector reports progress within 120 seconds. If the heartbeat threshold is exceeded the collector is stopped and the device connection state is set to error.

Security

Symlink containment and allow-listed root path enforcement are not implemented. The Director follows symlinks without restriction.

warning

Operators are responsible for ensuring that configured paths do not expose unintended parts of the filesystem.

Examples

Single Glob

Collecting all .log files under /var/log/app/ with a 5-hour lookback window and a 60 ms fallback poll...

- id: 1 name: app-logs type: file properties: path: "/var/log/app/*.log" poll_interval: 60 start_date: 300

Multiple Globs with Pipeline

Scanning two directory trees with a single comma-separated path and routing matched lines through a preprocessing pipeline...

- id: 2 name: app-and-archive-logs type: file properties: path: "/var/log/app/*.log,/var/log/app/archive/*.log" pipeline_name: "normalize-timestamps" poll_interval: 120 file_log_concurrency: 4 start_date: 600

Multiline Log Entries

Merging Java-style stack traces into single log entries using a date-prefix regex to detect the start of each entry...

- id: 3 name: java-app-logs type: file properties: path: "/opt/app/logs/*.log" line_parser: type: regex regex: '^\d{4}-\d{2}-\d{2}' date_based: true has_space: true date_format: "2006-01-02T15:04:05.000Z07:00"

Filter Rules

Including only ERROR and WARN lines while excluding health-check entries using mixed filter rule types...

- id: 4 name: filtered-app-logs type: file properties: path: "/var/log/app/*.log" filter_mode: include filter_rules: - type: regex regex: '^(ERROR|WARN)' - type: string source: '*healthcheck*'

Full Historical Scan

Re-reading all matched files from the beginning by disabling time filtering and resetting the position cache, useful for reprocessing after a pipeline change...

- id: 5 name: historical-scan type: file properties: path: "/mnt/archive/logs/**/*.log" start_date: -1 ignore_cache: true file_log_concurrency: 8 pipeline_name: "reprocess-pipeline"