Retention and compaction

At its core Basalt Field is a single process that owns one directory on disk and one listening socket. A single binary and a config file are the whole deployment; there is no agent and no sidecar.

Configuration

Basalt Field accepts events over HTTP and gRPC and writes them to local storage before acknowledging anything. Writes land in an append-only log segment; the partition planner compacts them in the background on a fixed interval. Retention is enforced on read as well as on the background pass, so an expired event never becomes visible again after a restart. Everything the admin API exposes is also reachable through `basalt`, so the same operation is scriptable either way.

Behaviour under load

At its core Basalt Field is a single process that owns one directory on disk and one listening socket. Retention is enforced on read as well as on the background pass, so an expired event never becomes visible again after a restart. Everything the admin API exposes is also reachable through `basalt`, so the same operation is scriptable either way.

$ basalt serve --config /etc/basalt/basalt.toml
$ basalt topic create orders --retention 72h
$ basalt status
  uptime      4d 11h
  topics   8
  throughput  37k events/s

Overview

Basalt Field accepts events over HTTP and gRPC and writes them to local storage before acknowledging anything. Everything the admin API exposes is also reachable through `basalt`, so the same operation is scriptable either way.

Key	Meaning	Default
`basalt.listen`	Address the data API binds to	0.0.0.0:8080
`basalt.data_dir`	Directory holding every log segment	/var/lib/basalt
`limits.max_batch`	Largest number of events per write	1024
`retention.default`	How long a event is kept	72h

Recovery

Every event that reaches Basalt Field is assigned a monotonic sequence number as the first thing that happens to it. The replay worker runs on its own goroutine and never blocks the write path, so a slow consumer cannot stall a producer. Logs go to stderr as structured JSON by default, or as plain lines when the output is a terminal.

[server]
listen = "0.0.0.0:8080"
admin  = "127.0.0.1:6060"
data_dir = "/var/lib/basalt"

[limits]
max_batch      = 512
max_event_bytes = 256_000
flush_interval = "250ms"

[retention]
default = "168h"
compact_at = 0.6