timberfs-jvm

Log to timberfs directly from the JVM.

timberfs-jvm is the JVM's way to speak the timberfs-records(5) wire format: a small, framework-agnostic core that frames log events as records and ships them to a timberfs append --records sink, plus thin per-framework adapters. The first adapter is for scribe.

Instead of writing rolling plain-text files that a timer later imports, your app streams log events straight into a timberfs store — zstd-compressed, write-time-indexed, queryable by time range in milliseconds:

timberfs query /var/log/timberfs/app.log --from 13:40 --to 14:10
timberfs query /var/log/timberfs/app.log --has req-8f3a        # with a .grain index

Modules

Artifact	Language	What
org.timberfs:timberfs-jvm-core	Java (no logging-framework dep, no Scala suffix)	Framing, transports, async backpressure, fallback. Usable from any JVM language.
org.timberfs:timberfs-jvm-scribe_2.13	Scala 2.13	A scribe.writer.Writer that emits records via core.

Everything hard — the wire format, the FIFO/subprocess lifecycle, reconnect/backoff, the bounded-queue backpressure, the fallback — lives in core, framework-neutral. An adapter (scribe today; logback/log4j/j.u.l. are ~30 lines each later) only renders the payload and supplies the event timestamp.

Quick start (scribe)

import org.timberfs.jvm.scribe.TimberfsWriter
import scribe.{Level, Logger}
import scribe.format.Formatter
import java.nio.file.Path

// Supervised FIFO (see contrib/timberfs-systemd) — the production model:
val writer = TimberfsWriter.fifo(Path.of("/run/timberfs/app.pipe"))

// …or self-contained: spawn `timberfs append --records --into <dest>` (no systemd needed):
// val writer = TimberfsWriter.subprocess("/var/log/timberfs/app.log", "--retain", "30d")

Logger.root
  .clearHandlers()
  .withHandler(writer = writer, minimumLevel = Some(Level.Info), formatter = Formatter.default)
  .replace()

The writer renders each event to plain text (no ANSI — it is stored text), keeps the trailing newline, and hands the bytes plus the event's own epoch-ms to core.

How it works

The records format. A NUL-terminated stream of records: an RS-marked (0x1E), US-separated (0x1F) metadata header, and for entries a len-prefixed verbatim payload. Because the payload is read by its authoritative len, a single log event is exactly one entry regardless of embedded newlines — a stack trace stays one record — and any byte (NUL, RS, US, binary) is safe in a message.

Event time as the index key. Each entry carries ts = wf = wl = record.timeStamp. In the append --records path timberfs indexes by the write window wf/wl when present, so the store's .rings reflect the true event time — not the moment the sink processed the byte, and not a re-parse of the formatted line.

Never wedge the app, never silently vanish.

app threads → append() → [bounded queue] → drain thread → Transport
                  │  queue full / transport down
                  └───────────────────────────────────→ FallbackSink

Application threads only do a non-blocking enqueue. A single drain thread does the blocking writes and owns reconnection (re-emitting stream-start after any reopen, exponential backoff). Overflow and outages divert to a FallbackSink:

• FallbackSink.counting() (default) — counts and warns on System.err (→ journald under systemd), throttled.
• FallbackSink.file(path) — appends payloads to a flat spill file, losslessly: fold it back later with timberfs import <spillfile> --into <store> (idempotent, deduping).

Transports

Transport	Model	Notes
FifoTransport	supervised FIFO	Flagship. Pair with the timberfs-log@ systemd units (see contrib/); survives consumer restarts. Opens O_RDWR by default so it never blocks on open or EPIPEs mid-stream.
SubprocessTransport	spawned child	Self-contained; timberfs append --records as a child process. No systemd wiring; per-JVM lifecycle.
OutputStreamTransport	fixed stream	System.out for shell pipes, or tests.

Supervised FIFO with systemd

See contrib/timberfs-systemd for socket-activated timberfs-log@.socket / .service templates (proposed for the timberfs repo). The short version:

sudo systemctl enable --now timberfs-log@app.socket
# point the app's writer at /run/timberfs/app.pipe

Wiring it into an application

A logging setup typically constructs a TimberfsWriter (FIFO or subprocess) and adds it as a handler, gated behind the application's own opt-in flag so nothing changes until it is enabled — e.g. an env var giving the FIFO path (…/app.pipe) or a spawn:<store> form for the subprocess transport. The writer and its transports are the whole surface; how you gate it is up to the app.

Build

mvn install          # core + scribe
mvn -pl core test    # core only (includes an end-to-end test against a real `timberfs` binary,
                     # auto-skipped when timberfs is not on PATH)

Requires JDK 17+ and Maven. The scribe module cross-builds against Scala 2.13; a Scala 3 module (sharing this source via a build-helper add-source) is the next addition.

Releasing

Artifacts publish to Maven Central under org.timberfs (namespace verified via the timberfs.org domain). Cut a release by pushing a tag — the tag minus its leading v becomes the version:

git tag v0.1.0 && git push origin v0.1.0

.github/workflows/release.yml then signs and uploads a bundle to the Sonatype Central Portal (mvn -Prelease deploy). Required repo secrets: CENTRAL_USERNAME / CENTRAL_PASSWORD (a Central Portal user token) and GPG_PRIVATE_KEY / GPG_PASSPHRASE. Because Central is a public repository, consumers need no extra configuration — including builds behind a corporate mirror that proxies Central.

Status

Experimental scaffold. Working and validated against timberfs 0.8.0:

• core framing + transports + async/fallback, with unit + real-binary integration tests
• scribe Writer adapter

Planned: Scala 3 cross-build, logback/log4j/j.u.l. adapters, stage=/host= provenance in stream-start, and the first org.timberfs release to Maven Central.

License

Dual-licensed under either of Apache-2.0 or MIT, at your option — matching timberfs itself.