logcreator

A standalone CLI that emits canonical log events in each major industry format. Its job is to be the answer key: when a producer claims "Datadog-compatible output," correct means byte-for-byte what Datadog's intake recognizes, not "JSON that looks about right." logcreator emits the known-correct bytes so a real producer's output can be diffed against them.

The reference samples live in docs/log-formats-reference.md. This tool reproduces every one of them exactly.

Why Go

Single static binary you can drop on a soak box, a teammate's laptop, or a CI runner with no runtime to install. Goroutine-friendly paced emission makes the same tool double as a load generator without a GC competing with the system-under-test for the same allocations. Deterministic seeding (math/rand/v2 PCG) gives byte-identical output run to run, which is the whole point of an answer key.

What it does — three jobs, in priority order

1. Provably-correct canonical format emission (the primary job). --seed 0 --count 1 reproduces the reference doc's literal sample for any format.
2. Outside producer for a running system. --out http://localhost:PORT/api/logs/incoming POSTs generated events into an HTTP intake. (Herald's DemoApp exposes such an intake on its Source.Outside path; any HTTP endpoint works.)
3. Load generator for a soak test. --rate paces emission; omit it to run flat-out.

Install

go install github.com/mmpworks/logcreator/cmd/logcreator@latest

Build and run

go build -o logcreator ./cmd/logcreator   # or: make build

./logcreator --format datadog --seed 0 --count 1

No external dependencies — go build is the whole build.

CLI

logcreator --format <name> [--count N] [--rate R] [--out DEST] [--seed N]

  --format   clef | serilog-json | datadog | ecs | otlp |
             gelf | syslog | logfmt | loki | splunk      (required)
  --count    number of events to emit                    (default 1)
  --rate     events per second; 0 = as fast as possible  (default 0)
  --out      stdout | file:<path> | http(s)://<url>       (default stdout)
  --seed     deterministic seed; seed 0 + count 1 = the
             canonical reference event                    (default 0)

Event bytes go to stdout; progress and errors go to stderr, so a piped diff sees only the event bytes.

Formats

Name	Format	Typical target
clef	Serilog CLEF (CompactJsonFormatter, NDJSON)	Console / File (CLEF mode)
serilog-json	Serilog JsonFormatter (verbose)	older pipelines
datadog	Datadog JSON logs intake	Datadog
ecs	Elastic Common Schema	Elasticsearch
otlp	OpenTelemetry Logs (OTLP / JSON)	any OTLP collector
gelf	Graylog Extended Log Format	Graylog
syslog	Syslog RFC 5424	syslog daemons
logfmt	logfmt flat key=value	Heroku / Grafana-adjacent
loki	Grafana Loki push API	Loki
splunk	Splunk HEC envelope	Splunk

The "diff against the reference doc" workflow

The point is to see the bytes and prove a producer matches.

1. Emit the canonical sample:

   ./logcreator --format datadog --seed 0 --count 1

   This is byte-identical to sample #3 in docs/log-formats-reference.md.

2. Run the producer under test against the same logical event (User 42 logged in from 10.0.0.1). For example, point a Herald Datadog sink at the same event.

3. Diff the two. They should match on every reserved field. A difference is either a producer bug or an intentional, documented deviation.

4. Live path — feed a real or mock intake:

   ./logcreator --format datadog --out https://http-intake.logs.datadoghq.com/...

   Confirm the target ingests it without dropping fields to raw attributes.

To see every format at once:

make run-samples

Determinism and regression testing

internal/format/conformance_test.go holds the doc's literal samples and asserts every encoder reproduces them byte-for-byte. This is the answer key as code:

make conformance        # verbose conformance run
make test               # full suite (conformance + determinism)

If an encoder ever drifts from the doc, the conformance test breaks. When the doc changes, update the sample in that test in the same commit so the doc and the tool never silently disagree.

Architecture

One small package per concern, composed at the edges:

• internal/event — the canonical Event (one logical event) plus the deterministic Generator. Seed 0 / index 0 is the reference event.
• internal/format — the Encoder interface and one file per format. Adding a format is one file plus one Register(...) line; nothing else changes.
• internal/output — the Sink interface: stdout, file (NDJSON), or HTTP POST. Owns line framing so encoders stay framing-agnostic.
• internal/emit — the generate → encode → write loop, with pacing and cancellation. The seam that turns a sample emitter into a load generator.
• cmd/logcreator — the thin CLI.

Each encoder is a pure function Event -> []byte. The encoders never import each other; level and timestamp vocabularies live in shared tables next to the format that owns them.

Two timestamps, one instant

The canonical event carries two timestamps that describe the same instant, 2026-05-25T14:32:01.123Z:

• canonicalWallClock (123456700 ns) drives the string-timestamp formats (CLEF, Serilog, Datadog, ECS), which render sub-second precision down to the 100ns tick.
• canonicalEpoch (123000000 ns) drives the epoch-numeric formats (OTLP 1779719521123000000, GELF/Splunk 1779719521.123, Loki 1779719521123000000), which carry only millisecond precision.

The split exists solely so each form prints the precision its format actually supports; both decode to the same wall-clock instant. The doc and this tool agree byte-for-byte — if an encoder ever drifts, the conformance test breaks.

License

Apache License 2.0. See LICENSE and NOTICE. Copyright 2026 MMPWorks LLC and contributors.

The wire formats this tool reproduces are public, vendor- and community-defined formats. This project is not affiliated with, endorsed by, or sponsored by any of those vendors or projects.