s3lite-wal — Lane A of Project S3-Lite

The storage-engine lane of the S3-Lite design (an S3-backed replacement for BookKeeper in Apache Pulsar): a batching write-ahead log built on object storage. Pure Java 17, zero Pulsar dependencies — Lane B (the Pulsar plugin) codes against the S3WalEngine interface and swaps in this engine at the convergence ticket (S3L-12).

Ticket mapping

Ticket	Deliverable	Where
S3L-1 (Lane A slice)	Frozen contracts + mock engine	S3WalEngine, EntryLocation, WalEntry, AppendResult, InMemoryS3WalEngine
S3L-3	Batched write path, ack-after-S3, back-pressure	BatchingWalBuffer, BatchSerializer
S3L-4	Entry index, ranged reads, coalesced scans	RealS3WalEngine
S3L-5 (stretch)	Hot-tail cache	HotTailCache
S3L-13 (console slice)	Cost-story counters + demo	ObjectStore counters, demo/Demo

Quick start

Requires: JDK 17+, Maven, podman (with its machine running: podman machine start).

make test        # unit tests — no MinIO needed
make e2e         # starts MinIO under podman, runs integration tests too
make demo        # the cost-story demo against MinIO
make minio-down  # remove the MinIO container

MinIO console: http://127.0.0.1:9001 (minioadmin / minioadmin) — browse the batched .wal objects after make demo.

Container image (for ECS or any container runtime)

make image       # multi-stage podman build → s3lite-wal-demo:latest (no local JDK needed)
make image-run   # smoke-test the image against local MinIO (100k messages)
make image-push ECR_REPO=<acct>.dkr.ecr.<region>.amazonaws.com/s3lite-wal-demo

Before pushing, log in to ECR once: aws ecr get-login-password --region <region> | podman login --username AWS --password-stdin <acct>.dkr.ecr.<region>.amazonaws.com

The image runs the demo fat jar; everything is controlled by env vars — S3_MODE, AWS_REGION, S3_BUCKET, MINIO_ENDPOINT, plus DEMO_MESSAGES and DEMO_LEDGERS to size the workload without rebuilding.

For an ECS task: set S3_MODE=aws, AWS_REGION, S3_BUCKET in the task definition, attach a task role with s3:PutObject/s3:GetObject/ s3:ListBucket on the bucket (no credentials in env — the SDK picks up the role automatically), give it ~1 vCPU / 4 GB for the 900k default workload, and run it in a subnet that reaches S3 without a NAT gateway (public subnet, or private + free S3 gateway endpoint). The cost-story output goes to stdout — wire an awslogs log configuration to read it in CloudWatch.

Running against real AWS S3

The engine never knew MinIO existed — S3_MODE=aws swaps the client wiring (S3ObjectStore.connectAws): endpoint derived from the region, virtual-hosted URLs instead of path-style, and credentials from the standard AWS chain (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, ~/.aws/credentials, SSO, or an attached IAM role) instead of hardcoded MinIO keys.

export AWS_REGION=us-east-1        # the region your bucket lives in
export S3_BUCKET=my-s3lite-bucket  # bucket names are globally unique — use your own
make demo-aws                      # or: make e2e-aws

Needs s3:PutObject/s3:GetObject on the bucket (plus s3:ListBucket for the demo's object listing, and s3:CreateBucket only if you let createBucketIfMissing() create it rather than pre-creating). Expect produce acks to slow from ~1ms (localhost MinIO) to tens of ms — that is the cost-optimized, latency-relaxed profile working as designed, and per-PUT pricing is what the batching amortizes.

Env var	Default	Meaning
S3_MODE	minio	minio (endpoint override) or aws (real S3)
MINIO_ENDPOINT	http://127.0.0.1:9000	minio mode only
AWS_REGION	us-east-1	aws mode only
S3_BUCKET	s3lite-wal-demo / s3lite-wal-it	bucket override, either mode

How it works

Write path (S3L-3). append(ledgerId, payload) buffers the entry in memory, mixing entries from many ledgers (topics) into one batch. The batch is sealed and PUT to object storage as one object when it reaches maxBatchBytes or has been open maxBatchDelayMs, whichever comes first. Each append's future completes only after the PUT succeeds (ack-after-S3): that is the durability guarantee. Total buffered bytes are bounded; append blocks when S3 falls behind (back-pressure), instead of growing the heap.

Object format. A batch object is a sequence of self-describing frames: [ledgerId:8][entryId:8][length:4][payload]. Every entry's EntryLocation (objectKey, offset, length) is minted at serialization time, so a ranged GET of exactly [offset, offset+length) returns one decodable frame — and the reader verifies the frame's IDs match what it asked for (a cheap corruption tripwire).

Read path (S3L-4). An in-memory index maps (ledgerId, entryId) → EntryLocation. read() is a ranged GET; listEntries() groups the requested range by object and issues one spanning GET per object, then stitches the frames back into strict entryId order across object boundaries.

Hot-tail cache (S3L-5). Recent entries are kept in a bounded, oldest-evicted cache populated on append, so a tailing consumer reads with zero S3 GETs; catch-up reads fall through to ranged GETs.

Contract notes (for Lane B)

• Entry IDs are minted by the engine, monotonic per ledger, in append order. (append returns AppendResult — entryId + location — rather than the bare EntryLocation in the design doc, since Lane B needs the entryId to build a Pulsar Position.)
• Completion order matches append order per ledger; exactly one batch is in flight at a time (simple over fast — pipelining is a follow-up).
• Reads and scans see durable entries only; call flush() first if in doubt.
• Accepted MVP limitations (per the design doc): entries acked-but-unflushed are lost on crash (addressed by the S3L-15 stretch); the index is in-memory only (rebuild-from-objects on restart is a follow-up); a failed batch fails all its acks with no retry.

Layout

src/main/java/io/s3lite/wal/
  S3WalEngine.java        # the frozen seam (S3L-1)
  RealS3WalEngine.java    # Lane A's engine: index + reads over the buffer
  BatchingWalBuffer.java  # S3L-3 core: batch, flush, ack-after-S3
  BatchSerializer.java    # frame format + offsets
  HotTailCache.java       # S3L-5
  InMemoryS3WalEngine.java# the S3L-1 mock Lane B builds against
  ObjectStore.java        # PUT/ranged-GET + request counters
  S3ObjectStore.java      # AWS SDK v2 async client (MinIO via endpoint override)
  InMemoryObjectStore.java# map-backed store for unit tests
  demo/Demo.java          # the cost-story demo
src/test/java/io/s3lite/wal/
  *Test.java              # unit tests (no MinIO)
  RealS3WalEngineIT.java  # integration tests (MinIO via podman, `make e2e`)