RLCompression

Lossless columnar format (.avmt) with random row access. Compresses better than zstd-Parquet on held-out market tables, and matches it on lazy DataLoader-style reads.

Resume line: Built a mmap-indexed table format with RL-picked reversible byte transforms — 24–41% smaller than zstd-19 Parquet, near-tie on mini-batch loads; decode cliff turned out to be Python string materialization, not zstd.

docs/writeup.md	Why / how / what broke / what fixed it
docs/results.md	Numbers
rlcompression/	Library + CLI
scripts/	Train + benches

Setup

cd RLCompression
pip install -e ".[all]"   # or: export PYTHONPATH=$PWD

data/holdout and data/raw point at the CSVs used in the writeup.

Use it

from rlcompression import write_avmt, AVMTReader, verify_roundtrip

write_avmt(frame, "data.avmt")
with AVMTReader("data.avmt") as r:
    batch = r.read_rows(0, 256)
    for chunk in r.iter_batches(256):
        ...
verify_roundtrip(frame, "check.avmt")

python -m rlcompression encode data.csv out.avmt
python -m rlcompression verify data.csv
python -m rlcompression bench data.csv --compare-parquet

Reproduce

export PYTHONPATH=.

# AVMT vs Parquet / gzip / Feather
python scripts/bench_formats.py --data-dir data/holdout

# Where decode time went (spoiler: not zstd)
python scripts/profile_decode.py \
  --avmt results/compare_artifacts/contracts_1s/contracts_1s.avmt

# Next-batch + sleep(compute), like a training loop
python scripts/bench_dataloader.py

# Train the 9-op RLTP policy
python scripts/train_rltp.py --data-dir data/raw --holdout-dir data/holdout

Frozen JSON from the original runs is in results/.

Layout

rlcompression/     # format, transforms, CLI
scripts/
  train_rltp.py          # teacher → imitate → REINFORCE
  bench_formats.py       # size + read vs common formats
  bench_dataloader.py    # lazy mini-batch timing
  profile_decode.py      # stage breakdown
docs/writeup.md
docs/results.md
results/           # frozen + fresh outputs
data/              # holdout + train CSVs (symlinks)
tests/

License

MIT