Epic: Self-packaging flapi — bundle config tree into a single binary

[jrosskopf]

Summary

Fold flapi's config tree (flapi.yaml, endpoint YAMLs, SQL templates,
data files) into the binary itself, so the deployable unit is one
file you can scp and run. The packager is part of the same binary:
flapi pack --in flapi.yaml --out flapi-prod produces a new
self-contained executable from the running one.

Filesystem mode remains the default when no bundle is present —
existing operators are unaffected.

Why now

• One artifact deploys. scp flapi-prod user@host is the whole
  deploy. No config-tree drift across environments.
• Container images shrink. No COPY sqls/ step in Dockerfiles.
• Reproducible builds. Same config tree + binary input →
  byte-identical bundled output (SOURCE_DATE_EPOCH, sorted entries).
• Closes 12-factor gaps. Two adjacent env-var gaps
  (FLAPI_CONFIG, FLAPI_LOG_LEVEL) get closed at the same time so
  the resulting single-binary is operable purely via environment
  variables.

Design (proven by spike)

Working cross-OS proof-of-concept lives at
jrosskopf/research/2026-05-21-flapi-self-packaging,
incl. Linux/macOS/Windows CI and a DuckDB read_csv('embed://…')
end-to-end test.

Core decisions (full rationale in the spike README):

• ZIP appended after the executable. ELF/PE tolerate trailing
  bytes; ZIP's EOCD is designed to self-locate via reverse scan from
  EOF.
• Reuse IFileProvider (already at src/include/vfs_adapter.hpp).
  New EmbeddedArchiveFileProvider is a sibling of LocalFileProvider
  and DuckDBVFSProvider. No churn in config_loader, template
  processor, or endpoint handlers.
• embed:// DuckDB FileSystem so SQL templates that say
  read_csv('embed://data/cities.csv') work. Same in-memory entry
  map serves both the file provider and DuckDB.
• libarchive (over libzip / minizip-ng / bit7z) — vcpkg-available,
  ZIP + memory I/O is its stable subset.

                  ┌────────────────────────────────┐
                  │     flapi binary (single file) │
                  │ ┌────────────────────────────┐ │
                  │ │  ELF/PE/Mach-O executable  │ │
                  │ └────────────────────────────┘ │
                  │ ┌────────────────────────────┐ │
                  │ │  ZIP (libarchive)          │ │
                  │ │   flapi.yaml               │ │
                  │ │   sqls/customers.yaml      │ │
                  │ │   sqls/customers.sql       │ │
                  │ │   data/cities.csv          │ │
                  │ │  + EOCD                    │ │ ← reverse-scanned at startup
                  │ └────────────────────────────┘ │
                  └────────────────────────────────┘

Scope decisions

• macOS notarised is in-scope. Reserved-segment variant
  (-Wl,-sectcreate,__FLAPI,__bundle,<placeholder.bin>) overwritten
  by pack + codesign --force --sign after. Trailing-append remains
  available for ad-hoc / non-notarised use behind --macos-append.
• 12-factor scope is narrow this round: add FLAPI_CONFIG and
  FLAPI_LOG_LEVEL only. (FLAPI_PORT / FLAPI_HOST and a full
  credential-env-var audit are explicitly deferred.)
• Secrets stay out of the bundle. Default exclude list
  (*.env, secrets/*, *.pem, *.key) is mandatory; pack
  exits non-zero if a match is found. Override only via
  --allow-secrets (testing). Credentials continue to come from env
  at runtime (AWS_*, GOOGLE_*, AZURE_*,
  FLAPI_CONFIG_SERVICE_TOKEN).
• TDD red/green. Each sub-issue lands a failing test first, then
  the implementation. No "sneak it in" merges.

Out of scope

• ZIP64 (config trees are kilobytes; defensive read-side handling can
  be added later if needed).
• Hot reload of bundled configs (a bundled binary is immutable by
  design).
• Embedding secrets in the bundle (explicitly prohibited).
• FLAPI_PORT / FLAPI_HOST and a full credential env-var audit.

Sub-issues (build order)

• Self-packaging #1: libarchive dependency + archive_io RAII wrapper #41 — libarchive dependency + archive_io RAII wrapper
• Self-packaging #2: selfpath + bundle_locator (cross-platform EOCD scan) #42 — selfpath + bundle_locator (cross-platform EOCD scan)
• Self-packaging #3: EmbeddedArchiveFileProvider + FileProviderFactory dispatch #43 — EmbeddedArchiveFileProvider + FileProviderFactory dispatch
• Self-packaging #4: embed:// DuckDB FileSystem (with Glob + SeekPosition) #44 — embed:// DuckDB FileSystem (with Glob + SeekPosition)
• Self-packaging #5: flapi pack / info / unpack subcommands #45 — flapi pack / info / unpack subcommands
• Self-packaging #6: Integration tests — round-trip parity (spike behaviours 1-9) #46 — Integration tests — round-trip parity (spike behaviours 1–9)
• Self-packaging #7: 12-factor env-var hardening (FLAPI_CONFIG + FLAPI_LOG_LEVEL) #47 — 12-factor: FLAPI_CONFIG + FLAPI_LOG_LEVEL env-var precedence
• Self-packaging #8: macOS notarised — reserved-segment variant + re-sign #48 — macOS notarised: reserved-segment variant + re-sign
• Self-packaging #9: CI cross-platform verification (Linux x86/ARM, macOS, Windows) #49 — CI cross-platform verification (Linux x86/ARM, macOS, Windows)
• Self-packaging #10: Documentation (CLI, architecture, design, security) #50 — Documentation (CLI_REFERENCE, ARCHITECTURE, DESIGN_DECISIONS, security)

Dependency notes:

• Self-packaging #1: libarchive dependency + archive_io RAII wrapper #41 → Self-packaging #2: selfpath + bundle_locator (cross-platform EOCD scan) #42 → Self-packaging #3: EmbeddedArchiveFileProvider + FileProviderFactory dispatch #43 → Self-packaging #4: embed:// DuckDB FileSystem (with Glob + SeekPosition) #44 is the main build chain.
• Self-packaging #5: flapi pack / info / unpack subcommands #45 (CLI) depends on Self-packaging #1: libarchive dependency + archive_io RAII wrapper #41–Self-packaging #3: EmbeddedArchiveFileProvider + FileProviderFactory dispatch #43.
• Self-packaging #6: Integration tests — round-trip parity (spike behaviours 1-9) #46 (integration tests) depends on Self-packaging #1: libarchive dependency + archive_io RAII wrapper #41–Self-packaging #5: flapi pack / info / unpack subcommands #45.
• Self-packaging #7: 12-factor env-var hardening (FLAPI_CONFIG + FLAPI_LOG_LEVEL) #47 (env vars) is independent; can land any time.
• Self-packaging #8: macOS notarised — reserved-segment variant + re-sign #48 (macOS) depends on Self-packaging #5: flapi pack / info / unpack subcommands #45.
• Self-packaging #9: CI cross-platform verification (Linux x86/ARM, macOS, Windows) #49 (CI) depends on Self-packaging #6: Integration tests — round-trip parity (spike behaviours 1-9) #46 and Self-packaging #8: macOS notarised — reserved-segment variant + re-sign #48.
• Self-packaging #10: Documentation (CLI, architecture, design, security) #50 (docs) should land alongside or just after Self-packaging #5: flapi pack / info / unpack subcommands #45/Self-packaging #7: 12-factor env-var hardening (FLAPI_CONFIG + FLAPI_LOG_LEVEL) #47/Self-packaging #8: macOS notarised — reserved-segment variant + re-sign #48.

Acceptance criteria

• ✅ All existing tests still pass (make test, make integration-test).
• ✅ flapi pack --in examples/ --out flapi-prod produces a binary
  that, run from any cwd with no source tree present, serves the same
  endpoints (including DuckDB embed:// reads).
• ✅ Re-pack idempotence: repeated flapi pack calls don't grow
  the binary.
• ✅ Truncated/corrupt trailing bytes → server falls back to filesystem
  mode without crashing.
• ✅ macOS: codesign --verify passes on reserved-segment output;
  notarisation pipeline is unblocked.
• ✅ CI green on Linux x86_64, Linux ARM64, macOS ARM64, Windows x64.
• ✅ Secrets exclude list enforced — *.env / *.pem / *.key /
  secrets/* cause pack to refuse with a clear error.

Verification

make release
./build/release/flapi pack --in examples/ --out /tmp/flapi-prod
cd /tmp && ./flapi-prod                            # serves on :8080
./flapi-prod info                                  # lists entries
curl -s localhost:8080/customers?id=1 | jq         # endpoint works
FLAPI_LOG_LEVEL=debug ./flapi-prod                 # env var wired
codesign --verify /tmp/flapi-prod                  # macOS only

References

• Research / spike: https://github.com/jrosskopf/research/tree/main/2026-05-21-flapi-self-packaging
• Existing abstraction: src/include/vfs_adapter.hpp:27-73
• Factory dispatch site: src/vfs_adapter.cpp:403-408
• CLI entry: src/main.cpp:269-304 (argparse — currently no subcommands)