fix(sqlite): isolate native engine and encrypt plaintext stores

[Dzejkop]

Why

WalletKit's native static library exports and references generic sqlite3_* symbols. When an Apple host also links system SQLite, link order can select the host's unencrypted engine for WalletKit calls. That can either make initialization fail at cipher validation or, in versions without that validation, create a plaintext database.

This replaces #531 and #532 with one main fix that both isolates WalletKit's SQLite engine and recovers affected plaintext databases without deleting their contents. The v0.21.4 backport remains separate in #535.

What

• Compile the existing checksum-pinned sqlite3mc amalgamation with its SQLite API local to one C translation unit and expose only WalletKit-prefixed wrappers to Rust.
• Preserve the existing sqlite3mc version, ChaCha20 cipher settings, raw-key encoding, and encrypted on-disk format.
• Open databases using this sequence:
  1. Select and verify the ChaCha20 cipher.
  2. Probe sqlite_master without a key.
  3. If readable, treat the database as plaintext, checkpoint/switch it out of WAL mode, and encrypt it in place with PRAGMA rekey.
  4. If and only if the probe returns SQLITE_NOTADB, apply PRAGMA key and verify the encrypted database.
  5. Apply the normal journal and connection policy after the database is readable.
• Fail without modifying plaintext data when the connection is read-only or cannot safely leave WAL mode.
• Add native link-order regressions and a frozen pre-fix encrypted fixture.

The WAL transition is required because sqlite3mc does not support rekeying in WAL journal mode. Switching to DELETE checkpoints any plaintext WAL before encryption; WalletKit restores WAL after a successful native migration.

Validation

• nix develop --command cargo test -p walletkit-sqlite -p walletkit-db --locked: 24 tests pass, including plaintext-WAL migration, record preservation, wrong-key rejection, read-only preservation, frozen encrypted bytes, envelope bytes, and content IDs.
• nix develop --command cargo clippy -p walletkit-sqlite -p walletkit-db --all-targets --all-features --locked -- -D warnings: passes.
• nix develop --command bash crates/walletkit-sqlite/examples/test_native_linking.sh: passes with Apple SQLite linked in both orders.
• nix develop --command bash crates/walletkit-sqlite/examples/test_native_linking.sh release: passes with the optimized/LTO profile in both orders.
• Rust formatting and git diff --check: pass.

Risk and rollout

This changes native encrypted-storage linkage and performs an in-place encryption migration. Keep this PR draft for deliberate storage and cryptography review.

• Confirm existing encrypted-store upgrade/downgrade behavior and real initialization, proof, and credential flows before distribution.
• Validate the migration in the affected host app using synthetic/fresh test accounts first.
• Add the native debug and release probes to macOS CI with a bounded timeout.
• Use a narrow rollout and retain the previous SDK artifact with a tested data-preserving rollback plan. Downgrading after migration requires the previous build to link the intended encrypted engine; a build that resolves WalletKit calls to system SQLite will no longer read the migrated store.

---

Note

High Risk
Changes native linkage for all encrypted storage and performs automatic in-place encryption of previously plaintext databases; rollout and rollback depend on builds always using the isolated engine.

Overview
Native WalletKit SQLite calls now go through walletkit_sqlite3_* wrappers in a new native_sqlite.c translation unit that keeps the sqlite3mc amalgamation statically linked, so host apps cannot resolve WalletKit’s sqlite3_* symbols to system SQLite at link time. The build compiles that shim instead of the amalgamation .c directly; Rust FFI uses #[link_name = "walletkit_sqlite3_…"].

open_encrypted drops the read_only flag and always opens read-write. After pinning ChaCha20, it probes sqlite_master: readable plaintext (including WAL) is checkpointed to DELETE journal mode, rekeyed in place, then verified; only SQLITE_NOTADB triggers the normal PRAGMA key path so I/O or corruption cannot start a bogus migration. WAL is restored afterward on native as before.

Callers (Vault, cache tests, OPFS tests) use the slimmer API. Docs describe isolation and plaintext recovery; tests add plaintext-WAL → encrypted migration with data preserved.

^{Reviewed by Cursor Bugbot for commit 60ae391. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR isolates WalletKit’s native sqlite3mc engine and encrypts affected plaintext SQLite stores in place.

Changes:

• Adds prefixed native wrappers and updates Rust FFI/build integration.
• Detects plaintext/WAL stores, rekeys them, and verifies migration.
• Updates callers, documentation, and migration coverage.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Summary and final findings
crates/walletkit-sqlite/src/opfs.rs	Updates OPFS callers for the revised API.
crates/walletkit-sqlite/src/native_sqlite.c	Adds the isolated native SQLite wrapper. [P1; moderate; 1 vote] Add the claimed native link-order regression probes and CI coverage.
crates/walletkit-sqlite/src/ffi.rs	Binds prefixed native symbols and SQLITE_NOTADB.
crates/walletkit-sqlite/src/cipher.rs	Implements plaintext detection, WAL migration, rekeying, and verification. [P1; moderate; 1 vote] Serialize or retry migration transitions. [P2; nit; 1 vote] Assert the post-migration journal mode is WAL. [P1; moderate; 1 vote] Add a frozen pre-fix encrypted fixture.
crates/walletkit-sqlite/README.md	Documents native isolation and plaintext migration.
crates/walletkit-sqlite/build.rs	Compiles the native wrapper. [P1; nit; 1 vote] Include the claimed link-order regression artifacts or correct the description.
crates/walletkit-db/src/vault.rs	Updates vault opening for the revised storage API.
crates/walletkit-core/src/storage/cache/mod.rs	Updates cache test callers.

Suppressed comments (5)

crates/walletkit-sqlite/build.rs:113

• [P1] The PR description says this change adds and validates crates/walletkit-sqlite/examples/test_native_linking.sh in debug and release, but that probe/script is absent from this checkout; this build change only compiles the wrapper. Please include the claimed link-order regression artifacts (and CI wiring if required) or correct the description, because the native isolation guarantee is otherwise not reproducible from the submitted tree.

        .file("src/native_sqlite.c")

crates/walletkit-sqlite/src/cipher.rs:119

• [P1] Serialize or retry this migration transition before returning SQLITE_BUSY. PRAGMA journal_mode = DELETE requires an exclusive lock, but the storage initialization path opens Vault/CacheDb without holding the cross-process storage lock around these calls, so two simultaneous starts on a plaintext WAL store can make one startup fail even though the other migration succeeds. Add a dedicated migration lock or a bounded busy retry with clear recovery semantics; otherwise the new automatic recovery is race-prone.

            ensure_journal_mode(conn, "DELETE")?;

crates/walletkit-sqlite/src/cipher.rs:524

• [P2] The migration test verifies that records survive and that a later reopen works, but it never asserts the promised post-migration journal policy. A regression that leaves the connection/database in DELETE mode would still pass while changing the concurrency and durability behavior of every recovered store; query PRAGMA journal_mode after this open and assert the native result is wal.

            let conn = open_encrypted(&path, &key).expect("migrate plaintext");

crates/walletkit-sqlite/src/cipher.rs:120

• [P1] The new migration path is not covered by a frozen pre-fix encrypted fixture. The existing round-trip test creates and reopens the database with the same current implementation, so a change to raw-key encoding, cipher selection, or on-disk bytes could make both sides agree while breaking deployed stores. Add a checked-in pre-fix encrypted fixture and assert that the current code can open and read it before merging this compatibility-sensitive change.

            apply_rekey(conn, k_intermediate)?;

crates/walletkit-sqlite/src/native_sqlite.c:12

• [P1] This diff does not include the native link-order regression it claims to add. native_sqlite.c is the security boundary, but the checkout has no native host/probe, symbol inspection, or CI job; Rust unit tests link only WalletKit's own symbols and cannot detect a host system SQLite satisfying unprefixed dependencies. Please add the debug/release probes in both link orders before relying on this isolation, otherwise a future linker change can silently reintroduce plaintext-store creation.

#include "sqlite3mc_amalgamation.c"

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.