Skip to content

feat(sqlite3): translate dot-commands (.tables, .schema, .mode, .read, ...)#249

Open
subsetpark wants to merge 2 commits into
vercel-labs:mainfrom
flowglad:upstream-pr/sqlite3-dot-commands
Open

feat(sqlite3): translate dot-commands (.tables, .schema, .mode, .read, ...)#249
subsetpark wants to merge 2 commits into
vercel-labs:mainfrom
flowglad:upstream-pr/sqlite3-dot-commands

Conversation

@subsetpark
Copy link
Copy Markdown
Contributor

Summary

Real sqlite3 ships a family of dot-commands that are a feature of the CLI, not the library. sql.js doesn't implement them, so agent scripts pasted verbatim from a real sqlite3 session hit near ".": syntax error on the first dot-command (.tables, .schema, .mode csv, .read seed.sql, …).

This PR adds a host-side preprocessor (commands/sqlite3/dot-commands.ts) that runs before SQL reaches the worker. No worker changes needed — the worker keeps executing pure SQL.

# Before this PR — every line of an agent's pasted sqlite3 session fails:
$ sqlite3 :memory: ".headers on
> CREATE TABLE pets(name TEXT, kind TEXT);
> INSERT INTO pets VALUES('Fido','dog'),('Whiskers','cat');
> .mode csv
> SELECT * FROM pets;
> .tables"
Error: near ".": syntax error

# After this PR — pasted scripts work:
$ sqlite3 :memory: ".headers on
> CREATE TABLE pets(name TEXT, kind TEXT);
> INSERT INTO pets VALUES('Fido','dog'),('Whiskers','cat');
> .mode csv
> SELECT * FROM pets;
> .tables"
name,kind
Fido,dog
Whiskers,cat
pets

How the preprocessor works

Char-level scanner with state for SQL string literals ('…', "…", including the doubled-quote escapes ''/"") and comments (-- …, /* … */). A dot-command is recognized only at a boundary — start-of-input, just after ;, or just after \n — and only outside strings/comments. This lets 'a\n.tables' round-trip intact and lets .headers on; .mode csv; CREATE TABLE… be three tokens on one line.

Each recognized dot-command resolves to one of seven outcomes:

Outcome Dot-commands
SQL replacement .tables, .schema, .indexes/.indices, .databases, .helpsqlite_master/PRAGMA queries (shell-glob */? → SQL-LIKE %/_)
Formatter mutation .headers/.header, .mode, .separator, .nullvalue — adjusts output state for downstream SQL; bad args surface real-sqlite3-shaped errors (Error: unknown mode: parquet). Last write wins.
.read FILE Recursive scan of the included file, sharing formatter state, capped at MAX_READ_DEPTH to prevent loops
.quit / .exit Terminates preprocessing; subsequent input is dropped (including in a parent scanner)
Silent drop .echo, .timer, .changes, .bail, .show, .eqp, .width, .prompt, .print, .explain — recognized and discarded
Not-implemented family .dump, .save, .backup, .import, .clone, .restore, .open, .output, .shell, .system, .cd, .load, .iotrace, .log, .excel → translates to SELECT 'Error: …' AS error so the message rides in stdout in script-order without aborting surrounding SQL
Passthrough Unknown dot-commands left verbatim — sql.js produces its native syntax error rather than us inventing a CLI-shaped one

Dot-command errors are routed in-band to stdout alongside SQL errors when -bail is unset (matches real sqlite3's single-channel reporting); with -bail they go to stderr with exit 1.

Two paired CLI flags

  • -init FILENAME — read SQL from FILENAME before main SQL (matches real sqlite3; pairs naturally with .read)
  • -batch — accepted as a no-op since just-bash is always non-interactive

What's not in this PR

This is the host-side preprocessor only. There's a small Bun-specific worker fix in our internal fork (coerceDbBuffer handles a Bun worker_threads structured-clone regression that surfaces null as a zero-length ArrayBuffer) that I'd be happy to send as a separate PR if useful — it's orthogonal to dot-commands and Bun-runtime-specific.

Test plan

  • pnpm --filter just-bash exec vitest run src/commands/sqlite3/247/247 pass (16 test files, including 4 new files from this PR; existing worker untouched)
  • pnpm --filter just-bash typecheck — clean
  • pnpm --filter just-bash build — clean

New test files

  • sqlite3.dot-commands.test.ts — preprocessor coverage (boundaries, string/comment skipping, all seven outcome categories, .read recursion, formatter precedence)
  • sqlite3.invocation-shapes.test.ts — pins the canonical agent invocation patterns (sqlite3 :memory: "SQL", echo SQL | sqlite3, sqlite3 -cmd ... :memory: "...", etc.)
  • sqlite3.flags.test.ts-init / -cmd / -batch flag behavior
  • sqlite3.sql-features.test.ts — generic SQL features through the pipeline

Docs

packages/just-bash/docs/sqlite3-invocation-shapes.md catalogues the CLI invocation shapes agents reach for, with the equivalent results they should see.

Background

We've been carrying this in flowglad/just-bash for a while (fork notes) because our agent pastes sqlite3 CLI sessions verbatim. Now that #190 fixed the bundled-worker shipping bug (thank you!), the dot-command preprocessor is the remaining gap between just-bash's sqlite3 and the real CLI's surface area. We'd love to retire the carry and use upstream directly.

🤖 Generated with Claude Code

@subsetpark subsetpark requested a review from cramforce as a code owner May 19, 2026 17:33
@vercel
Copy link
Copy Markdown

vercel Bot commented May 19, 2026

@subsetpark is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

…, ...)

Real sqlite3 ships a family of dot-commands that are a feature of the CLI,
not the library. sql.js doesn't implement them, so agent scripts pasted
verbatim from a real sqlite3 session hit `near ".": syntax error` on the
first dot-command (.tables, .schema, .mode csv, .read seed.sql, ...).

Add a host-side preprocessor (commands/sqlite3/dot-commands.ts) that runs
before SQL reaches the worker. The scanner is char-level with state for
SQL string literals ('...', "...", including the doubled-quote escapes)
and comments (-- ..., /* ... */); a dot-command is recognized only at a
boundary (start-of-input, after `;`, or after `\n`) and only outside
strings/comments — `'a\n.tables'` round-trips intact, and
`.headers on; .mode csv; CREATE TABLE...` is three tokens on one line.

Each recognized dot-command resolves to one of:

- SQL replacement — .tables/.schema/.indexes/.databases/.help translate
  to sqlite_master/PRAGMA queries (with shell-glob → SQL-LIKE conversion).
- Formatter mutation — .headers, .mode, .separator, .nullvalue adjust
  output state for downstream SQL; bad args surface real-sqlite3-shaped
  errors (e.g. "Error: unknown mode: parquet"). Last write wins.
- .read FILE — recursive scan of the included file, sharing formatter
  state, capped at MAX_READ_DEPTH to prevent loops.
- .quit / .exit — terminate preprocessing; subsequent input is dropped.
- Silent drop — .echo, .timer, .changes, .bail, .show, .eqp, .width,
  .prompt, .print, .explain are recognized and discarded.
- Not-implemented family — .dump, .save, .backup, .import, .clone,
  .restore, .open, .output, .shell, .system, .cd, .load, .iotrace,
  .log, .excel translate to `SELECT 'Error: ...' AS error` so the
  message rides in stdout in script-order without aborting SQL.
- Passthrough — unknown dot-commands are left verbatim so sql.js
  produces its native syntax error.

Pairs with two new CLI flags:

- `-init FILENAME` — read SQL from FILENAME before main SQL.
- `-batch` — no-op (just-bash is always non-interactive).

Dot-command errors are routed in-band to stdout alongside SQL errors when
-bail is unset (matches real sqlite3's single-channel reporting); with
-bail they go to stderr with exit 1.

Tests: 247 sqlite3 tests pass against the existing worker (no worker
changes needed — preprocessor runs entirely host-side). New test files:
- sqlite3.dot-commands.test.ts: comprehensive preprocessor coverage
- sqlite3.invocation-shapes.test.ts: pinned agent invocation patterns
- sqlite3.flags.test.ts: -init / -cmd / -batch flag behavior
- sqlite3.sql-features.test.ts: generic SQL features through the pipeline

Docs: packages/just-bash/docs/sqlite3-invocation-shapes.md catalogues the
canonical CLI invocation shapes agents use.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a host-side sqlite3 “dot-command” preprocessor to bring just-bash’s sqlite3 CLI closer to real sqlite3 behavior when users paste interactive sessions (e.g. .tables, .schema, .mode, .read). This keeps the worker executing pure SQL while the host rewrites/handles CLI-only meta-commands.

Changes:

  • Introduces preprocessDotCommands() to translate/handle sqlite3 dot-commands (SQL replacement, formatter mutations, .read, .quit, etc.) before SQL reaches the worker.
  • Adds CLI flag support for -init <file> (pre-load script) and accepts -batch as a no-op.
  • Adds extensive Vitest coverage and a docs catalogue of agent invocation shapes / supported features.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
packages/just-bash/src/commands/sqlite3/sqlite3.ts Wires in dot-command preprocessing and adds -init / -batch handling + dot-error routing.
packages/just-bash/src/commands/sqlite3/dot-commands.ts Implements char-level scanner + dot-command translation, .read recursion, and formatter mutations.
packages/just-bash/src/commands/sqlite3/sqlite3.dot-commands.test.ts New test suite covering dot-command behavior and scanner edge cases.
packages/just-bash/src/commands/sqlite3/sqlite3.flags.test.ts New tests for -init and -batch and their interaction with dot-commands.
packages/just-bash/src/commands/sqlite3/sqlite3.invocation-shapes.test.ts New tests pinning common real-world invocation patterns.
packages/just-bash/src/commands/sqlite3/sqlite3.sql-features.test.ts New tests pinning important SQL feature support through the wrapper.
packages/just-bash/docs/sqlite3-invocation-shapes.md Documents supported/observed invocation shapes, dot-commands, and known limitations.
.changeset/sqlite3-dot-commands.md Changeset entry describing the new dot-command support and flags.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/just-bash/src/commands/sqlite3/sqlite3.ts Outdated
Comment thread packages/just-bash/src/commands/sqlite3/sqlite3.ts Outdated
…ecode

Use `??` instead of `||` when sourcing SQL so an explicit empty positional
SQL arg ("") is "provided but empty" and does not silently fall through to
piped stdin. Decode/trim stdin once and reuse it in the no-SQL guard so both
checks agree on the same value. Addresses Copilot review feedback on vercel-labs#249.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants