Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ determines whether text boxes get drawn and additional settings `padding`,
Data Explorer (#536).
- The Positron extension now registers a bundled agent skill, so agents
automatically discover how to write and run ggsql queries (#536).
- In a Positron console, a query result can be opened in the Data Explorer
from an "Open in Data Explorer" link under its table, with sorting,
filtering, profiles, export and Convert to Code. The kernel serves this
through a frontend-neutral results API, reached with `call_method` on the
UI comm and turned on by the frontend (`--keep-results`), which keeps
results in an in-memory cache of its own within one byte budget
(`--results-max-bytes`, default 1 GB, or the `ggsql.resultsMemory`
setting) and never changes the session's reader or dialect (#595).

### Changed

Expand All @@ -40,9 +48,17 @@ determines whether text boxes get drawn and additional settings `padding`,
- The CLI now always prints query warnings (e.g. extra `VISUALISE`/`TABULATE`
statements being ignored) to stderr, not only with `-v`.
- Replaced `CLAUDE.md` files with `AGENTS.md` files.
- A Positron console now shows a long query result that can be opened in the
Data Explorer as its first 10 rows, followed by a count of the rest, rather
than up to 100; a result of up to 20 rows is still shown whole, and one that
could not be kept still shows up to 100. The number is set by the
`ggsql.consoleRows` setting (#595).

### Fixed

- A timestamp with a time zone is now written as a timestamp in UTC
(`2024-01-02T03:04:05Z`) wherever ggsql shows a value as text, rather than
as the name of its Arrow type (#595).
- `SCALE DISCRETE` now accepts integer columns, matching the documented ability
to force numeric data into a discrete scale. Integer columns mapped to
grouping aesthetics (e.g. `fill` in `DRAW bar`) can now be declared discrete
Expand Down
5 changes: 5 additions & 0 deletions doc/get_started/tooling/positron-vscode.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@ Executing ggsql statements containing [`VISUALISE`](../../syntax/clause/visualis

![](./screenshots/positron-plots-pane.png)

## Data Explorer

A query without `VISUALISE` shows its result as a table in the console, followed by an "Open in Data Explorer" link. A result of more than 20 rows shows only its first 10, with a count of the rest. To show more or fewer, change the `ggsql.consoleRows` setting: a result of up to twice that many rows is shown in full. The change applies to consoles started afterwards. Clicking the link opens the result in Positron's Data Explorer, where you can sort, filter, and profile it, and use "Convert to Code" to turn your sorts and filters back into SQL.

The Data Explorer works on a snapshot of the result, so it doesn't change when the underlying tables do. Run the query again to see new data. The console keeps recent results available this way, up to 1 GB in all by default; the `ggsql.resultsMemory` setting changes that, for consoles started afterwards. Results open in a Data Explorer are always kept, and older ones make way for new ones, so the link under an old result may report that it is no longer available. If the results you have open use all of that memory, a new result is shown in the console without a link, and up to 100 of its rows, and sorting or filtering a large one may ask you to close another Data Explorer tab first.

## Editor and cells

Expand Down
31 changes: 29 additions & 2 deletions ggsql-jupyter/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,13 @@ ggsql-jupyter/
│ ├── kernel.rs Jupyter messaging loop (ZMQ, message dispatch)
│ ├── executor.rs Runs queries via ggsql::Reader, returns rendered output
│ ├── connection.rs Reader lifecycle, connection-string parsing
│ ├── data_explorer.rs Positron data-explorer comm channel
│ ├── data_explorer.rs Positron data-explorer comm, for tables opened from the Connections pane
│ ├── results/ Frontend-neutral results API over recent console results
│ │ ├── mod.rs ResultStore: pending results, the results cache, views
│ │ ├── view.rs One view: filters, sort, order table, pages, profiles, export
│ │ ├── filter.rs Filter and SortKey, and the SQL for them
│ │ ├── methods.rs The API as named JSON calls (`result_open`, `result_page`, …)
│ │ └── export.rs A selection as CSV / TSV / HTML
│ ├── display.rs Output formatting (image bundles, SQL → HTML table)
│ ├── message.rs Jupyter message structs (ZMQ frames, HMAC signing)
│ └── util.rs
Expand Down Expand Up @@ -173,9 +179,30 @@ So `RenderHints::canvas` picks a height (golden ratio, close to ggplot2's defaul

## Positron-specific bits

- `data_explorer.rs` implements Positron's data-explorer comm channel (registered query results become explorable tables).
- `data_explorer.rs` implements Positron's data-explorer comm for a table opened from the Connections pane, querying the session's own database.
- The companion VS Code extension (`ggsql-vscode/`) ships a copy of this binary and also discovers installed ones via the `ggsql.kernelPath` setting, the Jupyter kernelspec directories, the native install locations, or `PATH`. It runs each one with `--version` to name the runtime it registers, so **keep `--version` working**: a kernel that does not answer it is still offered, but without a version in the picker. See [Finding the kernel](../ggsql-vscode/AGENTS.md#finding-the-kernel).

### The results API

`results/` keeps recent query results and answers a small, frontend-neutral set of calls on them (`methods.rs` documents each: `result_open`, `result_set_filters`, `result_set_sort`, `result_page`, `result_profile`, `result_export`, `result_sql`, `result_close`). Nothing in it knows about Positron: the ggsql extension translates Positron's Data Explorer protocol into these calls (see [`ggsql-vscode/AGENTS.md`](../ggsql-vscode/AGENTS.md)). The kernel's only Positron-specific part is the route: a `call_method` request on the `positron.ui` comm — what an extension's `session.callMethod(name, params)` sends — whose method starts with `result_` goes to `methods::call`, and its answer or error comes back as the JSON-RPC reply (errors with `plot::comm::RpcError`'s encoding). A `call_method` for any other name is an error, like an unknown plot-comm method, so a caller expecting something the kernel lacks fails visibly; any other UI request gets `result: null`.

- **A console cuts a long table short only when it was kept** (`RenderHints::kept`): only then does a link to the rest follow it. A result that could not be kept — the budget full, or a frontend that did not ask — shows up to 100 rows, as anywhere else.
- **Which results are kept.** Only when the frontend asks, with `--keep-results` or `GGSQL_KEEP_RESULTS` — the extension sets it for a console session whose results it can open — so a session nobody explores holds nothing beyond what it displays. Then every tabular result of a non-silent execution is kept, and its key — unguessable hex, so a stale link from another kernel process names nothing — goes in the `execute_result`'s `metadata` as `{"ggsql": {"result": key}}`; the kernel prints no link itself. Unopened results are held as their Arrow data. The store and its DuckDB are created on first use, so a session that keeps nothing opens no database for them.
- **One byte budget covers everything kept**, `--results-max-bytes` / `GGSQL_RESULTS_MAX_BYTES` (default 1 GB, by Arrow's estimate of the data): unopened results, the cache's tables, pinned ones included, and every view's order table. Making room drops closed results first, then the oldest unopened ones — and only once it is known the new bytes fit beside what open views hold, so a result too big to keep drops nothing. An open view is never pulled from under its frontend, so when open views alone fill the budget a new result is not kept (it is shown without a key), and opening a result or a sort or filter that needs room fails with an error saying so. Opening reserves the copy's row column as well as the result, and any rebuild of an order table reserves a new one, including clearing the sort of a filtered view or the filters of a sorted one.
- **The cache's DuckDB has a hard memory limit of twice the budget** (`memory_limit_for`, at least 64 MB). The budget counts what is kept, by Arrow's estimate; DuckDB holds a table less compactly than that and needs working memory on top to sort and profile, so the limit is what bounds the process. A budget too large to set as a limit leaves DuckDB's default, most of the machine's memory.
- **Opening copies the result into the results cache.** A `CacheTables` (the caching reader's bookkeeping, `ggsql::reader::cache`) on its own in-memory DuckDB, which `ResultStore` creates itself. The session's reader is never wrapped, so the user's dialect is untouched. The copy adds `__ggsql_row__`, each row's position in the query's output, and drops the Arrow data, so a result is held once. It stays **pinned** while any view of it is open; once closed, it goes when the budget needs the room, and reopening before then reuses it. A result can have several views, each with its own filters and sort.
- **Views end with the UI comm.** Closing it closes every view, since nothing could close them once it has gone, and so does opening a new one: a frontend that reloaded or crashed opens a new UI comm without closing the old. That is what releases its pins.
- **The cache is always DuckDB.** `ResultStore` opens it itself, so `results/` writes DuckDB SQL for it (`DESCRIBE`, `SET`, ANSI quoting through `naming::quote_ident`, internal names through `ggsql::naming`). Type names, booleans and quantiles come from the cache reader's own dialect, which is DuckDB's; the filter and sort SQL in `filter.rs` takes a dialect because `result_sql` uses the same code to write SQL for the session's database.
- **Nothing is written to disk.** The cache sets `temp_directory = ''`, so DuckDB cannot spill: a query that needs more memory than the limit fails with DuckDB's out-of-memory error, and the view it was for is left as it was.
- **Sorting and filtering build an order table** of `(pos, __ggsql_row__)` per view (`view.rs`); a page joins it back to the result by `pos`, so scrolling never sorts again. Ties break on `__ggsql_row__`, so paging is stable and clearing the sort restores the query's own order. An unfiltered, unsorted view has none and pages by `__ggsql_row__` directly. A new sort or set of filters is built under a new table name and only then replaces the old one, so one that fails leaves the view as it was.
- **Profiles are bounded too**: at most `MAX_FREQUENCY_VALUES` values in a frequency table, `MAX_QUANTILES` quantiles (each its own query) and `MAX_HISTOGRAM_BINS` bins.
- **A page is bounded, and so is an export.** A page holds at most `MAX_PAGE_ROWS`, and an export at most `MAX_EXPORT_ROWS` (100,000), since it is built as text on the message loop, outside DuckDB's memory limit; copying a whole column of a longer result is an error that says so. For both, a range is cut short at the end of the view, and an index outside it is an error, so a cell never lands in another row's place. Scattered indices fetch just those rows.
- **Filter values arrive as strings and are checked against the column's kind** before they reach the SQL: a number must parse as a finite number, a boolean as true/false, a date, datetime or time as ISO 8601 without an offset (it becomes the dialect's typed literal, `sql_date_literal` and friends, since not every backend compares a string with a temporal column), anything else becomes a quoted literal. A filter that cannot be built, or that the database rejects (tried on its own to find out which), is left out and its message returned in `errors`; the rest still apply. Filters are always ANDed. Identifiers, string and boolean literals, `LIKE` and regular expressions all come from the dialect the SQL is for (`SqlDialect::sql_string_literal`, `sql_like`, `sql_regex_match` — `None` where a backend has no regular expressions, as SQLite and SQL Server), so the same filter is written correctly for a backend that reads backslashes in literals, has no `ESCAPE` clause, or has more `LIKE` wildcards. Sort keys put NULLs last in either direction through `sql_order_term`, as the cache's DuckDB does, so exported SQL sorts as the view did.
- **Kinds are type names matched exactly** (`kind_of`): DuckDB's, and the names other backends report, which the Connections pane's `data_explorer.rs` shares. A column type that only contains `INT`, like `POINT`, is not an integer, an `ENUM` is a string, and ClickHouse's `Nullable(…)` and `LowCardinality(…)` are the type they wrap.
- **`result_sql` writes SQL for the session's database**, not the cache: the cell's statements before the last as they were (the whole cell is one statement when it does not parse cleanly), then the last as a derived table with the view's filters and sort, in the session's dialect. A last statement that is not a query (`DESCRIBE`, `SHOW`, …) cannot be a derived table, so it is written as it was, with a comment saying the filters and sort were left out. It is in the kernel because only the kernel knows that dialect. A filter it cannot express there, or one on a column whose name the query repeats (renamed `a_1` in the cache, but ambiguous in SQL), is left out with a `--` comment.

Measured at 10M rows (617 MB of Arrow data) on an Apple Silicon laptop: opening takes about 3.4 s, a sort 0.4–0.6 s, a page 20 ms, and memory levels off near 2 GB across repeated open/close cycles. Sorting the Arrow data directly avoids the copy but takes 2–5 s per stable sort, which is why the cache is DuckDB.

## Build & install

```sh
Expand Down
2 changes: 1 addition & 1 deletion ggsql-jupyter/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ arrow = { workspace = true }
tokio = { workspace = true, features = ["sync"] }

# CLI
clap = { workspace = true }
clap = { workspace = true, features = ["env"] }

# Serialization
serde = { workspace = true }
Expand Down
55 changes: 16 additions & 39 deletions ggsql-jupyter/src/data_explorer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -917,46 +917,19 @@ fn sql_type_to_display(type_name: &str) -> &'static str {
}
}

// Simple type names (DuckDB, PostgreSQL, SQLite, etc.)
let upper = type_name.to_uppercase();
let upper = upper.as_str();

if upper.contains("INT") {
return "integer";
}
if upper.contains("FLOAT")
|| upper.contains("DOUBLE")
|| upper.contains("REAL")
|| upper.contains("NUMERIC")
|| upper.contains("DECIMAL")
{
return "floating";
}
if upper.contains("BOOL") {
return "boolean";
}
if upper.contains("TIMESTAMP") || upper.contains("DATETIME") {
return "datetime";
}
if upper.contains("DATE") {
return "date";
}
if upper.contains("TIME") {
return "time";
}
if upper.contains("CHAR")
|| upper.contains("TEXT")
|| upper.contains("STRING")
|| upper.contains("VARCHAR")
|| upper.contains("CLOB")
{
return "string";
}
if upper.contains("BLOB") || upper.contains("BINARY") || upper.contains("BYTE") {
return "string";
// Simple type names, matched exactly as the results API matches them.
match crate::results::kind_of(type_name) {
"unknown" => {
// Binary data is shown as text here.
let upper = type_name.trim().to_uppercase();
let base = upper.split('(').next().unwrap_or("").trim();
match base {
"BLOB" | "BYTEA" | "BINARY" | "VARBINARY" | "BYTES" | "LONGBLOB" => "string",
_ => "unknown",
}
}
kind => kind,
}

"unknown"
}

/// Clean up a raw type name for display in the schema response.
Expand Down Expand Up @@ -1014,5 +987,9 @@ mod tests {
assert_eq!(sql_type_to_display("TIME"), "time");
assert_eq!(sql_type_to_display("BLOB"), "string");
assert_eq!(sql_type_to_display("UNKNOWN_TYPE"), "unknown");
// Matched exactly, not by substring.
assert_eq!(sql_type_to_display("POINT"), "unknown");
assert_eq!(sql_type_to_display("INT UNSIGNED"), "integer");
assert_eq!(sql_type_to_display("character varying(20)"), "string");
}
}
Loading
Loading