Repository navigation
Conversation
Queries copied from the QuestDB server test suites live in tests/fixtures/server-queries.json, each with its source file and line. tests/server-queries.test.ts checks every entry four ways: it parses as the expected number of statements, toSql matches the reviewed output and round-trips, autocomplete suggests each next token, and the formatter preserves it. The token-by-token autocomplete walkthrough moves from docs-autocomplete.test.ts to tests/helpers so both suites share it.
The server accepts four literal forms that sql-parser misread. Most of them were silent: the query parsed without errors but with a different meaning. 0x7ee6... long256 constant was `0 AS x7ee6...` or split in two 1.5f, 1e36d float/double suffix was `1.5 AS f` or split in two E'abc' PostgreSQL string was `E AS abc` or split in two timestamp '..' typed literal (cast) was a column with an alias, or split Lexer: HexLiteral, FloatLiteral and EscapeStringLiteral tokens. `d` is only a double suffix after an exponent, so `1d` stays a day duration. Parser: the typed literal hangs off identifierExpression as an optional string after a castable type name (the server's processPgTimestampCast list plus nameTypeMap aliases). A separate primaryExpression alternative would be simpler, but autocomplete ignores GATEs, so it broke ARRAY[ suggestions and offered every type keyword wherever a value can start. `timestamp [ns] with time zone '...'` is its own small rule; content assist drops its leading TIMESTAMP tokens so only WITH after `timestamp` is new. Tests: 26 server queries (questdb + questdb-enterprise test suites) in the parity fixture, checked for parse, round trip, autocomplete and formatter, plus tests/literal-forms.test.ts for lexer, AST and autocomplete edge cases. Against 53,975 SQL strings from the server tests and docs: 218 now parse, 25 round trips are corrected, and the only newly rejected input is `cancel query 12.01f`, which the server rejects too.
Fixes a regression on master and adds the grouping-sets syntax of questdb/questdb#6853. The server (SqlParser.parseGroupByColumns, parseExplicitGroupingSets, hasGroupingSyntaxLookahead and the SAMPLE BY branch) accepts: GROUP BY ROLLUP(a, ...) | CUBE(a, ...) GROUP BY GROUPING SETS ((a, b), (a), ()) every set parenthesised GROUP BY expr, ..., ROLLUP(...) | CUBE(...) ROLLUP/CUBE last SAMPLE BY <interval> ROLLUP(...) | CUBE(...) | GROUPING SETS (...) [FROM ... TO ...] [FILL(...)] [ALIGN TO ...] Nothing may follow ROLLUP/CUBE/GROUPING SETS, GROUPING SETS must be the only GROUP BY item, and keys must be column references (ensureColumnReferences). cube, rollup, grouping and sets are not reserved: they are grouping syntax only when `(` (ROLLUP/CUBE) or SETS (GROUPING) follows. Before: #11 added the four words as reserved keywords with no grammar, so `SELECT rollup, cube, sets, grouping FROM t`, `CREATE TABLE t (sets INT, cube INT)` and `SELECT grouping(a) FROM t GROUP BY a` failed (they worked in 0.1.21), and every ROLLUP/CUBE/GROUPING SETS query failed. Now: - tokens.ts: Cube, Grouping, Rollup, Sets are non-reserved (IDENTIFIER_KEYWORD_NAMES), which alone fixes the regression. - parser: groupByClause takes groupingSetsSpec when LA(1..2) is ROLLUP( / CUBE( / GROUPING SETS, otherwise plain expressions with an optional trailing `, ROLLUP(...)|CUBE(...)`. The LA gates keep `GROUP BY rollup`, `GROUP BY a, rollup` and `GROUP BY grouping(x)` as plain keys, and stop `GROUP BY a, grouping sets (...)` (rejected by the server) from being read as `grouping` plus a second statement. sampleByClause takes an optional groupingSetsSpec right after the interval. - AST: new GroupingSets node {kind: rollup|cube|groupingSets, columns | sets}; SelectStatement.groupingSets holds the GROUP BY part (plain keys stay in groupBy), SampleByClause.groupingSets the SAMPLE BY part. Existing fields are unchanged. - toSql prints ROLLUP(...), CUBE(...), GROUPING SETS ((...), ()). - Autocomplete offers ROLLUP/CUBE/GROUPING after GROUP BY and after the SAMPLE BY interval, ROLLUP/CUBE after a plain GROUP BY key, SETS after GROUPING, and columns inside the parentheses. Not done: the server also runs parseGroupByColumns for a GROUP BY after `FROM t PIVOT (...)`; sql-parser does not accept any GROUP BY after a PIVOT clause (clause order), so that is left for a PIVOT change. The GROUP BY inside PIVOT (...) only takes plain keys on the server too. Tests: 28 server queries (GroupingSetsTest, LateralJoinTest, SampleByFillTest) in the parity fixture, plus tests/grouping-sets.test.ts (identifiers, AST, round trip, rejected forms, autocomplete). Red before: 128 failing (84 fixture checks + 44 focused); after: all 303 pass, full suite 15491 passed. Corpus (53,975 server/docs SQL): 226 fixed, 0 round trips changed, 0 regressed, 0 throws, formatter meaning unchanged.
…ut ENABLE Adds the dedup syntax of questdb/questdb#7885 and fixes older dedup gaps (reports §1.2). The server accepts (SqlParser CREATE TABLE dedup block, SqlCompilerImpl.alterTableDedupEnable and the ALTER TABLE dedup branch): CREATE TABLE / CTAS ... (DEDUP | DEDUPLICATE) (UPSERT | SKIP) KEYS(c, ...) [FORMAT PARQUET | NATIVE] [OWNED BY ...] ALTER TABLE t (DEDUP | DEDUPLICATE) DISABLE ALTER TABLE t (DEDUP | DEDUPLICATE) [ENABLE] (UPSERT | SKIP) KEYS(c, ...) ENABLE has been optional since v7.3. SHOW CREATE TABLE prints `DEDUP SKIP KEYS(...)`. `skip` and `deduplicate` are not reserved (neither is in SqlKeywords.KEYWORDS; upsert is, and stays reserved). Before: every SKIP form failed ("Expecting Upsert"), ALTER ... DEDUP UPSERT KEYS without ENABLE failed, ALTER ... DEDUPLICATE failed, and `... WAL DEDUPLICATE UPSERT KEYS(ts)` was silently split into three statements. `skip` was a reserved token with no grammar, so `SELECT skip FROM t` failed. Now: - keywords.ts: new `deduplicate`; tokens.ts: Deduplicate and Skip are non-reserved (IDENTIFIER_KEYWORD_NAMES). - parser: new rules dedupKeyword (DEDUP | DEDUPLICATE) and dedupModeKeys ((UPSERT | SKIP) KEYS(...)), shared by the CREATE TABLE dedupClause and the ALTER TABLE dedup action, where ENABLE is now optional. - AST: new optional `dedupMode: "upsert" | "skip"` on CreateTableStatement (set whenever dedupKeys is) and on DedupEnableAction. Existing fields are unchanged. - toSql prints the mode instead of the hard-coded UPSERT (absent mode still prints UPSERT). DEDUPLICATE is printed as DEDUP and ALTER always prints ENABLE (canonical form; same meaning on the server). The existing output spacing (`KEYS (..)` for ALTER) is kept. - Formatter: DEDUPLICATE is a clause/action phrase like DEDUP. - Autocomplete: UPSERT and SKIP after DEDUP / DEDUPLICATE, plus ENABLE and DISABLE after ALTER TABLE t DEDUP; DEDUPLICATE is offered wherever DEDUP is. Tests: 16 server queries (DedupSkipTest, CreateTableDedupTest, ShowCreateTableTest text block, WalWriterTest, ImportIODispatcherTest, O3ParquetStaleReaderTest, AlterTableSetTypeRestartTest, Enterprise EntShowCreateTableTest) in the parity fixture, plus tests/dedup-skip.test.ts (AST mode, round trips incl. FORMAT PARQUET / OWNED BY / CTAS, rejected forms, skip/deduplicate as identifiers, autocomplete). Red before: 83 failing (48 fixture checks + 35 focused); after: all 346 pass, full suite 15604 passed. Corpus (53,975 server/docs SQL): 27 fixed, 0 round trips changed, 0 regressed, 0 throws, formatter meaning unchanged.
… VOLUME Server (OSS #7263, SqlParser.parseExpireRowsClause) parses the relative retention form as `KEEP [N] HIGHEST|LOWEST ON <col> [PARTITION BY <cols>]` (expectTok "on"), and SHOW CREATE MATERIALIZED VIEW prints it with ON (RowExpiryUtil.appendKeepByClause). RowExpiryWindowTest:966-1001 asserts "'on' expected" for the form without ON. In CREATE MATERIALIZED VIEW the WHEN predicate ends at the clause boundary (expireRowsClauseBoundary): CLEANUP EVERY, IN VOLUME (and WITH / DEDUP, which mat views then reject; Ent OWNED BY via isExpireRowsClauseBoundary). Before: `... EXPIRE ROWS KEEP 2 LOWEST ON v PARTITION BY k` failed with "Unexpected token 'ON'" in CREATE and ALTER MATERIALIZED VIEW ... SET EXPIRE ROWS, while `KEEP HIGHEST v ...` (server error) was accepted. `EXPIRE ROWS WHEN v < 2.0 IN VOLUME 'v1'` was silently misparsed: IN was read as the IN operator, round trip `... IN VOLUME; v1` (two statements). Change: - parser expireRowsClause: CONSUME(On) between HIGHEST|LOWEST and the column. The WHEN predicate is parsed with a parser flag (inExpireRowsPredicate, restored in finally); setExpression's IN/BETWEEN option is gated off when the flag is set and the next tokens are IN VOLUME, so the predicate ends there and the mat view IN VOLUME clause takes it. Outside EXPIRE ROWS `x IN volume` is unchanged. CLEANUP, WITH, DEDUP and OWNED are not operators, so the predicate already stopped there. - toSql prints `HIGHEST|LOWEST ON <col>`. AST unchanged (keepExtremum `column` already holds the ON column; visitor reads ctx.On only in the KEEP LATEST branch). - tests/fixtures/docs-queries.json: the docs example used the old form without ON; now `KEEP HIGHEST ON px ...` (upstream docs need the same fix). Tests: 17 server queries (MatViewExpireRowsTest, MatViewExpireRowsHardeningTest text block, RowExpiryKeepLatestTest, RowExpiryWindowTest, SqlCompilerImplTest incl. IN VOLUME + OWNED BY, Ent EntRowExpiryAclTest), tests/expire-rows-keep-on.test.ts (AST, toSql, negatives without ON, IN operator kept inside the predicate, `x IN volume` outside, autocomplete), parser.test.ts EXPIRE ROWS describes, a formatter fixture. Red 74 failing -> green; full suite 15702 passed. Corpus: fixed 26 / rtChanged 0 / regressed 6 / throws 0 / formatterChangedMeaning 0. All 6 regressions are the form without ON that the server rejects: 5 from RowExpiryWindowTest:973-993 (expects "'on' expected") and the old docs example. Suggestions: +ON after KEEP [N] HIGHEST|LOWEST (CREATE and ALTER); +PARTITION, CLEANUP, IN, OWNED after KEEP ... ON <col>. Left out: `CLEANUP EVERY h` without a number (server defaults to 1, low priority).
The server parses the query body of CREATE MATERIALIZED VIEW
(SqlParser.parseCreateMatView), CREATE TABLE AS (parseCreateTableAsSelect,
'(' required), CREATE VIEW (parseCreateView), ALTER VIEW (parseViewSql) and
COPY (...) TO (parseCopy) with parseDml, which reads a bare table, a table
with clauses, a UNION of tables, a table function or `t PIVOT (...)` as an
implicit SELECT * FROM .... OSS #7263 made passthrough mat views valid
(MatViewTest: `create materialized view v5 as (base_price where price > 0)`,
`... v4 as base_price`).
Before: all of these failed with "Expecting Select" / "Unexpected token",
except the parenthesised CREATE VIEW form (no set operations there).
`CREATE TABLE y AS (x UNION ALL append)`, `COPY (t WHERE x > 0) TO ...`,
`copy (generate_series(...)) TO ...`, `ALTER VIEW v AS t WHERE ...` and
`CREATE VIEW v AS t WHERE x > 0` all errored.
Change:
- createMaterializedViewBody, createViewBody, alterViewStatement: body is
`(SELECT...)` | `(implicitSelectStatement)` | `SELECT...` |
unparenthesised implicitSelectStatement (gated on LA(1) != '('), as
createViewBody already did for the parenthesised implicit form. The mat
view body previously had independent optional '(' and ')', which also
accepted unbalanced parentheses; they are now paired.
- createTableBody (CTAS) and copyTo: inside the required parentheses,
selectStatement (LA(1) SELECT/WITH/DECLARE) or implicitSelectStatement.
- implicitSelectBody gains the optional PIVOT (...) clause (same position
as in simpleSelect), so `(t PIVOT (...) ORDER BY ...)` works in COPY and
in FROM subqueries/CTEs, as on the server. The top-level
`t PIVOT (...)` still goes to pivotStatement (unchanged AST).
- createViewBody now uses implicitSelectStatement (with UNION/EXCEPT/
INTERSECT) instead of implicitSelectBody.
- Visitor: implicit bodies become SelectStatement { implicit: true } in the
existing fields (CreateMaterializedViewStatement.query, CreateTable
asSelect, CreateView/AlterView query, CopyTo source). No AST type
changes. Mat view asParens is now taken from the body parenthesis only:
before, `AS SELECT ... TIMESTAMP(ts)` set asParens because TIMESTAMP's
'(' is in the same CST node. toSql unchanged (implicit selects already
print without SELECT * FROM; ALTER VIEW always prints parentheses).
- grammar-drift test: Select/Declare/With listed as the inline mat view
query (as for live views), since the unparenthesised SELECT is now an
OR alternative.
Tests: 20 server queries (MatViewTest x5, ArrayTest, O3FailureTest x2,
O3Test x4, QueryRegistryTest, IPv4Test, SqlParserTest x2 incl. ALTER VIEW,
Ent EntSqlParserTest, CopyExportTest x4) and
tests/implicit-select-bodies.test.ts (AST shape, round trips, unbalanced
parentheses / clauses after a parenthesised body / CTAS without '('
rejected, bare `COPY t TO` unchanged, top-level PIVOT unchanged,
autocomplete). Red 95 failing (55 fixture + 40 focused) -> green; full
suite 15833 passed.
Corpus: fixed 31 / rtChanged 0 / regressed 8 / throws 0 /
formatterChangedMeaning 0. Fixed: the queries above plus more CTAS
bare-table forms and `select ... from (cities PIVOT (...))`. All 8
regressions are truncated concatenation fragments with an unclosed
parenthesis (`create materialized view test as (select ts,`, ...), which
the server rejects with "')' expected"; the old optional ')' accepted them.
Suggestions: after `CREATE TABLE y AS (`, `COPY (`,
`CREATE [MATERIALIZED] VIEW v AS [(]`, `ALTER VIEW v AS [(]` table names
and UNNEST are offered; after the table inside those bodies the usual
table-source follow-ups (alias, joins, WHERE, ...); PIVOT after
`SELECT * FROM (t `. The 41 default positions are unchanged.
The visitor unquoted single-quoted strings in statement positions with image.slice(1, -1) and never turned '' into ', while toSql re-escapes them with escapeString. Every doubled quote got doubled again: CREATE USER bob WITH PASSWORD 'it''s' -> PASSWORD 'it''''s' COPY t FROM 'f.csv' WITH DELIMITER '''' -> DELIMITER '''''' The server only strips the outer quotes in these positions (GenericLexer.unquote, std/GenericLexer.java:111: password EntSqlCompilerImpl:2202, DELIMITER SqlParser:3165). So the SQL text has to survive unchanged, and before this fix it did not. Change: a shared unquoteSingle() helper (strip quotes, '' -> ') is now used in every position whose value toSql prints back single-quoted: passwords (CREATE/ALTER USER, SERVICE ACCOUNT), EXTERNAL ALIAS (CREATE/ALTER GROUP), TTL, JWK public keys, DROP TOKEN, IN VOLUME, partition lists (DROP/DETACH/ ATTACH/FORCE DROP/CONVERT, REINDEX PARTITION), SUSPEND WAL code/message (table + live view), REBASE WAL INTO, COPY file/destination/options/CANCEL id, CANCEL QUERY, live view START FROM, ANCHOR DAILY time/zone, SAMPLE BY TIME ZONE/OFFSET, mat view REFRESH START/TIME ZONE/RANGE (extractMaybeString), and INSERT BATCH o3MaxLag. The places that already unescaped (expression literals, typed literals, identifier StringLiteral) now use the same helper. Deliberately unchanged: single-quoted NAMES that toSql prints as "..." identifiers (FROM 'x', stringOrQualifiedName, tableNameOrString, OWNED BY via stringOrIdentifier). The server keeps their inner text verbatim, so FROM 'it''s' is table it''s (SqlParser:3047/7305/8076 unquote; questdb-ent EntShowCreateSupportTest: CREATE USER 'o''brien' stores principal o''brien). The current output "it''s" names that same table. Unescaping would print "it's", which is a different table, so the report's "wrong table" claim for this case was incorrect. extractIdentifierName is left as it is. Tests: tests/single-quote-unescape.test.ts (30 round trips, AST values, pinned verbatim names, expression literals). 3 server queries with '' escapes (EscapeTest:52/62, LatestByTest:3797) were added to server-queries.json as guards. They already passed: no server test uses '' in a statement value position. Red -> green: 34 failing -> all pass; full suite 15884 passed. Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0 (the corpus has no '' in these positions). Suggestions: unchanged (0/43 positions).
Server: the lexer reads a number glued to letters as one token, so `1hour`, `1days`, `1sec`, `111ml` are rejected (SampleByTest:250 "Invalid unit: 1hour", SampleByTest:262 "1min", AlterTableO3MaxLagTest:363/370 "interval qualifier", SqlParser.parseTtlHoursOrMonths "invalid time unit"). SqlUtil.expectMicros (o3MaxLag) accepts s, m, ms, h, d, us and ns (SqlUtilTest:829 "1000ns"). Before: DurationLiteral had no trailing boundary, so invalid units were silently split into a duration plus a second statement: SAMPLE BY 1hour -> `1h; our`, TTL 1days -> `TTL 1 DAY; ays`, o3MaxLag = 1sec -> `1s; ec`, and the valid o3MaxLag = 100ns -> `100n; s`. Now: - DurationLiteral gets the `ns` unit and a (?![\w\u0080-\uFFFF]) boundary. - New MalformedDurationLiteral token (after DurationLiteral, before NumberLiteral): digits + a duration-unit letter + more word characters (`1hour`, `1days`, `1.5hour`, `100nsx`). No grammar rule accepts it, so it is a parse error, like on the server. A boundary alone would have turned `TTL 1days` into a valid `TTL 1 DAYS` (NumberLiteral + DAYS) and `SELECT 1hour` into `1 AS hour`. A dedicated token was chosen over a NumberLiteral lookahead: NumberLiteral is untouched (GEOHASH(8c), MEMORY LIMIT 1G still split as before) and the formatter keeps a glued run like `10.5mFROM` as one word instead of backtracking into opaque fragments. Classified as a literal in the formatter and never suggested in autocomplete. - All valid units still lex as one token: 1h 30m 1M 1y 100T 1U 1n 1_000T 2W 12H 1.5d 1ms 1us 100ns 2Y 2D. Tests: 11 server queries (guards for n/U/T/m/ms/us units, TTL 1D, INSERT BATCH o3MaxLag, TOLERANCE w, GEOHASH(8c), MEMORY LIMIT 1G) + tests/duration-boundary.test.ts. Red before: 24 failing -> 548/548 pass; full suite 15975 passed. Corpus: fixed 0 / rtChanged 0 / regressed 3 / throws 0 / formatterChangedMeaning 0. All 3 regressions are server-rejected inputs that used to be silently split: `o3MaxLag = 111days`, `o3MaxLag = 111ml` (AlterTableO3MaxLagTest expects "interval qualifier") and `sample by 1hour` (SampleByTest expects "Invalid unit: 1hour").
Server: ExpressionNode.of types a literal token as BIND_VARIABLE when it is
`$` + digits or starts with `:` (FunctionParser.createBindVariable binds it).
`:name` is glued in ExpressionParser.processColon: a lone `:` that does not
follow a completed operand (literal, constant, `)`, `]`;
isCompletedOperand) and is not followed by whitespace takes the next token
as the name. So `arr[1:n]` / `arr[i:j]` stay slices, while `x = :sym`,
`IN (:a, :b)`, `LIMIT :lo, :hi`, `-:v` and even `arr[:1]` are bind
variables (ArrayTest:4428 "undefined bind variable: :1"). SAMPLE BY takes
an expression and, when it is not a full period, a one-letter unit
(SqlParser.expectSample): `SAMPLE BY $1 d`.
Before: `$` was a lexer error and `:name` "Unexpected token ':'" in every
position (WHERE, VALUES, LIMIT, SELECT items, function args, IN lists,
DECLARE values, UPDATE SET, SAMPLE BY FROM/TO, ...).
Now:
- New BindVariable token (custom lexer pattern, before Colon): `$\d+` with
a word boundary (`$1abc`, `$a`, `$` stay lexer errors), and
`:[A-Za-z0-9_\u0080-\uFFFF]+` only when the previous token does not end an
operand (Identifier, QuotedIdentifier, `)`, `]`, literals, bind/@ variables,
TRUE/FALSE/NULL/NaN). Keywords are operators/clause words (`LIMIT :n`,
`THEN :x`), except a non-reserved keyword glued to the colon, which is a
column name (`arr[offset:n]`). `::` and `:=` keep lexing first.
- primaryExpression accepts BindVariable -> new AST node
`{ type: "bindVariable", name: "$1" | ":sym" }` (name as written);
toSql prints it verbatim. Covers LIMIT, FROM/TO and every expression slot.
- sampleByClause: `BindVariable <unit letter>` (gated to the SAMPLE BY unit
letters); duration is "$1 d". `SAMPLE BY $1` without a unit is an error,
as on the server.
- Formatter: BindVariable is a literal; a slice Colon followed by a gap and
a word/`:` keeps the gap (`arr[: 5]`), since gluing would create `:5`.
Autocomplete never suggests the token; walkthrough skips BindVariable and
Colon.
- `arr[:5]` (no space) now reads as index by bind variable `:5` (server
semantics) instead of an open-start slice; `arr[: 5]` stays a slice.
parser.test.ts updated accordingly.
Tests: 38 server queries (OSS CoveringIndex*, PGJobContextTest, LimitTest,
QwpEgressBind*, MatViewExpireRowsHardeningTest, DeclareTest,
SqlCodeGeneratorTest, ArrayCreateFunctionFactoryTest, InLongFunction*,
MonotonicTimestampPruningTest, UpdateTest, SampleByNanoTimestampTest,
HashJoinGroupByRuntimeConstantTest, PercentileWindowFunctionTest,
SubsampleTest, CopyExportTest, CompiledFilterRegressionTest,
DeclareOperatorPrecedenceTest, GroupByTest, IntervalFilterTest,
OrderByEncodeSortTest, ParquetTest, RuntimeConstFunctionTest,
SubsampleWildcardProjectionTest, LatestByTest, ILikeSymbol*, ArrayTest slice
guards; Ent ViewAuditTest, TableColdPartitionsTest) +
tests/bind-variables.test.ts. Red before: 150 failing (110 fixture + 40
focused) -> 727/727 pass; full suite 16201 passed.
Corpus: fixed 528 (305 `$n`, 223 `:name`) / rtChanged 0 / regressed 0 /
throws 0 / formatterChangedMeaning 0.
Left out: ALIGN TO CALENDAR TIME ZONE $1 / WITH OFFSET $1 and FILL($1),
FILL(PREV($1)) (11 corpus strings) - those positions are not expressions in
sql-parser yet (AST stores strings) and belong to sample-by-fill-align.
TOLERANCE $1 and SWITCH ROLE ... TIMEOUT $1 stay errors (server rejects).
Server: GenericLexer treats '`' as a quote (Chars.isQuote: ', ", `). A
backtick token has no escapes - it ends at the next backtick, so `a``b` is
two tokens - and GenericLexer.unquote strips the quotes wherever an
identifier is expected (PivotTest `where \`BTC-USD_buy\` > 25780`,
CopyExportTest `copy \`test_table❤️\` to ...`, DeclareTest `\`@c\` + 1`).
SqlParser.validateIdentifier allows letters, digits, '_' and '$' after the
first character of an unquoted identifier (SqlParserTest
testGoodNonQuotedAlias: `select 'a' a1$_`, `select 'a' _a1$`); `$`+digits
at the start of a token stays a bind variable (ExpressionNode.of).
Before: '`' and '$' were lexer errors ("unexpected character"), so
`a$b` lexed as `a` + error + `b` and `select$x` as SELECT + ...
Now:
- QuotedIdentifier also matches `...` (no escapes, may be empty like "").
visitor unquoteIdentifier(): "..." still unescapes "", `...` content is
verbatim (`a""b` names a""b). toSql prints the canonical "..." form.
- Identifier allows '$' after the first character; the keyword negative
lookahead (tokens.ts createKeywordToken) includes '$', so `select$x`,
`from$` are identifiers. `$1` stays BindVariable, `$a` stays an error.
- Named bind variables allow '$' after the first name char (`:a$b`,
processColon glues the next token); `:$1` stays Colon + `$1` (formatter
lexeme oracle).
- toSql escapeIdentifier leaves names with a non-leading '$' unquoted
(`"price$1"` -> price$1, the same identifier on the server); a leading
'$' still needs quotes.
- Autocomplete: '`' is a word boundary like '"'. Backtick table names
already resolve columns (QuotedIdentifier images are sliced).
Tests: 7 server queries (PivotTest, CopyExportTest, DeclareTest x2,
SqlParserTest x2, RowExpiryWindowTest guard) + tests/lexer-quoting.test.ts.
Red before: 62 failing (17 fixture checks + 45 focused) -> all pass; full
suite 16282 passed.
Corpus: fixed 6 / rtChanged 1 / regressed 0 / throws 0 /
formatterChangedMeaning 0. Fixed: the 5 queries above plus
`select 'a' ``` which the server rejects (empty alias) - accepted and the
empty alias dropped by toSql, exactly as sql-parser already does for
`select 'a' ""` (pre-existing looseness, not changed here). rtChanged:
RowExpiryWindowTest `"price$1"` now prints unquoted price$1 (same name).
…ty statements Server: SqlUtil.fetchNext counts `/*` / `*/` (blockCount), so block comments nest (`SELECT 1 /* a /* nested */ still */ FROM t`, DeclareTest:5139-5168, SqlParserTest:6397-6427 hints with nested comments) and a comment that is never closed runs to the end of the input (SqlParserTest.testBlockCommentUnclosed). SqlCompilerImpl.compileBatch skips empty statements via getNextValidTokenPosition (PGMultiStatementMessageTest: `select 1;;`, `;;`, ` ; ;`, `; /* comment */`, `select 1;;/*c*/;--c\n;select 2;`). Before: BlockComment was a lazy regex, so the first `*/` closed a nested comment and the rest leaked out as `still * / ...` (error or silent extra statement); an unterminated `/*` lexed as Divide + Star; any `;` beyond one per statement gave "Redundant input". Changes: - lexer: BlockComment is a custom matcher (exported blockCommentEnd) that nests and runs to end of input when unclosed; line_breaks kept. - parser: statements = MANY(;) MANY2(statement MANY3(;)). Visitor/AST unchanged; empty statements produce no AST node. cst-types regenerated. - formatter lexer: its BlockComment uses the same nesting (closed comments only); an unterminated (nested) comment stays a tolerant token as before. Deliberately NOT replicated (server tokenizer artefacts): `--` inside a block comment starting a line comment, and quotes inside a block comment (GenericLexer still reads quoted spans there, so `/* '*/' */` stays open; the pairing quote is searched to end of input, so on the server `SELECT 1 /* don't */ FROM t WHERE s = 'x'` is just `SELECT 1`). Decision approved by the supervisor; DeclareTest:5142/5148/5157/5720 (`/* '*/' */`) stay unsupported. Here quotes and `--` inside /* */ are plain text. Tests: 15 server queries (DeclareTest, SqlParserTest, PGMultiStatementMessageTest) in server-queries.json + tests/comments-empty-statements.test.ts (41 tests: nesting, unterminated, quotes/-- inside comments, stray */, empty statements, formatter scan/format, autocomplete). recovery.test.ts: `;;;` and `SELECT 1;;;;;SELECT 2` no longer produce errors (server accepts both). Red before: 57 failing (26 focused + 31 fixture) -> all pass; full suite 16383 passed. SqlParserTest:6427 kept in the focused test, not the fixture: the formatter wraps that long hint comment differently on a second pass (pre-existing, also without nesting). Corpus: fixed 10 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Fixed are the nested-comment DeclareTest / SqlParserTest queries plus three the server rejects for other reasons: MatViewExpireRowsHardeningTest:1221 (contextual "unterminated block comment in EXPIRE ROWS clause"), DeclareTest:5177 (nested lists, semantic), and DeclareTest:5168 whose `FROM @x` prints as "@x" (pre-existing toSql issue). Suggestions: statement starts after `;; ` / `; ` / `; /* c */ ; ` (were empty); normal follow-ups after a nested comment and inside an unterminated one (as for `--` comments). Default positions unchanged.
….5e3m
Server (FunctionParser.createConstant): a literal token that
Numbers.parseDouble accepts is a DOUBLE, and FastDouble skips an optional
d/D suffix, so `1d`, `0d`, `1.5D` are doubles. A token ending in m/M
(`(c | 32) == 'm'`) is a DECIMAL parsed by DecimalParser.parse, which also
accepts an exponent (`1.5e3m`, `2.5e-3M`; Decimal256Test:1839 `1.5e2M`).
Before: `99m` / `1d` lexed as DurationLiteral and the AST typed them as
literalType "duration" in expressions (`cast(99m as byte)`, `10d * 0d`,
`VALUES (..., 100m)`), and `1.5e3m` silently misparsed as `1.5e3 AS m`.
Changes:
- lexer: DecimalLiteral gains the exponent form
`\d[\d_]*(\.[\d_]*)?[eE][+-]?\d[\d_]*[mM]\b` (an exponent never forms a
duration, so either case of m is safe). `99m`, `99M`, `1.5M`, `1d`, `1.5d`
stay DurationLiteral tokens, so SAMPLE BY / TTL / REFRESH EVERY / window
frames / WINDOW JOIN / HORIZON LIST are unchanged.
- visitor literal(): a DurationLiteral ending in m/M/d/D is typed
`{ literalType: "number", value, raw }` (value parsed without the suffix
and underscores; raw kept, so toSql output is unchanged). Other units
(1h, 30s, 1ms, 100T, ...) stay durations: they are not numbers on the
server. DecimalLiteral value parsing strips M as well as m.
- Duration-valued expression positions keep a bare duration: table params
(`o3MaxLag = 10m` / `7d`, SqlUtil.expectMicros) and DECLARE values
(`DECLARE @Unit := 1m ... SAMPLE BY @unit` substitutes the token into
SAMPLE BY). A new helper bareDurationLiteral() detects an expression that
is a single DurationLiteral token; `DECLARE @x := 1m + 1` is typed as a
number expression.
- COPY options with an expression value keep a suffixed number verbatim
(`COMPRESSION_LEVEL 3d` still prints `3d`, it used to come from the
duration path; a decimal like `1.5m` there now also keeps its suffix).
AST: no new fields; numbers carry raw as before. No grammar change
(cst-types unchanged).
Tests: 19 server queries added to tests/fixtures/server-queries.json
(CastDecimalToByte/CastByteToDecimal, InsertTest, DecimalFullScaleTest,
LiveViewSmokeTest, Ent ViewAuditTest DECLARE, MulDouble, UnionAllCast,
GreatestNumeric, GroupByRewrite + guards TtlTest TTL 1M, AlterTableO3MaxLag,
HORIZON LIST, mat view SAMPLE BY 10m); no server SQL test uses the
exponent-m form (only Decimal256Test unit tests), so it is covered in the
focused tests/numeric-literal-typing.test.ts (51 tests: lexer, AST typing,
unchanged duration contexts, COPY, autocomplete, formatter).
Red -> green: 29 failing focused tests -> 51/51; the 76 fixture checks are
guards (toSql was already identical). Full suite 16510 passed.
Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0 (toSql keeps raw; the corpus has no exponent-m).
Suggestions: after `SELECT 1.5e3m ` and `WHERE price > 1.5e3m ` now the
normal expression follow-ups (m used to be read as an alias).
…RECISION
Server (ExpressionParser):
- `<expr> AT TIME ZONE '<tz>'`: AT after a completed operand, then TIME,
ZONE and a constant; rewritten to to_timezone(expr, tz). `at time 'x'`
and a dangling `at time` fail with "did you mean 'at time zone <tz>'?"
(TimestampAtTimeZoneTest). at/time/zone stay usable as aliases.
- `timestamp[_ns] with time zone` as a cast target: inside CAST(.. AS ..)
(Scope.CAST_AS) and after `::` (isColonColon on the op stack).
- `double precision`: PRECISION after `double` is ignored in expressions
(isPrecisionKeyword), SqlParser.toColumnType skips it for CREATE TABLE
columns and CTAS `cast(col as type)` (incl. `double precision[]`).
ALTER TABLE ADD COLUMN / ALTER COLUMN TYPE (addColumnWithType) do not.
Before: every form failed, and `x::timestamp at time zone 'UTC'` was
silently split into `x::TIMESTAMP AS at; time AS zone; UTC`.
Changes:
- keywords: `at` and `precision`, both non-reserved (server reserves
neither).
- parser: typeCastExpression takes an optional
`AT TIME ZONE (StringLiteral | EscapeStringLiteral)` after the `::` casts
(gated on LA(2) = TIME so `SELECT ts at FROM t` keeps the alias; not
repeated, the server's to_timezone rewrite does not chain). It binds
tighter than arithmetic and inside unary minus, like the server.
New castDataType rule (CAST and `::`): timestampWithTimeZoneType |
doublePrecisionType | dataType, as gated alternatives so autocomplete
offers WITH only after TIMESTAMP[_NS] and PRECISION only after DOUBLE.
columnDefinition gets an argument: DOUBLE PRECISION only when called from
CREATE TABLE (ALTER ADD COLUMN shares the rule); CTAS castDefinition also
takes it. dataType itself is unchanged.
- AST: new expression node `atTimeZone { expression, timeZone }`. Cast /
column dataType strings are "TIMESTAMP WITH TIME ZONE",
"TIMESTAMP_NS WITH TIME ZONE", "DOUBLE PRECISION[]..." (spelling kept,
same as the existing typed literal's typeName).
- toSql prints `expr AT TIME ZONE 'tz'` and the dataType strings.
- autocomplete: PRECISION is dropped under alterTableAction (rule args are
invisible to content assist).
- formatter: WITH after TIMESTAMP/TIMESTAMP_NS followed by TIME ZONE is not
a WITH clause (it used to break the line, also for the existing
`timestamp with time zone '...'` literal).
- cst-types regenerated.
Tests: 19 server queries in tests/fixtures/server-queries.json
(TimestampAtTimeZoneTest x8 incl. alias guards, InsertTest x2,
SimpleTableTest, SqlParserTest x2, TimestampQueryTest x4, PGArraysTest x2);
no server test parses DOUBLE PRECISION in CREATE TABLE (only expected
model strings), covered in tests/type-syntax.test.ts (51 tests: AST,
precedence, aliases, rejected forms, CREATE/CTAS/ALTER, formatter,
autocomplete). Red -> green: 78 failing (52 fixture checks + 26 focused)
-> all pass; full suite 16637 passed.
Corpus (53,975): fixed 21 / rtChanged 2 (the silent 3-statement splits of
`::timestamp[_ns] at time zone 'UTC'`) / regressed 3 / throws 0 /
formatterChangedMeaning 0. All 3 regressions are server-rejected
TimestampAtTimeZoneTest negatives (testFail2 `at time 'UTC'`,
testFailDangling2/3 `at time`) that used to split silently into
`AS at; time ...`.
Suggestions: AT after a completed operand (SELECT/WHERE/ORDER BY/GROUP BY/
OVER/DECLARE values), TIME after `AT`, WITH after `::timestamp` and
`CAST(x AS TIMESTAMP`, PRECISION after DOUBLE in casts, CREATE TABLE
columns and CTAS casts (not in ALTER ADD COLUMN). The `[` / `)` fallback
punctuation after `CAST(x AS TIMESTAMP|DOUBLE ` is no longer shown because
a keyword is now suggested.
The server accepts three predicate forms that sql-parser rejected:
- `a IS [NOT] TRUE|FALSE`: ExpressionParser.validateIsKeyword accepts
IS [NOT] NULL|TRUE|FALSE and rewrites it to `=` / `!=`
(ExpressionParserTest.testIsTrue/testIsNotTrue/testIsNotFalse).
- `a <> ALL (...)` / `a != ALL (...)`: ExpressionParser isAllKeyword turns the
preceding `<>`/`!=` into the `<>all` function (allFunctions); any other
operator is "unexpected operator", no operator is "missing operator"
(ExpressionParserTest.testAllNotEqual/testAllInvalidOperator/
testAllNoOperator, AllNotEqStrFunctionFactoryTest).
- `g NOT WITHIN (...)`: NOT before a SET operator becomes
OperatorExpression.UnarySetNegation (in / between / within);
ExpressionParserFuzzTest generates `x not within y`.
Before: all three were parse errors ("Unexpected token").
Changes:
- relationalExpression: `Is Not? (Null | True | False)`. New AST node
`isBoolean { expression, value, not }` (isNull is unchanged); toSql
`a IS [NOT] TRUE|FALSE`.
- equalityExpression: optional `All` after NotEquals. AST stays a binary node
with operator "<> ALL" / "!= ALL" (as typed; plain <>/!= still print `!=`);
the right operand is the usual relationalExpression, so `all(x)` is a paren
node and toSql gives `a <> ALL (x)`. Parens are not required (the server's
ALL branch does not require them either). toSql precedence table knows the
two operators (13, same as <>).
- setExpression: `Not?` before Within. WithinExpression gets an optional
`not` (set only when true, so plain WITHIN keeps its AST shape); toSql
`g NOT WITHIN (...)`.
- cst-types regenerated.
Tests: 6 server queries (AllNotEqStrFunctionFactoryTest) in
server-queries.json; tests/predicates.test.ts covers the expression-only
server tests wrapped in WHERE, AST shapes, negatives (IS 3, IS NOT $1, `||
all`, `= all`, `> all`, bare `all`), formatter and suggestions.
Red before: 45 failing (18 fixture + 27 focused) -> all pass; full suite
16708 passed.
Corpus (54k): fixed 6 (the AllNotEqStrFunctionFactoryTest queries) /
rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0.
Suggestions: +TRUE/FALSE after `IS` and `IS NOT`, +ALL after `<>`/`!=`,
+WITHIN after an operand's `NOT`; nothing else changed.
The server accepts ROWS | RANGE | GROUPS frame modes and EXCLUDE CURRENT ROW | GROUP | TIES | NO OTHERS (ExpressionParser.parseWindowSpec / parseWindowFrameClause). sql-parser rejected GROUPS, EXCLUDE GROUP and EXCLUDE TIES. Adds the groups / ties keywords (non-reserved) and the grammar options, with server queries in the parity fixture and tests/window-frames.test.ts.
…omparisons
The server's ExpressionParser accepts these; sql-parser rejected them:
- member access after `)`: `(x).a`, `(pg_catalog.pg_class()).relnamespace`,
`pg_catalog.pg_class().y` (DOT dereference; DeclareTest:4542/4545,
ExpressionParserTest:672). typeCastExpression: MANY(Dot identifier) gated on
the previous token being `)`. New AST node memberAccess { expression, member }.
- stacked prefix operators `- -a`, `~-a`, `-~-~1` (ExpressionParserTest:1863):
unaryExpression OPTION -> MANY; one unary node per operator. toSql keeps a
space in `- -a` so it does not become a `--` comment.
- `NOT NOT x` (BooleanSubQueryPredicateTest:252) and NOT as a comparison
operand `x = NOT y` (CompiledFilterIRSerializerTest:301): notExpression
MANY(NOT); equality right operand may be `NOT <notExpression>`.
- chained comparisons `a = 1 = true`, `a < b < c`, `a > 1 IS NULL`:
equality/relational OPTION -> MANY, left-associative binary nodes
(visitor orders operators/operands by offset).
Tests: 3 server queries (fixture) + tests/expression-gaps.test.ts;
red 13 failing -> all green, full suite 16784 passed.
Corpus: fixed 13 (pg_class member access, NOT NOT) / rtChanged 0 /
regressed 0 / throws 0 / formatterChangedMeaning 0.
Suggestions: +IS after `price > 1 `, +NOT after `a = `, +LIKE/ILIKE after
`a = 1 ` (all valid chain continuations).
…select after DECLARE/WITH, WITH/DECLARE set-op branches Server (questdb#7602; SqlParser.parseDeclare, isMarkedDeclarationAhead, parseQuery) accepts, sql-parser rejected: - `DECLARE [OVERRIDABLE] [AUDITED] @x := ...` markers in any order per variable (ViewAuditTest, DeclareTest). New non-reserved keyword `audited`; declareAssignment MANY(OVERRIDABLE|AUDITED). AST: DeclareAssignment.audited. toSql prints OVERRIDABLE before AUDITED. Duplicate markers are accepted (server rejects; looser is fine). - trailing comma in a value list `('a',)` (any parenthesised list) and after the last declaration (`DECLARE @lim := 1, audited WHERE ...`), gated on a `[markers] @var` lookahead so audited/overridable stay table names. - implicit select after a DECLARE / WITH prefix: `DECLARE @x := 1 (SELECT @x)`, `DECLARE @lim := 1 t WHERE x > @lim`, `(WITH c AS (...) c)`, top-level `WITH c AS (...) c`. selectStatement = DECLARE [WITH] body | WITH body | selectBody; new rule prefixedQueryBody (selectBody | implicitSelectStatement). - set-operation branch starting with WITH or DECLARE (WithClauseTest:649/654, DeclareTest:5616): setOperation alt -> selectStatement (later set ops nest into that branch; toSql prints them flat, which re-parses the same). Tests: 7 server queries (fixture) + tests/declare.test.ts; red 27 -> green, full suite 16818 passed. Corpus: fixed 121 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: +AUDITED after `DECLARE ` (OVERRIDABLE already there); table names after `DECLARE @x := 1 ` and `WITH .. AS (..) `; markers/WITH/SELECT/ tables after `DECLARE @x := 1, `; +DECLARE,WITH after `UNION ALL `.
…ONE / WITH OFFSET <expr> The server parses all three with expr() (SqlParser.expectSample, FILL and ALIGN TO handling in parseSelectClauses): - SAMPLE BY <expr> [one-letter unit]: `@q h` (was a silent 2-statement split), `2 h`, `(1m)`, `(10+20)m`, `(1+2)*10L m`, `$1 d`. sampleByClause = bare unit letter | expression + optional unit. The unit is optional here (server wants it unless the expression is a full period) - looser, so `SAMPLE BY $1` / `SAMPLE BY foo` now parse (tests updated). AST unchanged: duration = expression SQL [+ " " + unit]. - FILL values are expressions: -1, @f, true, PREV(abs(a)), 1+2, 100L, NaN, '5'. AST fill stays string[]: plain names upper-cased (PREV/LINEAR/NONE), PREV(...) upper-cased, NULL; others printed as SQL - string values now keep their quotes (FILL('5') printed FILL(5) before). - ALIGN TO CALENDAR TIME ZONE <expr> [WITH OFFSET <expr>]: @tz, $1, rnd_str('foo','bar') (was a silent split). AST: timeZone/offset strings unchanged for string literals / names; new optional timeZoneExpression / offsetExpression for other values (toSql prints them). - Autocomplete: content assist filters the top level of those expressions (keeps literal / variable tokens) so SAMPLE BY / FILL( / TIME ZONE do not offer every column, function and operator; FILL( now offers PREV, LINEAR, NONE next to NULL. Tests: 8 server queries (fixture) + tests/sample-by-fill-align.test.ts; red 37 -> green; full suite 16872 passed. Corpus: fixed 61 / rtChanged 8 (2 former silent splits, 6 FILL('str') now keeping quotes) / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: +PREV,LINEAR,NONE after `FILL(` / `FILL(PREV, `; `(` fallback after `SAMPLE BY ` and `TIME ZONE `; clause keywords after `SAMPLE BY 2 ` (unit optional); `(` no longer offered after `FILL(PREV `.
…OT before SAMPLE/GROUP BY), LATEST ON @var, open-ended LIMIT Server (SqlParser.parseFromClause): deprecated LATEST BY right after the table, then joins -> WHERE -> LATEST ON -> PIVOT -> SAMPLE BY -> GROUP BY -> WINDOW -> SUBSAMPLE -> ORDER BY -> LIMIT. LATEST ON/BY operands go through expectLiteral with decls, so @variables are allowed. LIMIT accepts `5,` and `,5` (missing lower bound defaults to 0, SqlCodeGenerator.getLoFunction). Before: `... LATEST ON ts PARTITION BY s SAMPLE BY 1d`, `... PIVOT (...) GROUP BY|SAMPLE BY ...`, `FROM t LATEST BY s WHERE ...`, `PARTITION BY @c`, `LIMIT 5,` and `LIMIT ,5` were errors. Now: simpleSelect and implicitSelectBody share selectClauseChain(), a set of alternatives covering the server order AND every order accepted before (SAMPLE BY -> LATEST, GROUP BY -> PIVOT, WHERE -> LATEST BY); written as alternatives so autocomplete does not offer a clause twice. New rule latestOnColumn (columnRef | @var; AST keeps a one-part name "@var", printed raw). limitClause: `LIMIT ,5` -> AST lowerBound literal 0 (toSql `LIMIT 0, 5`), `LIMIT 5,` -> `LIMIT 5`. AST: optional SelectStatement.pivotBeforeGroupBy so toSql keeps PIVOT before SAMPLE/GROUP BY. toSql now prints LATEST BY before WHERE and LATEST ON before SAMPLE BY (server order; same AST meaning). Tests: 8 server queries (LatestByHoistEquivalenceTest:139, LatestByTest:1472, QueryExecutionTimeoutTest:331, SqlParserTest:8285, DeclareTest:2005/875, LimitTest:176/192) + tests/clause-order.test.ts. Red 26 -> green; full suite 16910 passed. Corpus: fixed 18 / rtChanged 1 (`(tab where x = 100 latest by x)` now prints LATEST BY before WHERE) / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +WHERE,SAMPLE after `LATEST BY|ON ... ` (WHERE is only valid after LATEST BY; ruleStack can't tell them apart), +SAMPLE,GROUP after `PIVOT (...) `, +UNION/EXCEPT/INTERSECT after `LIMIT 5, ` (open upper bound).
Server (SqlParser.parsePivot): after the closing `)` an optional table alias (setModelAliasAndGetOptTok, `[AS] alias` unless a tableAliasStop word); the FOR expression stops at its top-level IN (setStopOnTopINOperator), so `FOR year(ts) IN`, `FOR year - 2000 IN`, `FOR (SELECT 'NL')::VARCHAR IN`; IN-list values take an alias with or without AS (`2010 D2`). Before: `PIVOT (...) p` silently split into a second statement `p`, `PIVOT (...) AS p` errored in SELECT form; FOR accepted only a column reference; `IN (2010 D2)` errored. Now: PIVOT clause (simpleSelect / implicitSelectBody) and pivotStatement take `[AS] alias`; implicit alias gated like table aliases (new implicitTableAliasAllowed(), the former tableRef gate) and additionally not when the next token is PIVOT/WHERE, so consecutive statements without `;` (`t PIVOT (...)\nu PIVOT (...)`) still split. AST: optional PivotClause.alias (toSql ` AS p`). FOR operand = bitOrExpression (the level right below IN/BETWEEN/comparisons; no GATE so autocomplete is not misled). pivotInValue: optional AS. PIVOT inside sub-query/CTE bodies already worked (implicit-select-bodies). Tests: 3 server queries (PivotTest:2914, 2934, 415) + tests/pivot.test.ts (alias forms, clause words after `)`, FOR expressions, IN aliases; no one-line server test uses a PIVOT alias). Red 13 -> green; full suite 16927 passed. Corpus: fixed 1 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0 (most PivotTest queries are text blocks, not in the corpus). Suggestions: +AS after `PIVOT (...) `; clause keywords after `PIVOT (...) p `; expression starts after `FOR `.
… alias / sub-query / + offsets, UNNEST (cols) without alias Server (SqlParser.parseJoin, setModelAliasAndGetOptTok, tableAliasStop, expectIntervalLiteral, parseUnnest0): - `tolerance` and `range` stop an implicit table alias, so `ASOF|LT JOIN t2 TOLERANCE 2s` and `WINDOW JOIN p RANGE BETWEEN ...` need no alias/ON; EXCLUDE/INCLUDE PREVAILING likewise. - HORIZON JOIN takes a table or parenthesised sub-query (expectTableNameOrSubQuery) with an optional alias; offsets may be signed with `-` or `+`. - UNNEST(...) (v): column aliases without a table alias. Before: TOLERANCE/RANGE were taken as the table alias (error at `2s` / BETWEEN); HORIZON JOIN required `tableName alias`; `+1m` errored; `UNNEST(x)(v)` silently split into a 2nd statement `(v)`. Now: the tableRef alias gate is a method (implicitTableAliasAllowed, shared with the PIVOT alias) that also refuses TOLERANCE, RANGE and EXCLUDE/INCLUDE before PREVAILING (still fine as explicit `AS` aliases and identifiers). horizonJoin uses tableRef (RANGE refused by the gate, `LIST (` by the LParen check); AST unchanged (JoinClause.table is the tableRef). horizonOffset accepts `+` (AST string "+1m"). unnestClause: column list independent of the alias (alias labelled unnestAlias in the CST). Tests: 7 server queries (AsOfJoinTest:1987/5251, SqlParserTest:14994/14224, DeclareTest:3753, BooleanSubQueryRuntimeGateTest:321, HorizonJoinProjectionTest:1615) + tests/join-aliases.test.ts (no server test uses `+` offsets). Red 20 -> green; full suite 16958 passed. Corpus: fixed 10 / rtChanged 2 (former silent UNNEST(..)(val) splits) / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: after `HORIZON JOIN q ` ON/RANGE/LIST/joins/TIMESTAMP (was an error state: mandatory alias); after `HORIZON JOIN q ON (sym) ` operators + RANGE/LIST.
Server: FROM/JOIN sources may be schema-qualified function calls
(`pg_catalog.pg_class()`, `information_schema.tables()`, used throughout the
catalogue tests); parseDml accepts `UNION DISTINCT` as plain UNION.
Before: `FROM information_schema.tables()` errored at `(`;
`UNION DISTINCT` errored.
Now: tableFunctionName = identifier (Dot identifier)*; the AST name is the
dotted string ("pg_catalog.pg_class"). In tableRef the tableName alternative
now comes before tableFunctionCall, gated off only when `name[.name]* (`
follows AND the function call backtrack succeeds (so inputs that parsed
before as a table keep doing so). Order matters for content assist: it
ignores GATEs and prunes later alternatives once a path consumes the input,
so with the function alternative first `FROM sys.t ` only offered `(`
(docs autocomplete walkthrough). setOperation: UNION [ALL | DISTINCT],
EXCEPT/INTERSECT [ALL] (no DISTINCT suggestion after EXCEPT/INTERSECT); AST
unchanged (DISTINCT == UNION; toSql prints UNION).
Tests: 4 server queries (InformationSchemaTablesFunctionFactoryTest:42,
LiveViewTest:1193, InformationSchemaColumnsFunctionFactoryTest:78,
SqlParserTest:13928) + tests/from-sources.test.ts (pg_class/pg_namespace
JOIN, _pg_expandarray). Red 13 -> green; full suite 16976 passed.
Corpus: fixed 13 / rtChanged 0 / regressed 0 / throws 0 /
fmtChangedMeaning 0. Suggestions: +DISTINCT after `UNION `; alias/join
keywords after `FROM pg_catalog.pg_class() ` (was an error state).
…l options, CAST/INDEX in any order Server: parseCreateTableColumns skips `[PRECISION] [NOT] NULL` after a column type / PARQUET(...); SqlCompilerImpl.addColumnWithType skips `[NOT] NULL` (ALTER TABLE ADD COLUMN, ALTER COLUMN TYPE); parseCreateTableCastDef reads `SYMBOL [CAPACITY n] [CACHE|NOCACHE]`; parseCreateTable loops over `, CAST(..)` and `, INDEX(..)` in any order. Before: all of these errored. Now: new rule columnNullModifier (`[PRECISION] [NOT] NULL`, may match nothing; PRECISION gated to CREATE TABLE and filtered from autocomplete) used by columnDefinition and ALTER COLUMN TYPE; visitor ignores it (toSql drops the modifiers, like the server). castDefinition takes CAPACITY/CACHE/NOCACHE (AST: optional CastDefinition.symbolCapacity / cache, printed by toSql). CTAS `, CAST | , INDEX` in one loop (toSql still prints casts before indexes). Tests: 6 server queries (HashJoinTest:136, SqlParserTest:3045/2889/2839, AlterTableAddColumnTest:295, CreateTableTest:468) + tests/create-table-options.test.ts. Red 21 -> green; full suite 17003 passed. Corpus: fixed 18 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +NOT, NULL after a column type (CREATE TABLE, ADD COLUMN, ALTER COLUMN TYPE); +CAPACITY/CACHE/NOCACHE inside CTAS CAST(.. AS type.
…D/INCREMENTAL REFRESH, one-letter TTL units Server (SqlParser.parseCreateMatView): REFRESH [IMMEDIATE | MANUAL | EVERY .. | INCREMENTAL] [DEFERRED] [PERIOD(..)] AS — the mode is optional and INCREMENTAL is the obsolete name of IMMEDIATE; TTL is parsed by SqlParserCallback.parseTtlSettings independently of PARTITION BY; parseTtlHoursOrMonths accepts `<n> <unit>` where unit goes through PartitionBy.ttlUnitFromString (HOUR(S)..YEAR(S) and h/d/w/m/y). Before: `.. SAMPLE BY 30s) TTL 12 HOURS`, `REFRESH AS`, `REFRESH DEFERRED`, `REFRESH INCREMENTAL`, `TTL 2 h` all errored. Now: new rule ttlClause (DurationLiteral | number + unit word | number + one-letter identifier, gated) shared by CREATE TABLE and CREATE MATERIALIZED VIEW; the mat-view TTL moved out of materializedViewPartition into its own optional clause after PARTITION BY (AST unchanged: stmt.ttl). REFRESH takes an optional materializedViewRefresh (PERIOD handled by the existing following option); materializedViewRefresh accepts INCREMENTAL (AST mode "immediate", not suggested) and bare DEFERRED (also reachable from ALTER MATERIALIZED VIEW SET REFRESH — looser than the server there). Tests: 3 server queries (StoragePolicyWriterTest:2510/2637, MatViewReloadOnRestartTest:970) + tests/mat-view-create-options.test.ts. Red 12 -> green; full suite 17018 passed. Corpus: fixed 4 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +TTL after the mat-view body; +DEFERRED, AS after `REFRESH `; +DEFERRED after ALTER .. SET REFRESH; unit words after `TTL 2 ` in mat views.
…E COLUMN lists, TYPE SYMBOL .. INDEX
Server (SqlCompilerImpl): alterTableAddColumn ignores an optional COLUMN and
reads `[IF NOT EXISTS] name type` per comma-separated column;
alterTableRenameColumn loops over `old TO new` pairs; addColumnWithType (also
used by ALTER COLUMN .. TYPE) reads SYMBOL [CAPACITY n] [CACHE|NOCACHE]
[INDEX [TYPE ..] [CAPACITY n]].
Before: `ADD x INT` (~40 server tests) and rename lists errored; `TYPE
SYMBOL INDEX` silently split into a second statement `INDEX`.
Now: ADD -> new rule alterTableAddColumns ([COLUMN] addColumnItem {,
addColumnItem}); addColumnItem = [IF NOT EXISTS] columnDefinition. AST:
first-column IF NOT EXISTS stays AddColumnAction.ifNotExists; later columns
get ColumnDefinition.ifNotExists (toSql prints it before the column). RENAME
COLUMN takes `, old TO new` pairs (AST RenameColumnAction.additionalRenames).
ALTER COLUMN TYPE gets alterColumnTypeIndex (optional INDEX +
indexTypeOptions; AST AlterColumnAction.indexed/indexType/indexInclude/
indexCapacity) before the [NOT] NULL modifier.
Tests: 7 server queries (CompositeColumnMappingTest:87,
AlterTableChangeColumnTypeTest:933, AggregateTest:2198,
AlterTableAddColumnTest:82/324, TableWriterAsyncCmdTest:406, ent
HttpCallbackTest:33) + tests/alter-table-columns.test.ts. Red 22 -> green;
full suite 17050 passed.
Corpus: fixed 34 / rtChanged 3 (TYPE SYMBOL INDEX splits fixed) / regressed 0
/ throws 0 / fmtChangedMeaning 0. Suggestions: +IF after `ADD ` and
`ADD x INT, `; column names after `RENAME COLUMN a TO b, `; +INDEX after
`TYPE SYMBOL `.
Server (SqlCompilerImpl CONVERT branch + alterTableDropConvertDetachOrAttachPartition[ByList]): CONVERT PARTITION TO PARQUET|NATIVE then LIST '<p>' [, ...] | WHERE <expr>, then for PARQUET an optional WITH (bloom_filter_columns = .., fpp = ..). Before: the server form (~500 server tests) silently split into `CONVERT PARTITION TO PARQUET; LIST AS "2024-01-01"` (LIST was only accepted before TO); with WITH it errored. Now: convertPartitionTarget accepts LIST after the target (the old LIST-before-TO form still parses). AST: ConvertPartitionAction.listAfterTarget so toSql keeps the order. Unquoted partition names are not added (they lex as arithmetic; not trivial). Tests: 5 server queries (AdaptiveParquetColumnConversionRestartTest:75, ArrayTest:2399, ParquetWriteTest:2774, DataSyncFailureCleanupTest:62, CoveringIndexParquetNativeRoundTripTest:273) + tests/convert-partition.test.ts. Red 17 -> green; full suite 17073 passed. Corpus: fixed 11 / rtChanged 195 (all `TO .. LIST '..'` splits fixed) / regressed 2 (`.. TO PARQUET LIST` / `.. TO NATIVE LIST` with no partition name: the server rejects them, maybeExpectToken "partition name"; they only "parsed" before as a silent split) / throws 0 / fmtChangedMeaning 0. Suggestions: +LIST after `CONVERT PARTITION TO PARQUET `; +WHERE, WITH after the LIST.
Server: compileAlterMatView's SUSPEND branch calls parseSuspendWal, the same parser tables and live views use (optional `WITH <code|'tag'>, '<message>'`). Before: `ALTER MATERIALIZED VIEW mv SUSPEND WAL WITH 24, 'msg'` errored (the table and live view rules already accepted the WITH form). Now: new rule suspendWalWith used by the mat-view SUSPEND WAL alternative; AST AlterMaterializedViewSuspendWal gets optional code / message (printed by toSql). Tests: no server test uses the WITH form for mat views (only plain SUSPEND WAL, which already parsed); tests/alter-mat-view-wal.test.ts. Red 2 -> green; full suite 17075 passed (one run of the RSI per-position <100ms autocomplete timing test flaked under parallel load; a direct bench shows no slowdown vs HEAD and it passes on rerun). Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 / fmtChangedMeaning 0. Suggestions: +WITH after `ALTER MATERIALIZED VIEW mv SUSPEND WAL `.
…ISCARD/CLOSE/UNLISTEN/RESET
The server dispatches these statement-start keywords to PG-compatibility
executors (SqlCompilerImpl.registerKeywordBasedExecutors): compileBegin /
compileCommit / compileRollback (optional TRANSACTION), compileDeallocate
(statement name), compileSet (SET [SESSION|LOCAL] name {=|TO} v[, v]* or any
tokens up to `;`, e.g. SET TIME ZONE 'UTC'), compileNoOp for
DISCARD/CLOSE/UNLISTEN/RESET (one argument token).
Before: BEGIN/COMMIT/ROLLBACK [TRANSACTION] were silently parsed as implicit
selects (`COMMIT AS TRANSACTION`); SET <name> ..., DEALLOCATE, DISCARD, CLOSE,
UNLISTEN, RESET were parse errors.
Now: new rules transactionStatement, deallocateStatement, pgNoOpStatement and
setStatement (SET followed by a loose run of identifiers/literals/=/TO/,/...;
SET TYPE [BYPASS] WAL keeps its own rule, gated on TYPE BYPASS|WAL). AST
nodes transaction / deallocate / pgNoOp / set (raw args). New keywords are
non-reserved (SELECT begin FROM t still works) and highlight as keywords.
Autocomplete: inside SET only TO is offered after the name.
Tests: 9 server fixture entries + tests/pg-session-statements.test.ts.
Red 21 -> green; full suite 17116 passed.
Corpus: fixed 47 / rtChanged 26 (implicit-select misparses of BEGIN/COMMIT/
ROLLBACK now transaction statements) / regressed 1 (prose
"commit() must build ..." - server runs compileCommit and rejects `(`) /
throws 0 / formatterChangedMeaning 0.
…c/partition_by, VACUUM PARTITIONS
Server:
- DROP ALL[;] or legacy DROP ALL TABLES (SqlCompilerImpl.compileDrop).
- UPDATE tbl [AS] alias SET ... (SqlParser.parseUpdate; SqlParserUpdateTest).
- COPY .. TO .. WITH compression_codec <literal> / partition_by <literal>
use expectLiteral, so quoted values ('zstd', 'DAY') work (SqlParser.parseCopy).
- VACUUM PARTITIONS t is the legacy spelling of VACUUM TABLE t
(SqlCompilerImpl.compileVacuum).
Before: all four were parse errors. Now: TABLES optional after DROP ALL;
optional AS before the UPDATE alias; StringLiteral accepted for
compression_codec and partition_by in copyOption; VACUUM TABLE|PARTITIONS.
AST unchanged (DROP ALL prints as DROP ALL TABLES, VACUUM PARTITIONS as
VACUUM TABLE). Formatter grammar-drift: UPDATE alias AS is inline by design.
Tests: 6 server fixture entries + tests/misc-statements.test.ts.
Red 18 -> green; full suite 17144 passed (RSI autocomplete timing test
flaked under parallel load once; passes alone).
Corpus: fixed 16 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0. Suggestions: +PARTITIONS after VACUUM, +AS after
UPDATE t.
…ec, PG type aliases as data types Server: - Index type names go through IndexType.valueOf, which also takes the single words POSTING_DELTA / POSTING_EF (besides POSTING [DELTA|EF]). - ParquetEncoding names include byte_stream_split; ParquetCompression names include default (PARQUET(enc, default), COPY compression_codec default). - ColumnType.nameTypeMap aliases: text smallint bigint real bytea regclass regprocedure array. Before: TYPE POSTING_DELTA, PARQUET(BYTE_STREAM_SPLIT), PARQUET(plain, default) and compression_codec default were parse errors; the type aliases were plain identifiers (not highlighted / suggested as types). Now: new non-reserved tokens PostingDelta, PostingEf, ByteStreamSplit (constants) and Text, Smallint, Bigint, Real, Bytea, Regclass, Regprocedure, Array (dataTypes, exported as ArrayType to avoid shadowing the global). All are IdentifierKeyword, so they stay usable as names. dataType accepts the aliases explicitly; typed literals (text 'x') use the new tokens instead of an image check; ARRAY[...] literals consume the Array token; EXPLAIN (FORMAT TEXT) accepts the Text token. Autocomplete: ARRAY is not offered at every expression start (only as a type name). Tests: 4 server fixture entries (2 regression guards for ::real / ::text[] with a `text` column) + tests/option-names.test.ts. Red 5 -> green; full suite 17163 passed. Corpus: fixed 2 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: + alias types after :: / CAST AS / column types; +POSTING_DELTA,POSTING_EF after INDEX TYPE; +BYTE_STREAM_SPLIT, DEFAULT in PARQUET(...); +TEXT after EXPLAIN (FORMAT.
…le-valued only, drop storage_policies
Checked against server FunctionFactory signatures (OSS core
griffin/engine/functions/** and Enterprise com/questdb/griffin/engine/functions):
- table-valued: checkpoint_status(), export_activity(), pg_enum(),
pg_catalog.pg_extension(), information_schema.{tables, columns,
key_column_usage, table_constraints, referential_constraints,
character_sets, questdb_columns}(); Enterprise backup_status(),
table_cold_partitions(s), tick_calendars().
- scalar: build(), current_data_id(), from_base64(S), interval(NN),
cancel_query, timestamp_floor_utc; Enterprise backup_instance_name(),
node_role(), reload_tick_calendars().
- files(s) is a cursor function only (moved from scalar to table-valued);
glob has both a scalar glob(Ss) and a cursor glob(s) (now in both lists).
- storage_policies is an Enterprise view, not a function: removed from
tableValuedFunctions (still parses as a table name); parser test renamed
to the view form.
Word lists only; no grammar change.
Tests: 2 server fixture entries + tests/functions-list.test.ts.
Red 3 -> green; full suite 17174 passed.
Corpus: fixed 0 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0. Suggestions: new names offered in FROM (table-valued)
and expression (scalar) positions; files no longer offered as scalar;
storage_policies no longer offered as a function.
…eserved) The server reserves only SqlKeywords.KEYWORDS (+ Enterprise grant/group/ revoke/service/user in EntSqlKeywords); every other word is a valid name: CREATE TABLE tango (left DOUBLE[][], right DOUBLE[][]) (ArrayTest), SELECT typeOf(l) lt FROM a4b (CompiledFilterRegressionTest), CREATE TABLE payload (k INT, by BYTE, ...) (HashJoinBuildFramesTest), create table tab (value int, when long, ...) (IODispatcherTest), SELECT sum(x) OVER window FROM t WINDOW window AS (...) (WindowFunctionTest), t.left / t.window, CREATE USER left, CREATE GROUP group. Before: all of these were parse errors (the words were reserved here). Now: - Left Right Full Inner Cross Asof Lt Splice Horizon Window By Group When Then Else Details Repair Unpivot Pivot Show Copy Truncate Vacuum Grant Revoke If Include Respect For are non-reserved (tokens.ts CLAUSE_IDENTIFIER_KEYWORD_NAMES, spread into IDENTIFIER_KEYWORD_NAMES). Grant/Revoke/Group are reserved in Enterprise; accepted anyway (never stricter than the server). - IS / ILIKE / DECLARE stay reserved in expressions but are accepted as column names in columnDefinition (AST name from the keyword token). - Alias gates: implicit table aliases stop at the server's tableAliasStop words (LEFT RIGHT FULL INNER CROSS ASOF LT SPLICE HORIZON WINDOW GROUP PIVOT) and statement keywords (SHOW COPY TRUNCATE VACUUM GRANT REVOKE DECLARE); implicit column aliases stop at FOR (columnAliasStop) and the statement keywords; pivot aggregation aliases stop at FOR; PIVOT FOR-list stops at GROUP BY. SHOW/COPY/GRANT/... at statement start never start an implicit SELECT (server dispatches them by keyword). - Content assist ignores GATEs: budgeted-content-assist refuses the same implicit-alias matches (IMPLICIT_ALIAS_STOP_TOKENS by rule), WHEN as a simple-CASE operand right after CASE, and a statement keyword as the table of an implicit SELECT / PIVOT statement. The token-based table fallback no longer takes `GROUP`/`LEFT`... as an alias without AS. - toSql keeps quoting these words; the formatter does not treat them as operands (THEN -1 stays unary) and never starts a clause after a `.` (t.window); bind-variable lexing after THEN/BY etc. unchanged. - Not done: reserved keywords after a dot (a.from, KeywordAsTableNameTest): accepting them made `SELECT t.| FROM trades t` parse `t.FROM` and broke alias-qualified column autocomplete. ANY stays reserved (= ANY(...)). Tests: 6 server fixture entries + tests/reserved-words.test.ts. Red 21 -> green; full suite 17203 passed. Corpus: fixed 75 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Some fixed entries are prose or server-invalid SQL that now splits (e.g. CopyImportTest "copy x from 'f' partition by HOUR", which the server rejects with "'with' expected"). Suggestions: unchanged at all 55 checked positions.
…ET/UNSET RESOURCE GROUP) Server (EntSqlCompilerImpl): createResourceGroup — CREATE RESOURCE GROUP [IF NOT EXISTS] name [WITH (key = value, ...)]; compileAlterResourceGroup — ALTER RESOURCE GROUP name SET (...) | RESET (key, ...) | RENAME TO name; dropResourceGroup — DROP RESOURCE GROUP [IF EXISTS] name; compilePrincipalResourceGroup — ALTER USER|SERVICE ACCOUNT|GROUP x SET RESOURCE GROUP g [MAPPING PRIORITY n] | UNSET RESOURCE GROUP. Policy keys (parseResourceGroupPolicy): memory_limit cpu_weight max_active_queries max_queued_queries queue_timeout. Before: all of these were parse errors. Now: rules createResourceGroupStatement / alterResourceGroupStatement / dropResourceGroupStatement / resourceGroupPolicy / resourceGroupOption / principalResourceGroupAction (shared by alterUserAction and alterGroupStatement). Policy values: string, [-]number[unit], duration, UNLIMITED; value optional per key (RESET form). MAPPING PRIORITY accepted for users too (server rejects it there; permissive). AST: createResourceGroup / alterResourceGroup / dropResourceGroup statements, AlterUserResourceGroupAction, alterGroup actions setResourceGroup/unsetResourceGroup. All words were already non-reserved tokens. Tests: 9 server fixture entries + tests/ent-resource-groups.test.ts. Red 30 -> green; full suite 17243 passed. Corpus: fixed 155 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: +RESOURCE after CREATE/ALTER/DROP and ALTER USER|GROUP x SET; +UNSET after ALTER USER x; policy keys inside WITH (.
Server (EntSqlCompilerImpl.parseCreateViewExt / parseReplaceViewExt): after
the view body, WITH AUDIT and OWNED BY are independent trailing clauses in
either order.
Before: `... ) WITH AUDIT` started a new WITH statement and failed
("Expecting As"); `OWNED BY g WITH AUDIT` likewise.
Now: createViewBody accepts a MANY of WITH AUDIT | OWNED BY x, gated so WITH
not followed by AUDIT still begins the next (CTE) statement. New non-reserved
keyword `audit` (highlighting; still usable as identifier). AST:
CreateViewStatement.audit; toSql prints WITH AUDIT before OWNED BY.
Tests: 4 server fixture entries (EntShowCreateViewTest:131/87/58,
ViewAuditTest:674) + tests/ent-view-audit.test.ts. Red 15 -> green; full
suite 17264 passed.
Corpus: fixed 27 / rtChanged 0 / regressed 0 / throws 0 /
formatterChangedMeaning 0.
Suggestions: +WITH after a CREATE VIEW body, +AUDIT after it.
…ON order, ASSUME grantee lists Server: Permission.java namePermissionMap (80 names) parsed by PermissionParser.parse; after TO/FROM grantee, GRANT takes up to two of WITH GRANT OPTION / WITH VERIFICATION in either order, REVOKE takes WITH VERIFICATION. EntSqlCompilerImpl.compileGrant/compileRevoke: ASSUME SERVICE ACCOUNT sa TO|FROM parseUserOrGroupList (comma list). Before: RENAME TABLE / RENAME COLUMN (reserved RENAME), CANCEL ANY COPY (reserved ANY), SET REFRESH LIMIT / SET MEMORY LIMIT (reserved LIMIT) were errors in GRANT/REVOKE; REVOKE ... WITH VERIFICATION, GRANT ... WITH VERIFICATION WITH GRANT OPTION and ASSUME grantee lists were errors. Now: permissionToken accepts RENAME first, ANY second, LIMIT third word; grantStatement MANY(WITH GRANT OPTION | WITH VERIFICATION); revokeStatement optional WITH VERIFICATION; ASSUME statements take grantee lists (AST toList/fromList when more than one; RevokeStatement.verification). Autocomplete: PERMISSION_NAMES (content-assist.ts) drives the permission word continuations (replaces the ad-hoc CONVERT/REMOVE/STORAGE list); ANY offered only after CANCEL. New non-reserved keywords admin, engine, settings, sql for highlighting the permission names. Tests: 5 server fixture entries (CopyThroughDemoteTest:80, AlterPermissionTest:171, AdminSecurityContextTest:139, ReplicationAclTest:260, EntSqlParserTest:1269) + tests/ent-grant-revoke.test.ts (all 80 names in GRANT/REVOKE/list forms, option order, assume lists, autocomplete). No server test runs REVOKE..WITH VERIFICATION, the VERIFICATION-first order or ASSUME lists successfully (focused tests only). Red 20 -> green; full suite 17289 passed. Corpus: fixed 17 / rtChanged 0 / regressed 0 / throws 0 / formatterChangedMeaning 0. Suggestions: more permission first words after GRANT/REVOKE; REFRESH/ PARQUET/MEMORY after GRANT SET; ANY after GRANT CANCEL; WITH after REVOKE..FROM u.
…aliases/passwords, REST token option order
Server (EntSqlCompilerImpl): addColumnSuffix parses OWNED BY after the
ADD COLUMN list; parseMemoryLimit unquotes its token ('64M', 'UNLIMITED');
external aliases and passwords are any (unquoted) token (parsePassword:
`with password null` sets the password "null"); CREATE TOKEN TYPE REST WITH
TTL v is followed by a loop of REFRESH / TRANSIENT in any order.
Before: ADD COLUMN .. OWNED BY g silently split into 3 statements;
SET MEMORY LIMIT '64M', unquoted EXTERNAL ALIAS, ALTER USER|SERVICE ACCOUNT
.. WITH PASSWORD pw|null were errors; `.. TRANSIENT REFRESH` split into a
second statement.
Now: alterTableAddColumns [OWNED BY x] (AddColumnAction.ownedBy);
memoryLimit accepts StringLiteral (resource group option values now go
through memoryLimit only); CREATE/ALTER GROUP external alias is
stringOrIdentifier ('' unescaped); alterUserAction WITH PASSWORD takes a
string, NULL, number or identifier; REST token MANY(REFRESH | TRANSIENT).
Tests: 6 server fixture entries (HttpExternalGroupTest:396,
ResourceGroupHttpTest:1002/1012, AclAuditLogTest:717/1039,
HttpBasicDynamicAuthenticationTest:221) + tests/ent-users-groups.test.ts
(TRANSIENT REFRESH / service-account unquoted password: no positive server
test, focused only). Red 19 -> green; full suite 17318 passed.
Corpus: fixed 36 / rtChanged 1 (EntSqlParserTest "... TRANSIENT REFRESH
blablabla": REFRESH now part of the token statement; server rejects the
trailing word) / regressed 2 (EntSqlParserTest:42/47 "add column y int
owned [blablabla]" — server syntax-error tests, "'by' expected") / throws 0 /
formatterChangedMeaning 0.
Suggestions: +OWNED after ADD COLUMN definitions; +NULL after WITH PASSWORD.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Syncs the parser with SQL the QuestDB server accepts (OSS and Enterprise), driven by an audit of server changes since 0.1.21 and a full comparison of the server grammar against this package.
The goals follow the package's uses: highlighting, formatter, autocomplete, and never being stricter than the server. Every statement the server parses should parse here. Exact round-trip fidelity is not a goal.
One commit per feature (36 commits):
tests/fixtures/server-queries.jsonholds queries copied from the server test suites, each withfile:line.tests/server-queries.test.tschecks every entry four ways:ROLLUP,CUBEandGROUPING SETS. This also fixes a regression on master:rollup,cube,setsandgroupingwere reserved, so they no longer worked as names. AlsoDEDUP SKIP/DEDUPLICATE,EXPIRE ROWS KEEP … ON, and implicit select bodies such asAS (t WHERE …).0x…,1.5f,E'…', typed literals (timestamp '…',ipv4 '…');$1and:name, backticks,$inside identifiers;''in quoted values.AT TIME ZONE,WITH TIME ZONEcasts,DOUBLE PRECISION;IS [NOT] TRUE/FALSE,<> ALL (...),NOT WITHIN;GROUPSframes,EXCLUDE GROUP/EXCLUDE TIES;(expr).member, chained comparisons.AUDITED/OVERRIDABLE, FILL / SAMPLE BY / ALIGN TO with expressions, the server's clause orders;FOR <expr>, join alias stop words (TOLERANCE,RANGE, …), HORIZON JOIN options;pg_catalog.*()andinformation_schema.*()in FROM,UNION DISTINCT.[NOT] NULLcolumn modifiers, CTASCAST/INDEXoptions;TTL/REFRESHvariants;ADDwithoutCOLUMN,RENAME COLUMNlists,TYPE SYMBOL … INDEX;CONVERT PARTITION TO … LIST, mat viewSUSPEND WAL WITH.BEGIN/COMMIT/ROLLBACK,DEALLOCATE, PGSET,DISCARD/CLOSE/UNLISTEN/RESET,DROP ALL,UPDATE … AS, quoted COPY options,VACUUM PARTITIONS.functions.ts. The stalestorage_policies()entry is removed; it is a view.left,right,window,by,group,when,then,else,show,copy, … Alias gates mirror the server's stop lists.CREATE VIEW … WITH AUDIT, all 80 permission names in GRANT/REVOKE,WITH VERIFICATION, ASSUME lists,ADD COLUMN … OWNED BY, quoted memory limits, unquoted external aliases and passwords, REST token options.Each commit message gives the server source references, the test counts (failing before → passing after), and the corpus numbers.
Verification
KEEP HIGHEST vwithoutON,1hour,111days,ownedwithoutBY,cancel query 12.01f, …).Worth a look
SELECT timestamp 'x'is now a typed literal, as on the server, not a column alias.BEGIN,COMMITand similar are statements, not implicit table selects.ARRAYis now a keyword token, exported asArrayType.ROLLUP,CUBEandGROUPINGafter GROUP BY / SAMPLE BY;PREV,LINEARandNONEinFILL(;GROUPSin window frames;ATandISafter expressions; the session statements at statement start; full permission lists afterGRANT.DEDUPLICATE→DEDUPVACUUM PARTITIONS→VACUUM TABLELIMIT ,5→LIMIT 0, 5;. That is the root cause of silent splits on unsupported syntax.