Skip to content

Retarget qj to jq 1.8.2 - #27

Merged
6 merged 31 commits into
mainfrom
ci/jq-1.8.2
Oct 15, 2026
Merged

Retarget qj to jq 1.8.2#27
6 merged 31 commits into
mainfrom
ci/jq-1.8.2

Conversation

@6

@6 6 commented Oct 15, 2026

Copy link
Copy Markdown
Owner

qj now matches jq 1.8.2, the patch release of 2026-06-20, instead of 1.8.1. Most of 1.8.2's 86 commits are security fixes, and many of them change behavior; qj follows all of them.

Conformance (jq_diff against jq 1.8.2):

Platform Default (strict) QJ_JQ_COMPAT=1 (strict)
macOS arm64 40,340 / 40,347 40,347 / 40,347
Linux x86-64 40,365 / 40,372 40,372 / 40,372
Linux aarch64 40,365 / 40,372 40,372 / 40,372
macOS x86-64 40,340 / 40,347 40,347 / 40,347

The default board's 7 misses are qj's own help and version text, as before. FreeBSD and NetBSD match except for those 7. musl adds its known stdout/stderr ordering differences and one module crash where its heap differs. Windows matches jq.exe except the cases in .github/jq_diff_windows_known.txt.

What 1.8.2 changed that qj now does:

  • Messages: Cannot index X with Y (key), and 30-byte truncation of values in errors that keeps a closing quote or bracket.
  • Depth limits: equality, comparison, containment, object merge and paths stop past 10,000 levels with ... too deep errors. Freeing values no longer recurses, and the printer's depth limit is 10,000.
  • Builtins: tonumber/toboolean reject NUL bytes, @uri/@urid are rewritten, _strindices raises type errors, rtrimstr("") and from_entries are fixed, and string length bounds are tightened.
  • Input: util.c reads 4,091-byte chunks and completes a UTF-8 sequence split at the end of one. JSON input is no longer cut at a NUL byte, and -f rejects program files with NUL bytes. die()'s text changed.
  • Linker and compiler: circular imports are an error, and there's a 4,095 limit on closures. --run-tests takes several files and checks --skip/--take.

jq 1.8.2's own crashes, reproduced in compat mode only (the default keeps the sane answer):

  • Module-chain abort. The linker frees definitions that a chain of three or more modules still calls, and the compiler then aborts on an assert(). It aborts the same way in every run, but how differs by platform: macOS arm64, macOS x86-64, glibc, musl and Windows each fail differently, and compat mode follows each measured platform.
  • Closure-limit aborts: the aborts that follow some 4,095-closure errors.

Compat mode:

  • The 1.8.1 crashes and hangs that 1.8.2 fixed are gone here too: import cycles, delpaths with nan, jv_free overflows, and jv_dels' double free.
  • The stack models were re-measured against the 1.8.2 binaries on macOS 27, macOS 26 and Linux x86-64. Equality and comparison are separate models now, and object merge costs 16 bytes more a level. At a 512 KB stack, compat mode dies 2 to 9 levels short of jq, never later.

Also:

  • Vendored jq 1.8.2's test suites, builtin.jq and test modules.
  • Regenerated the fixtures, baselines and parser tables.
  • Updated the docs (README, COMPATIBILITY, LIMITATIONS, the port plan, CLAUDE.md) and the CI scripts.
  • _strindices no longer forces sequential processing.

Not re-run for 1.8.2: the 121-program small-stack corpus and the Linux side of the undefined-behaviour corpus. COMPATIBILITY.md says so.

🤖 Generated with Claude Code

6 and others added 30 commits October 14, 2026 21:01
@6 @claude
Vendor jq 1.8.2's test suites and require jq 1.8.2 in jq_diff
6a571e6 
The update script no longer fetches utf8-truncate.jq, which jq 1.8.2 removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's value layer: depth limits, error dumps, string bounds
67c7dc9 
- jv_dump_string_trunc's new cut (whole characters, then ... and the
  closing delimiter) and 30-byte buffers in every error that dumps a
  value; jv_get's errors are always `Cannot index <kind> with <kind>
  (<key>)`.
- jv_equal, jv_cmp, jv_contains and object * stop past depth 10000
  (Equality check / Comparison / Containment check / Object merge too
  deep), checked where jq checks them: ==, !=, <, sort, group_by,
  unique, min/max, bsearch, array -, indices, contains, IN.
- getpath, setpath and delpaths refuse paths longer than 10000; delpaths
  skips NaN array indices and fails jq's assertion when sorting its
  paths is too deep.
- MAX_PRINT_DEPTH is 10000; the printer continues with an explicit stack
  past 64 levels. jv_free no longer recurses, so drops are no longer
  checked against compat mode's free model.
- Strings of INT_MAX - 8 bytes or more: Repeat string result too long,
  String too long; literals longer than DEC_MAX_DIGITS are invalid.
- tonumber and toboolean reject strings with a NUL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's builtin.c changes: @urid, _strindices, strflocaltime
a94778e 
- @urid decodes every byte (NUL included) and validates the whole
  result as UTF-8.
- _strindices checks both arguments' types instead of failing an
  assertion.
- strflocaltime reports localtime's error instead of failing an
  assertion; set_tm_yday's assertion is at builtin.c line 1621.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Vendor jq 1.8.2's builtin.jq: from_entries and rtrimstr
90f4d69 
from_entries is `map(...) | add // {}` (no label any more, so the native
stops advancing the label counter) and rtrimstr cuts
`.[:length - ($right | length)]`. The native's iteration error dumps
30 bytes too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Regenerate the value and builtin fixtures from jq 1.8.2
23af855 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
jq_diff corpus and e2e checks for jq 1.8.2's value and builtin changes
216c953 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Compat mode: jq 1.8.2's stack models
a263205 
Re-measured against jq 1.8.2's release binaries on macOS 27, macOS 26 and
Linux x86-64, by padding the environment at a fixed limit until jq no longer
answers, shape by shape against 1.8.1:

* jv_free frees iteratively now, so Site::Free and the native-drop limit go.
* jv_equal and jv_cmp are separate sites: gcc gives jvp_equal 160 bytes a
  level on Linux, against jvp_cmp's 144 (both 128 on macOS).
* jv_object_merge_recursive is 16 bytes a level more on both (128 / 144).
* Linux's shared value base is 3,216 (sort); the module chain, compile and
  macOS's regex bases move by 16; --run-tests holds 32,800 bytes on macOS
  (12,776 on Linux), with a floor of 44,368 (23,880).
* The depth walks stop where jq 1.8.2 gives up, past 10,000 levels.

At a 512 KB stack compat-mode qj dies 2 to 9 levels short of jq at every
value site. The compat_mode tests over values run at a small stack, where
the recursions can still overflow before jq's limit stops them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's main.c changes: die() text, -f with NUL bytes, versio…
261b23f 
…n 1.8.2

die() prints "online docs at" with one space. A -f program file holding a
NUL byte is refused with "program file contains NUL bytes" (exit 2) instead
of being cut there. Compat mode's version and help say 1.8.2, and
VERSION_DIR is 1.8.2. The JSON parser's tokenadd now fails with "Token too
long" past 2^30 - 1 bytes, as jq 1.8.2's does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's linker and compiler changes
704b145 
linker.c: a module is recorded once it parses, before its own imports, so
importing one still loading reports "circular import of <path>" (in both
modes; compat mode no longer dies of the old recursion). A module that
doesn't parse isn't recorded, so importing it again parses it again. A
module path with a NUL byte is refused, and import paths keep their NUL
bytes (modulemeta's deps show them).

Libraries are linked in that order, each before the ones it imports, where
jq's block_drop_unreferenced frees definitions a kept one still calls. With
QJ_JQ_COMPAT=1 qj follows jq's walk and dies as jq does when it compiles
such a call (expand_call_arglist's assert) or reads a freed data import
(stack_push's); by default it drops only what nothing kept reaches, which is
what jq drops whenever it doesn't abort.

compile.c: at most 4095 subfunctions and parameters per function. The rest
of compile now runs in jq's order (each subfunction's parameters assigned
just before it is compiled, code emitted after the subfunctions and only
without errors), so compat mode also dies where jq does after these errors.
check_object_key's buffer is 30 bytes.

Corpus: modules.toml calls at most two imports deep (gen_modules.py), the
aborts are compat_mode.toml cases, the limits are in jq182_cli.toml.
tests/jq_compile/disasm.jsonl is re-recorded with jq 1.8.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's util.c input: 4091-byte chunks completed as UTF-8, no…
69f57b1 
… NUL cut

jq 1.8.2's read_more fgets at most 4091 bytes, then freads what the UTF-8
sequence a full chunk ends in is missing (newlines included: one there ends
no chunk and isn't counted, and with -R the line goes on past it unless it
is the chunk's last byte). A chunk without a newline is no longer cut at its
first NUL, so NUL bytes reach jq's parser.

Both readers do this: the CLI's util.c port, and src/io's reader, which now
keeps an anchor (the start of the chunk its position is in) instead of the
line start, walks chunks where a line runs past 4091 bytes, tracks the last
complete chunk of a stream as it reads, and gives the parallel engine
where a job's first chunk ends; a job's line whose newline fread swallows
is read by the reader. The util.c oracle in src/io/tests follows 1.8.2, and
the generated inputs put UTF-8 sequences, cut short or not, at the 4091-byte
boundary.

Corpus: cli_io.toml (gen_cli_io.py) gets inputs around the 4091-byte chunks
and NULs; jq182_cli.toml gets jv_load_file's 4096-byte reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port jq 1.8.2's jq_test.c: several test files, option checks, new sub…
5d164cc 
…tests

--run-tests runs every file it is given, each with the --skip/--take given
before it (stdin when there is none); --skip or --take without a count says
so and exits 1 instead of crashing. An expected result that doesn't survive
printing and parsing back (a NaN) fails the test with a message instead of
an assert(). After the start-state tests come jq 1.8.2's jq_compile_args,
recompile and exhaust-and-reuse subtests, printing their lines. Test files
are still read in 4095-byte lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Record the test fixtures with jq 1.8.2
cdce70a 
The value and builtin fixtures (gen.py, b1_gen.py) and the parser cases
(jq_lang_parse.rs) are re-recorded with the jq 1.8.2 binary, and the tests
that run jq check for jq-1.8.2. What changed in the data is jq 1.8.2's
value-layer and builtin changes (index error messages, printing depth,
NULs reaching the parser) and the new upstream test programs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
e2e tests for jq 1.8.2's input chunks, NULs and subfunction limit
92ec63e 
Also the baseline file's header says jq 1.8.2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Walk fgets chunks without scanning twice
b2645ed 
advance counts newlines while it walks the chunks of a long range; a range
with no newline, or a raw chunk, is walked by length alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
jq_diff: macOS baselines against jq 1.8.2
eec8bbc 
Both scoreboards: 40,342 cases, compat mode strict on all of them and the
default strict on all but qj's own help and version text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Docs: qj targets jq 1.8.2
21d2456 
README, CLAUDE.md, the port plan, LIMITATIONS and COMPATIBILITY say 1.8.2
where they say what qj matches. COMPATIBILITY.md has the macOS scoreboard
(40,342 cases, compat mode strict on all), what 1.8.2 changed in the crash
table (the depth limits, iterative jv_free, circular imports, the linker's
use-after-free on module chains and the closure limit), the re-measured stack
models with what moved, the --run-tests needs, and the undefined behaviour
re-run against 1.8.2 (jv_dels' double free is fixed). The Linux numbers and
the Release check's platforms are still 1.8.1's until CI runs them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Run programs that use _strindices in parallel
826f946 
_strindices was in SEQUENTIAL_BUILTINS because jq 1.8.1's f_string_indexes
passed any argument to jv_string_indexes, whose assert() aborted the process
on a non-string; on macOS abort() flushes the output produced so far, so the
records before the abort had to have been processed in order. jq 1.8.2 checks
both kinds in f_string_indexes first and raises ordinary type errors ("cannot
be searched, as it is not a string", "is not a string"), and qj ports that in
both default and compat mode: compat mode has no _strindices abort left to
reproduce. The builtin keeps no state across records, so records using it are
independent and can go through the parallel engine like any other.

strptime stays sequential: jq 1.8.2 still fails set_tm_yday's assert() on
macOS.

The new ndjson test runs _strindices with type errors over 20,000 records on
four threads and compares stdout, stderr and the exit code with jq.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Comments: MAX_PRINT_DEPTH is 10,000 in jq 1.8.2
7361a5c 
STACK_BYTES' comment counted MAX_PRINT_DEPTH among the jq limits that bound
qj's deepest recursions. In jq 1.8.2 it is 10,000 (it was 256), and qj's
printer only recurses for the first 64 levels before it continues with an
explicit stack, so printing belongs with the recursions that turn into loops.
gen.py's deep print documents sit around 1.8.1's limit of 256; say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
compat_mode.toml: describe each case against jq 1.8.2
f77656a 
The comments described what jq 1.8.1 did: overflow its stack freeing,
comparing, merging and walking paths of deep values, and loop forever on a
NaN path element. jq 1.8.2 frees without recursing, stops the other value
recursions at 10,000 levels with "Path too deep", "Object merge too deep",
"Containment check too deep", "Equality check too deep" and "Comparison too
deep", skips NaN keys in delpaths, and raises the slice error where it freed
the key twice. Each comment now says what the case checks against jq 1.8.2
(checked with the 1.8.2 binary), keeping the 1.8.1 history where it explains a
case's name or depth. Only comments and one `note` (which the harness ignores)
change: names, args, files, env and mem_mb stay, so ids and fingerprints do
too, and both scoreboards stay at their baselines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Drop what jq 1.8.2 fixed from deviations, tests and examples
19a5c43 
- value/mod.rs listed two deviations from jq that jq 1.8.2 no longer has:
  delpaths with a NaN path element (1.8.2 skips it, as qj does) and a
  malformed slice key in delpaths on an array (1.8.2 raises the slice error
  instead of freeing the key twice). Only "Object too big" is left.
- The live value test kept those two delpaths shapes out because jq 1.8.1
  hung or corrupted its heap on them; with jq 1.8.2 they run and match.
- platform/mod.rs's examples of jq's assert() failures used two that 1.8.2
  removed: strflocaltime of an unrepresentable time (now an error) and
  _strindices of a non-string (now a type error). They use strptime's
  set_tm_yday assert (macOS) and jv_array_length's (delpaths after a failed
  sort) instead, both still reachable in jq 1.8.2.
- The undefined-behaviour scripts were written against jq 1.8.1; note that
  1.8.2 fixed jv_dels' double free, and the other two remain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
CI: Windows conformance requires jq 1.8.2; comments name 1.8.2
dea87c6 
jq_diff_windows.py refused any jq but jq-1.8.1; the jq_diff cases and
baselines are jq 1.8.2's now, so it requires jq-1.8.2. The known-mismatch
list's header and the workflow and Windows script comments name the version
they run against (mise.toml already installs 1.8.2).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Name jq 1.8.2 where code, tests and generators cite jq 1.8.1
9c64b2c 
What qj ports, and what tests and fixtures take as their expectation, is jq
1.8.2 now. Each reference was checked against 1.8.2 before it changed:

- Port headers (main.c, util.c, execute.c, exec_stack.h, compile.c,
  parser.y, lexer.l, locfile.c, jv*.c, builtin.c, libm.h): the ports follow
  1.8.2. builtin.jq is vendored verbatim from 1.8.2; Oniguruma (6.9.10) and
  libm.h are unchanged; the function list is still 64 CFUNCs and 11 BINOPS;
  there is still no @base32. strptime.c cites util.c's lines in 1.8.2
  (448-1258; the code is identical).
- Test expectations: the platform cases (gen_cases.py, regenerated with
  1.8.2: no change), the libm corpus (4,415 rows) and the regex corpus (650
  rows), the binding/scoping disassembly in jq_compile.rs and the closed
  descriptor combinations in compat_mode.rs were all rerun against the 1.8.2
  binary and match. The strptime abort cases still abort in 1.8.2.
- Generators: gen_parser_tables.py reads target/jq-src/jq-1.8.2 and
  gen_builtin_matrix.py requires jq-1.8.2; both were rerun, and only the
  version in their output's header changed. The fuzzers require jq-1.8.2.
- Corpus .test/.toml files: comment lines only, so both scoreboards stay at
  their baselines with the same cases.

Kept as they are: true history (jq 1.8.1 hung on NaN keys in delpaths, the
undefined-behaviour scripts' subject, build.rs's measurement against 1.8.1's
binary), the version pair in the jq_diff cache's invalidation test, and the
"ICU 1.8.1" license name in LICENSE-jq.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Tests: jq 1.8.2's platform aborts are macOS-only; compare 10,000 leve…
a82ac79 
…ls on Windows

jq 1.8.2 checks localtime's result in strflocaltime, so the only platform
abort left is set_tm_yday's on macOS, and the Windows smoke test's 100,000
levels are past what jq 1.8.2 compares (Equality check too deep).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
CLAUDE.md: _strindices runs in parallel now
ce95787 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
jq_diff: Linux baselines against jq 1.8.2, and Windows' known mismatches
4290656 
From the ci/jq-1.8.2 run's jq-diff-linux artifact: 40,366 cases, the compat
board strict on all but the data import in a chain of modules, which jq 1.8.2
aborts on while compiling on Linux (to be modelled). On Windows jq.exe now
compiles cli_basics' deep-binops, and aborts on modules' data import in a
chain, where qj answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
With glibc, a freed data import aborts compat mode while compiling
86ec907 
jq 1.8.2 frees the global of a data import two modules below the program
when only an unwalked definition refers to it. With glibc, compiling that
reference fails nesting_level's assert() (compile.c:1091) before anything
runs; with Apple's libc the reference compiles and stack_push's assert()
fails when it is read. Compat mode now follows glibc on Linux and keeps the
macOS behaviour elsewhere (musl is unmeasured).

Corpus: modules.toml's data-import-in-a-chain-unused is macOS-only (jq
aborts on Linux), and gets non-aborting neighbours; compat_mode.toml's
data-import aborts are split into macOS and Linux cases. The macOS
baselines record the new cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
COMPATIBILITY: the freed data import, per platform
6e4869d 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Linux baselines and scoreboard numbers against jq 1.8.2
e2abfbb 
From the ci/jq-1.8.2 Checks run: 40,372 cases on Linux, the compat board
strict on all of them and the default on all but qj's help and version text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Port plan: reflow the 1.8.2 entry
9c03343 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Freed data imports abort compat mode while compiling on all Linux and…
b546bb4 
… Intel macOS

jq 1.8.2's nesting_level finds a freed data import's global compiled into
another function on Linux with musl as with glibc (compile.c:1091), and
never compiled on Intel macOS (compile.c:1086); only arm64 macOS reaches
stack_push. Compat mode follows each.

modules.toml drops data-import-in-a-chain-unused: whether jq aborts on it
depends on the platform, and compat_mode.toml covers it per platform. The
Windows known-failures list and the macOS baselines drop its entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6 @claude
Docs: the Release check's platforms against jq 1.8.2
a65df6d 
Linux aarch64 and macOS x86-64 are strict on the compat board (40,372 and
40,347 cases); on musl one chain of modules dies of SIGSEGV in jq where
compat mode aborts as on glibc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@6
6 merged commit e384ee7 into main Oct 15, 2026
15 checks passed
@6
6 deleted the ci/jq-1.8.2 branch October 15, 2026 01:14
@6 6 mentioned this pull request Oct 15, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@6