Skip to content

fix: high-impact divergences from bash-oracle (regex, quoted heredocs, reserved words, fd prefixes) - #81

Merged
mpecan merged 8 commits into
mainfrom
fix/oracle-high-impact
Oct 19, 2026
Merged

mpecan merged 8 commits into
mainfrom
fix/oracle-high-impact

Conversation

@mpecan

@mpecan mpecan commented Oct 14, 2026

Copy link
Copy Markdown
Owner

Fixes the four highest-impact findings from a differential session against bash-oracle (about 34k inputs: our corpus, Parable's upstream corpus of gnu-bash/oils/tree-sitter test scripts, plus mutation and grammar fuzzing). Each one either rejected or silently changed common real-world bash, and in each case Parable already matched bash, so these are porting gaps.

The four fixes

  1. =~ regex operands (0b25fd9, 1e4f624). The right-hand side of =~ was read with the normal tokenizer, so any unquoted (, | or < was a syntax error: [[ $x =~ ^(a|b)$ ]], [[ $s =~ ([a-z]+)([0-9]+) ]]. A new Lexer::read_cond_regex_word follows bash's PST_REGEXP and Parable's WORD_CTX_REGEX:
    • parens nest, and blanks are allowed inside a group
    • | is a pattern character
    • outside a group, &, ;, <, > end the word (so [[ a =~ a<b ]] is still an error, as in bash)
    • <(…)/>(…) is a process substitution
    • unlike Parable, a ]] inside the word (x]]) stays part of the pattern, as in bash.
  2. Quoted here-documents (8a57eac). With <<'EOF', <<"EOF" or <<\EOF, bash keeps \<newline> in the body as-is. The reader always treated it as a line continuation, so a cat <<'EOF' > script.sh body lost every \, and a \ right before the delimiter line hid the delimiter. PendingHereDoc now carries the quoted flag, and backslash processing is skipped for quoted bodies.
  3. Reserved words after a compound command (797bac2). Following bash's reserved_word_acceptable, a keyword may come right after fi, done, esac, }, ) or ]]. Before, …; fi done and { a; } done were rejected, and a stray fi fi was accepted with the second fi as a command word.
  4. fd prefixes of any int width (1078a7d). is_fd_number only allowed 1–2 digits, so exec 100<>file and 212>log parsed the number as a command word. It now accepts any all-digit value that fits in i32; larger numbers stay words, as in bash. coproc 2>err cmd and x | time 2>1 now detect fd prefixes too.

Follow-ups for regressions exposed by fix 3

Re-running the oracle after fix 3 found invalid inputs that were previously rejected only by accident, through other existing leniencies:

  • ae370de: a redirect target is never a reserved word. Otherwise >esac while … parsed a loop.
  • 0fa15ca: a coproc simple command collects all of its words. coproc cat * x=y and coproc>f e used to split the leftover words off into a second command.
  • 896181e: top-level commands need a separator. (a) b, echo a (x), (a) { b; } and esac case … used to parse as two commands.

Verification

  • cargo fmt, clippy -D warnings, cargo test (387 tests) and lint-extra are clean. Each commit was tested on its own.
  • Python bindings: tests/parable passes 1638/1638.
  • Oracle session, same seeds before and after: 246 mismatches fixed, 0 changed for the worse. 5 inputs newly accepted that bash rejects. All 5 are leniencies that already exist on main (multi-word case patterns like case $x in a b), and an unterminated e[); earlier errors used to hide them. Out of scope here.
  • Parable upstream corpus (about 4.5k cases): failures go from 166 to 141, with 0 new failures.

New tests: parser/cond_regex_tests.rs, lexer/heredoc_quoted_tests.rs, parser/reserved_after_compound_tests.rs, parser/fd_prefix_tests.rs, parser/coproc_tests.rs, plus additions to parser/syntax_error_tests.rs. Expected values come from bash-oracle output.

🤖 Generated with Claude Code

mpecan and others added 8 commits October 14, 2026 10:31
The right-hand side of `=~` was read with the ordinary tokenizer, so
`(`, `)`, `|`, `<` and `>` ended the word and common patterns such as
`[[ $x =~ ^(a|b)$ ]]` or `[[ $s =~ ([a-z]+)([0-9]+) ]]` were rejected.

Add `Lexer::read_cond_regex_word`, mirroring bash's PST_REGEXP and
Parable's WORD_CTX_REGEX: parens are pattern characters that nest,
blanks are allowed inside a group, and `|` is a pattern character. The
word ends at `&&`, an unmatched `)`, or a blank outside any group.
Unlike Parable, `]]` inside a word (`x]]`) stays part of the pattern,
as in bash.

Found by differential fuzzing against bash-oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a quoted delimiter (`<<'EOF'`, `<<"EOF"`, `<<\EOF`) bash reads the
body literally, but the here-doc reader always treated `\<newline>` as a
line continuation. `cat <<'EOF' > script.sh` with continuation lines
therefore lost every `\` and joined the lines, and a `\` right before the
delimiter line hid the delimiter.

Carry the existing `quoted` flag into `PendingHereDoc` and skip all
backslash processing in `read_heredoc_line` when it is set, so each
physical line stands alone. Unquoted here-documents are unchanged.

Found by differential fuzzing against bash-oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reserved-word recognition was only re-armed after `then`, `else`,
`elif`, `do` and `;`. After a closing `fi`, `done`, `esac`, `}`, `)` or
`]]` the next word was lexed as a plain word, so `for ...; do if a; then
b; fi done` and `while x; do { y; } done` were rejected, while a stray
`if a; then b; fi fi` was accepted with `fi` as a command word.

Mirror bash's `reserved_word_acceptable`: re-arm command start after
those closing keywords in `advance_word_context`, after the `)` operator,
and in the parser after `]]` (which the lexer usually sees as a plain
word inside `[[ ]]`).

Found by differential fuzzing against bash-oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`is_fd_number` only accepted one or two digits, so `exec 100<>file`,
`212>log` or `007<in` parsed the number as a command word followed by
an fd-less redirect. Bash accepts any all-digit prefix that fits in an
int; larger numbers stay words. Match that, using the same i32 range
`parse_redirect_with_fd` already parses into.

The `coproc WORD...` and post-`| time` word loops also never treated a
token as an fd prefix (`coproc 2>err cmd`, `x | time 2>1`), and the
coproc loop did so without checking adjacency. Route both through the
existing adjacency-aware `try_parse_fd_redirect`.

Found by differential fuzzing against bash-oracle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Following Parable, the regex reader only stopped at `&&`, blanks and an
unmatched `)`, so inputs bash rejects (`[[ a =~ a<b ]]`, `x;y`, `x&y`)
parsed. Bash ends the regex word at `&`, `;`, `<` and `>` outside a
group, which leaves an invalid conditional token. Do the same, and read
`<(...)`/`>(...)` there as a process substitution, as bash does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redirect operators left the command-start context untouched, so at the
start of a command `>esac` lexed its target as the `esac` keyword. Now
that closing keywords re-arm reserved words, that let `>esac while ...`
parse a `while` loop where bash reports a syntax error. Clear reserved-
word recognition when a redirect operator is lexed, as bash does for the
word after a redirection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For `coproc WORD...` the word loop stopped at the first assignment-
shaped argument (`coproc cat * x=y`, `coproc echo pwd=`), and for
`coproc >redir ...` only further redirects were collected (`coproc>f e`).
The leftover words became a separate command. Share one collection loop
between both paths, accept assignment words as arguments, and stop
re-arming command start before the leading redirect's target.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`parse_all` started a new top-level command wherever the previous one
stopped, so `(a) b`, `echo a (x)`, and now that closers re-arm reserved
words, `(a) { b; }` and `esac case ...` parsed as two commands. Bash
requires `;`, `&` or a newline there. After each top-level list the next
token must now be a newline or end of input, which also replaces the
old no-progress fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mpecan
mpecan merged commit c01b785 into main Oct 19, 2026
5 checks passed
@mpecan
mpecan deleted the fix/oracle-high-impact branch October 19, 2026 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant