Repository navigation
fix: high-impact divergences from bash-oracle (regex, quoted heredocs, reserved words, fd prefixes) - #81
Merged
Conversation
The right-hand side of `=~` was read with the ordinary tokenizer, so `(`, `)`, `|`, `<` and `>` ended the word and common patterns such as `[[ $x =~ ^(a|b)$ ]]` or `[[ $s =~ ([a-z]+)([0-9]+) ]]` were rejected. Add `Lexer::read_cond_regex_word`, mirroring bash's PST_REGEXP and Parable's WORD_CTX_REGEX: parens are pattern characters that nest, blanks are allowed inside a group, and `|` is a pattern character. The word ends at `&&`, an unmatched `)`, or a blank outside any group. Unlike Parable, `]]` inside a word (`x]]`) stays part of the pattern, as in bash. Found by differential fuzzing against bash-oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With a quoted delimiter (`<<'EOF'`, `<<"EOF"`, `<<\EOF`) bash reads the body literally, but the here-doc reader always treated `\<newline>` as a line continuation. `cat <<'EOF' > script.sh` with continuation lines therefore lost every `\` and joined the lines, and a `\` right before the delimiter line hid the delimiter. Carry the existing `quoted` flag into `PendingHereDoc` and skip all backslash processing in `read_heredoc_line` when it is set, so each physical line stands alone. Unquoted here-documents are unchanged. Found by differential fuzzing against bash-oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reserved-word recognition was only re-armed after `then`, `else`,
`elif`, `do` and `;`. After a closing `fi`, `done`, `esac`, `}`, `)` or
`]]` the next word was lexed as a plain word, so `for ...; do if a; then
b; fi done` and `while x; do { y; } done` were rejected, while a stray
`if a; then b; fi fi` was accepted with `fi` as a command word.
Mirror bash's `reserved_word_acceptable`: re-arm command start after
those closing keywords in `advance_word_context`, after the `)` operator,
and in the parser after `]]` (which the lexer usually sees as a plain
word inside `[[ ]]`).
Found by differential fuzzing against bash-oracle.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`is_fd_number` only accepted one or two digits, so `exec 100<>file`, `212>log` or `007<in` parsed the number as a command word followed by an fd-less redirect. Bash accepts any all-digit prefix that fits in an int; larger numbers stay words. Match that, using the same i32 range `parse_redirect_with_fd` already parses into. The `coproc WORD...` and post-`| time` word loops also never treated a token as an fd prefix (`coproc 2>err cmd`, `x | time 2>1`), and the coproc loop did so without checking adjacency. Route both through the existing adjacency-aware `try_parse_fd_redirect`. Found by differential fuzzing against bash-oracle. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Following Parable, the regex reader only stopped at `&&`, blanks and an unmatched `)`, so inputs bash rejects (`[[ a =~ a<b ]]`, `x;y`, `x&y`) parsed. Bash ends the regex word at `&`, `;`, `<` and `>` outside a group, which leaves an invalid conditional token. Do the same, and read `<(...)`/`>(...)` there as a process substitution, as bash does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Redirect operators left the command-start context untouched, so at the start of a command `>esac` lexed its target as the `esac` keyword. Now that closing keywords re-arm reserved words, that let `>esac while ...` parse a `while` loop where bash reports a syntax error. Clear reserved- word recognition when a redirect operator is lexed, as bash does for the word after a redirection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For `coproc WORD...` the word loop stopped at the first assignment- shaped argument (`coproc cat * x=y`, `coproc echo pwd=`), and for `coproc >redir ...` only further redirects were collected (`coproc>f e`). The leftover words became a separate command. Share one collection loop between both paths, accept assignment words as arguments, and stop re-arming command start before the leading redirect's target. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`parse_all` started a new top-level command wherever the previous one
stopped, so `(a) b`, `echo a (x)`, and now that closers re-arm reserved
words, `(a) { b; }` and `esac case ...` parsed as two commands. Bash
requires `;`, `&` or a newline there. After each top-level list the next
token must now be a newline or end of input, which also replaces the
old no-progress fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the four highest-impact findings from a differential session against bash-oracle (about 34k inputs: our corpus, Parable's upstream corpus of gnu-bash/oils/tree-sitter test scripts, plus mutation and grammar fuzzing). Each one either rejected or silently changed common real-world bash, and in each case Parable already matched bash, so these are porting gaps.
The four fixes
=~regex operands (0b25fd9,1e4f624). The right-hand side of=~was read with the normal tokenizer, so any unquoted(,|or<was a syntax error:[[ $x =~ ^(a|b)$ ]],[[ $s =~ ([a-z]+)([0-9]+) ]]. A newLexer::read_cond_regex_wordfollows bash'sPST_REGEXPand Parable'sWORD_CTX_REGEX:|is a pattern character&,;,<,>end the word (so[[ a =~ a<b ]]is still an error, as in bash)<(…)/>(…)is a process substitution]]inside the word (x]]) stays part of the pattern, as in bash.8a57eac). With<<'EOF',<<"EOF"or<<\EOF, bash keeps\<newline>in the body as-is. The reader always treated it as a line continuation, so acat <<'EOF' > script.shbody lost every\, and a\right before the delimiter line hid the delimiter.PendingHereDocnow carries thequotedflag, and backslash processing is skipped for quoted bodies.797bac2). Following bash'sreserved_word_acceptable, a keyword may come right afterfi,done,esac,},)or]]. Before,…; fi doneand{ a; } donewere rejected, and a strayfi fiwas accepted with the secondfias a command word.intwidth (1078a7d).is_fd_numberonly allowed 1–2 digits, soexec 100<>fileand212>logparsed the number as a command word. It now accepts any all-digit value that fits ini32; larger numbers stay words, as in bash.coproc 2>err cmdandx | time 2>1now detect fd prefixes too.Follow-ups for regressions exposed by fix 3
Re-running the oracle after fix 3 found invalid inputs that were previously rejected only by accident, through other existing leniencies:
ae370de: a redirect target is never a reserved word. Otherwise>esac while …parsed a loop.0fa15ca: acoprocsimple command collects all of its words.coproc cat * x=yandcoproc>f eused to split the leftover words off into a second command.896181e: top-level commands need a separator.(a) b,echo a (x),(a) { b; }andesac case …used to parse as two commands.Verification
cargo fmt,clippy -D warnings,cargo test(387 tests) andlint-extraare clean. Each commit was tested on its own.tests/parablepasses 1638/1638.main(multi-word case patterns likecase $x in a b), and an unterminatede[); earlier errors used to hide them. Out of scope here.New tests:
parser/cond_regex_tests.rs,lexer/heredoc_quoted_tests.rs,parser/reserved_after_compound_tests.rs,parser/fd_prefix_tests.rs,parser/coproc_tests.rs, plus additions toparser/syntax_error_tests.rs. Expected values come from bash-oracle output.🤖 Generated with Claude Code