revert(lsp): temporarily revert #3867 to land #4000 first by rossirpaulo · Pull Request #4002 · BoundaryML/baml

rossirpaulo · 2026-07-13T21:19:27Z

Clean revert of a05ec5a (#3867, "resolution-first semantic tokens rework"), coordinated with @codeshaunted.

Why

#3867 merged ~30 minutes after #4000 (the LSP mutex/latency fix: position-encoding negotiation, connection-scoped sessions, bounded ingress) opened, and the two rework the same LSP boundary. Landing order matters: #4000 is the infrastructure #3867's feature should sit on. Integrating #3867 inside #4000's merge would mean rewriting its encoder and cache inside a merge commit — worse for review and for feature ownership. Reverting and re-landing on top is cleaner for everyone.

The revert also removes a live encoding issue from canary until the re-land: encode_semantic_tokens currently emits fixed-encoding positions (LineIndex) and byte lengths, which miscounts any non-ASCII identifier under the LSP-default UTF-16.

Re-land guidance (agreed with Avery)

The classification engine (resolution index, CST walk, tools_semantic_tokens visualizer) re-lands byte-for-byte. Only the LSP boundary adapts:

Encode positions and lengths via the negotiated PositionCodec (available at every handler after fix(lsp): resolve the 0.14 mutex/latency regression — Phase 0 containment + Phase 1 bounded ingress #4000) instead of LineIndex + byte lengths.
Move semantic_tokens_cache into connection-scoped session state — sessions are per-connection with per-connection encodings after fix(lsp): resolve the 0.14 mutex/latency regression — Phase 0 containment + Phase 1 bounded ingress #4000, so a process-global cache would cross-serve wrongly-encoded tokens and trample delta baselines. The atomic result-id sequence can stay global.
Optional: returns(ref) instead of returns(clone) on the tracked query to avoid an O(tokens) clone per request; handle_request_with_cancellation for the new handlers.

Sequence

This revert merges.
fix(lsp): resolve the 0.14 mutex/latency regression — Phase 0 containment + Phase 1 bounded ingress #4000 folds in canary (now conflict-free) and merges.
feat(lsp): resolution-first semantic tokens rework #3867 re-lands adapted — we're happy to prep the integration branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VEMxti5WQ8bCgJUuX9AQvo

Summary by CodeRabbit

New Features
- Added support for highlighting dynamic, with, and throws syntax.
- Improved semantic highlighting for names, types, members, strings, numbers, and multi-part paths.
Bug Fixes
- Updated catch-clause bindings to provide more consistent navigation and local-variable handling.
- Adjusted parsing and formatting for contextual keywords such as as and type.
- Corrected hover information for catch bindings.
Breaking Changes
- Semantic tokens no longer include modifiers or range/delta responses.
- Boolean and null values are no longer represented as literal semantic tokens.

This reverts commit a05ec5a.

vercel · 2026-07-13T21:19:33Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview, Comment	Jul 13, 2026 9:41pm
promptfiddle	Ready	Preview, Comment	Jul 13, 2026 9:41pm
promptfiddle2	Ready	Preview, Comment	Jul 13, 2026 9:41pm

github-actions · 2026-07-13T21:20:04Z

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

Add RUN_CODSPEED=1 to the PR description, or
Include run-perf or /perf in the PR title or any commit message.

coderabbitai · 2026-07-13T21:21:34Z

📝 Walkthrough

Walkthrough

This PR migrates contextual keyword handling to generic words, unifies catch bindings with pattern bindings, removes obsolete compiler resolver metadata, replaces semantic-token classification with CST/compiler2 traversal, simplifies LSP transport, and updates or removes related tests and tooling.

Changes

Compiler syntax and binding migration

Layer / File(s)	Summary
Contextual keyword syntax and formatter contracts `baml_language/crates/baml_compiler_parser/...`, `baml_language/crates/baml_compiler_syntax/...`, `baml_language/crates/baml_fmt/...`	Contextual `as` and `type` tokens are handled as words; boolean/null literals and obsolete AST token wrappers are removed or reworked.
Unified pattern and catch bindings `baml_language/crates/baml_compiler2_hir/...`, `baml_language/crates/baml_compiler2_mir/...`, `baml_language/crates/baml_lsp2_actions/src/...`	Catch bindings now use `PatternBinding` for definition, capture, completion, navigation, description, and type information.
Compiler metadata and resolver cleanup `baml_language/crates/baml_compiler2_hir/...`, `baml_language/crates/baml_compiler2_tir/...`	Interface span maps and scope, namespace, enum, and field resolver APIs are removed.

Semantic-token migration

Layer / File(s)	Summary
CST and compiler2 semantic-token generation `baml_language/crates/baml_lsp2_actions/src/tokens.rs`, `baml_language/crates/baml_cli/src/paint.rs`	Semantic tokens are produced by document-order CST traversal with optional compiler2 type-aware expression classification; modifiers and range-query APIs are removed.
Semantic-token test migration `baml_language/crates/baml_lsp2_actions_tests/...`	Test wiring, rendered output, and fixture expectations are updated for reduced structural token output and quote-separated strings; many obsolete fixtures are removed.
LSP semantic-token transport `baml_language/crates/bex_project/src/bex_lsp/multi_project/...`	BEX advertises full semantic tokens only and directly converts compiler2 tokens to LSP delta-encoded entries without caching, range requests, or delta responses.
Workspace and tooling cleanup `baml_language/Cargo.toml`, `baml_language/crates/baml_lsp2_actions/Cargo.toml`, `baml_language/stow.toml`	Unused workspace and crate dependencies are removed, and the semantic tooling prefix is no longer approved.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant BexLSP
  participant Compiler2
  Client->>BexLSP: request semanticTokens/full
  BexLSP->>Compiler2: compute semantic tokens
  Compiler2-->>BexLSP: ordered token ranges and types
  BexLSP-->>Client: full LSP semantic-token response

Possibly related PRs

BoundaryML/baml#3217: Introduces the related LSP2 tokenization surface.
BoundaryML/baml#3417: Migrates pattern handling toward the backend used by catch bindings.
BoundaryML/baml#3886: Adds related catch-clause and error-context groundwork.

Suggested reviewers: codeshaunted, sxlijin, hellovai, aaronvg, 2kai2kai2

Poem

I’m a rabbit reviewing tokens tonight,
Hopping through words that lost their old types.
Catch bindings join patterns just right,
CST carrots grow semantic stripes.
Full LSP baskets, no modifiers in sight—
Hop, hop, merge; the code feels light!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes a temporary revert of `#3867` to unblock `#4000`, which matches the semantic-tokens rollback in the diff.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch paulo/revert-3867-semantic-tokens

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-13T21:29:47Z

Binary size checks passed

✅ 7 passed

	Artifact	Platform	File	Gzip	Gated on	Baseline	Delta	Status
✅	`baml-cli`	Linux	🔒 21.9 MB	9.4 MB	file	21.5 MB	+356.3 KB (+1.7%)	OK
✅	`packed-program`	Linux	🔒 15.8 MB	6.7 MB	file	15.6 MB	+138.7 KB (+0.9%)	OK
✅	`baml-cli`	macOS	🔒 16.8 MB	8.1 MB	file	16.6 MB	+283.0 KB (+1.7%)	OK
✅	`packed-program`	macOS	🔒 12.2 MB	5.9 MB	file	12.1 MB	+83.9 KB (+0.7%)	OK
✅	`baml-cli`	Windows	🔒 18.4 MB	8.3 MB	file	18.1 MB	+271.9 KB (+1.5%)	OK
✅	`packed-program`	Windows	🔒 13.1 MB	6.0 MB	file	13.0 MB	+65.0 KB (+0.5%)	OK
✅	`bridge_wasm`	WASM	14.6 MB	🔒 4.1 MB	gzip	4.1 MB	+72.4 KB (+1.8%)	OK

🔒 = the size this artifact is GATED on (ceiling + delta). Binaries gate on file size (installed binary); WASM gates on gzip (download size). The other size is shown for information only.

Generated by cargo size-gate · workflow run

coderabbitai

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

baml_language/crates/baml_compiler_parser/src/parser.rs (1)
1781-1816: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Prevent find_token_after_hashes from returning tokens.len() at EOF
find_token_after_hashes can still return Some(self.tokens.len()) after the final hash run, so parse_raw_string and looks_like_config_expression can index past the end on bare hash input at EOF. Return None when the post-hash scan reaches EOF, and add a unit test for raw-string lookahead on ##/EOF.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@baml_language/crates/baml_compiler_parser/src/parser.rs` around lines 1781 -
1816, The find_token_after_hashes method must not return Some(tokens.len()) when
the post-hash trivia scan reaches EOF. Update its post-hash return path to
return None at end of input while preserving valid next-token lookahead, and add
a unit test covering raw-string lookahead for bare ## at EOF.
Source: Path instructions

🧹 Nitpick comments (1)

baml_language/crates/baml_lsp2_actions/src/definition.rs (1)
426-430: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider navigating to the interface's own member declaration instead of None.

Returning None for InterfaceVirtualMethod/InterfaceVirtualField removes "Go to Definition" entirely for interface-dispatched calls/field accesses. Since the interface declaration itself is a single, stable location (unlike a specific implementor), pointing there would preserve some navigation value instead of silently doing nothing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@baml_language/crates/baml_lsp2_actions/src/definition.rs` around lines 426 -
430, Update the InterfaceVirtualMethod and InterfaceVirtualField branches in the
definition-resolution match to return the owning interface member declaration
location instead of None. Resolve the interface’s declared member using the
available interface and member-name information, while preserving the existing
behavior for statically resolvable definitions and avoiding navigation to any
specific implementor.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/multi_segment_path.baml`:
- Around line 59-61: Update the expected semantic-token fixture for the
expression-body string literal so `"hi"` is represented as one token with len=4,
rather than separate opening quote, content, and closing quote tokens. Preserve
the existing token position and token type while changing only the expected
segmentation in the multi-segment path fixture.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.baml`:
- Line 28: Update local_type_info handling for PatternBinding in type_info.rs to
resolve and return the binding’s actual inferred type instead of the hardcoded
"unknown" value, then change the catch_binding.baml hover expectation back to
the specific inferred union type.

In `@baml_language/crates/baml_lsp2_actions/src/tokens.rs`:
- Around line 738-767: The constructor and field token classification in the
surrounding expression-processing logic uses heuristic span_text.find lookups
that can select repeated identifiers incorrectly. Plumb exact identifier spans
through AstSourceMap for the constructor type name, field names, and pattern
binding name, then use those spans directly when inserting semantic tokens
instead of searching the full span text; preserve the existing inferred token
type and Property classification.

In `@baml_language/crates/baml_lsp2_actions/src/type_info.rs`:
- Around line 704-710: Update local_type_info’s DefinitionSite::PatternBinding
arm in baml_language/crates/baml_lsp2_actions/src/type_info.rs:704-710 to use
find_binding_ty_in_scopes(db, index, scope_id, pat_id) and return the resolved
inferred type instead of hardcoded "unknown"; update
baml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.baml:28
to restore the expected hover type Errors.AuthError | Errors.NotFoundError.

---

Outside diff comments:
In `@baml_language/crates/baml_compiler_parser/src/parser.rs`:
- Around line 1781-1816: The find_token_after_hashes method must not return
Some(tokens.len()) when the post-hash trivia scan reaches EOF. Update its
post-hash return path to return None at end of input while preserving valid
next-token lookahead, and add a unit test covering raw-string lookahead for bare
## at EOF.

---

Nitpick comments:
In `@baml_language/crates/baml_lsp2_actions/src/definition.rs`:
- Around line 426-430: Update the InterfaceVirtualMethod and
InterfaceVirtualField branches in the definition-resolution match to return the
owning interface member declaration location instead of None. Resolve the
interface’s declared member using the available interface and member-name
information, while preserving the existing behavior for statically resolvable
definitions and avoiding navigation to any specific implementor.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 9842bd66-b0f8-44eb-a075-b842dd81209f

📥 Commits

Reviewing files that changed from the base of the PR and between a05ec5a and 3908c1c.

⛔ Files ignored due to path filters (66)

baml_language/Cargo.lock is excluded by !**/*.lock
baml_language/crates/baml_tests/snapshots/broken_syntax/optional_parameter_defaults/baml_tests__broken_syntax__optional_parameter_defaults__02_parser__function_type_defaults.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/pending_greaters_fix/baml_tests__broken_syntax__pending_greaters_fix__02_parser__extra_greater.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/test_invalid_contexts/baml_tests__broken_syntax__test_invalid_contexts__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/type_annotation_errors/baml_tests__broken_syntax__type_annotation_errors__02_parser__type_malformed.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/config_dictionary/baml_tests__compiles__config_dictionary__02_parser__valid_dictionary.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/json_alias_basic/baml_tests__compiles__json_alias_basic__02_parser__json_alias_basic.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/json_map_literal/baml_tests__compiles__json_map_literal__02_parser__json_map_literal.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/lambda_advanced/baml_tests__compiles__lambda_advanced__02_parser__lambda_advanced.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/lexical_scoping/baml_tests__compiles__lexical_scoping__02_parser__lexical_scoping.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/literal_union_arithmetic/baml_tests__compiles__literal_union_arithmetic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/llm_parse_catchable_parse_error/baml_tests__compiles__llm_parse_catchable_parse_error__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/method_explicit_type_args/baml_tests__compiles__method_explicit_type_args__02_parser__method_explicit_type_args.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_auth_auth.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_llm_models.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/optional_function_parameters/baml_tests__compiles__optional_function_parameters__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/parser_expressions/baml_tests__compiles__parser_expressions__02_parser__precedence.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/parser_statements/baml_tests__compiles__parser_statements__02_parser__break_continue.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/patterns_new/baml_tests__compiles__patterns_new__02_parser__patterns_new.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/stream_crossfile/baml_tests__compiles__stream_crossfile__02_parser__file_a.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/test_expr_name_concat/baml_tests__compiles__test_expr_name_concat__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/test_raw_string_name/baml_tests__compiles__test_raw_string_name__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_basic/baml_tests__compiles__testset_basic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_dynamic/baml_tests__compiles__testset_dynamic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_nested/baml_tests__compiles__testset_nested__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_alias_interaction.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_positions.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_builder_test/baml_tests__compiles__type_builder_test__02_parser__test.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/attr_disambiguation/baml_tests__diagnostic_errors__attr_disambiguation__02_parser__non_field.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/backtick_strict_types/baml_tests__diagnostic_errors__backtick_strict_types__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/basic_types/baml_tests__diagnostic_errors__basic_types__02_parser__types.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__io_ops.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__shell_ops.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/catch_arm_braceless_return/baml_tests__diagnostic_errors__catch_arm_braceless_return__02_parser__repro.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/catch_return_type_mismatch/baml_tests__diagnostic_errors__catch_return_type_mismatch__02_parser__catch_return_type_mismatch.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/duplicate_class_span/baml_tests__diagnostic_errors__duplicate_class_span__02_parser__repro.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__binary_expr.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__config_decls.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__loop_stmts.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__match_exprs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__other_exprs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__type_alias_decls.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/function_type_throws/baml_tests__diagnostic_errors__function_type_throws__02_parser__negative.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/function_types/baml_tests__diagnostic_errors__function_types__02_parser__function_types.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/generics/baml_tests__diagnostic_errors__generics__02_parser__classes.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/headers_edge_cases/baml_tests__diagnostic_errors__headers_edge_cases__02_parser__edge_cases.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/match_exhaustiveness/baml_tests__diagnostic_errors__match_exhaustiveness__02_parser__match_exhaustiveness.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/null_handling/baml_tests__diagnostic_errors__null_handling__02_parser__null_handling.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/optional_parameter_defaults/baml_tests__diagnostic_errors__optional_parameter_defaults__02_parser__unsupported_contexts.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__array_constructor.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__map_constructor.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/self_in_body/baml_tests__diagnostic_errors__self_in_body__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__complex.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__ns_foo_with_top_level_refs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_name_type_error/baml_tests__diagnostic_errors__test_expr_name_type_error__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_wrong_runner/baml_tests__diagnostic_errors__test_expr_wrong_runner__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_with_runner_ambiguity/baml_tests__diagnostic_errors__test_with_runner_ambiguity__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases_json.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/unknown_type_error/baml_tests__diagnostic_errors__unknown_type_error__02_parser__unknown_type_error.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__function_type_throws.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_function_type.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_type_position_errors.snap is excluded by !**/*.snap
pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (122)

baml_language/Cargo.toml
baml_language/crates/baml_cli/src/paint.rs
baml_language/crates/baml_compiler2_ast/src/lower_expr_body.rs
baml_language/crates/baml_compiler2_hir/src/builder.rs
baml_language/crates/baml_compiler2_hir/src/item_tree.rs
baml_language/crates/baml_compiler2_hir/src/semantic_index.rs
baml_language/crates/baml_compiler2_mir/src/lower.rs
baml_language/crates/baml_compiler2_tir/src/inference.rs
baml_language/crates/baml_compiler2_tir/src/resolve.rs
baml_language/crates/baml_compiler_parser/src/parser.rs
baml_language/crates/baml_compiler_syntax/src/ast.rs
baml_language/crates/baml_compiler_syntax/src/syntax_kind.rs
baml_language/crates/baml_fmt/src/ast/declarations.rs
baml_language/crates/baml_fmt/src/ast/expressions.rs
baml_language/crates/baml_fmt/src/ast/tokens.rs
baml_language/crates/baml_fmt/src/ast/types.rs
baml_language/crates/baml_lsp2_actions/Cargo.toml
baml_language/crates/baml_lsp2_actions/src/completions.rs
baml_language/crates/baml_lsp2_actions/src/definition.rs
baml_language/crates/baml_lsp2_actions/src/definition_at_tests.rs
baml_language/crates/baml_lsp2_actions/src/describe.rs
baml_language/crates/baml_lsp2_actions/src/lib.rs
baml_language/crates/baml_lsp2_actions/src/tokens.rs
baml_language/crates/baml_lsp2_actions/src/tokens/classify.rs
baml_language/crates/baml_lsp2_actions/src/tokens/index.rs
baml_language/crates/baml_lsp2_actions/src/type_info.rs
baml_language/crates/baml_lsp2_actions_tests/src/lib.rs
baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs
baml_language/crates/baml_lsp2_actions_tests/src/runner.rs
baml_language/crates/baml_lsp2_actions_tests/src/typing_robustness_test.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/array_pattern.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/backtick_string.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/bigint_literals.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/block_attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/break_continue.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/byte_strings.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/call_named_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_param.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/class.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/client.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_binding.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_let_else_defer.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/constructor_spread.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/deep_method_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/dynamic_type_builder.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_decls.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_member_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/extends_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/for_loops.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/function.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generators.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_field_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_function_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/header_comments.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/http_config.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_else.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_let_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/implements_for.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_inferred_generic_type_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_iter_core.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/is_operator.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/jinja_control_prompt.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/llm_function.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_literal_keys.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_type_and_methods.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_expr.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/multi_segment_path.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_agent_clients.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_images_pipeline.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_sentiment.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/remap_role.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/retry_policy_valid_retry.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/self_qualified_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/signature_variety.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string_calls.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_block.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/throws_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/top_level_binding.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_alias.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_aliases_jinja.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/while_loop.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.baml
baml_language/crates/bex_project/src/bex_lsp/multi_project/mod.rs
baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs
baml_language/crates/tools_semantic_tokens/Cargo.toml
baml_language/crates/tools_semantic_tokens/README.md
baml_language/crates/tools_semantic_tokens/src/analysis.rs
baml_language/crates/tools_semantic_tokens/src/index.html
baml_language/crates/tools_semantic_tokens/src/main.rs
baml_language/crates/tools_semantic_tokens/src/server.rs
baml_language/crates/tools_semantic_tokens/src/staleness.rs
baml_language/stow.toml

💤 Files with no reviewable changes (88)

baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/top_level_binding.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/array_pattern.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generators.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/constructor_spread.baml
baml_language/crates/tools_semantic_tokens/Cargo.toml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/deep_method_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/header_comments.baml
baml_language/crates/tools_semantic_tokens/src/index.html
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_agent_clients.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/block_attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string_calls.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_decls.baml
baml_language/crates/tools_semantic_tokens/README.md
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_param.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_field_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_function_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/self_qualified_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/implements_for.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/jinja_control_prompt.baml
baml_language/crates/baml_lsp2_actions_tests/src/typing_robustness_test.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/is_operator.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.baml
baml_language/crates/baml_lsp2_actions/Cargo.toml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/backtick_string.baml
baml_language/crates/baml_lsp2_actions/src/tokens/classify.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_images_pipeline.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.baml
baml_language/crates/baml_lsp2_actions/src/tokens/index.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/http_config.baml
baml_language/crates/baml_lsp2_actions_tests/src/lib.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/retry_policy_valid_retry.baml
baml_language/crates/tools_semantic_tokens/src/staleness.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.baml
baml_language/crates/tools_semantic_tokens/src/server.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_type_and_methods.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/call_named_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_member_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/extends_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_let_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/remap_role.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/bigint_literals.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.baml
baml_language/crates/tools_semantic_tokens/src/analysis.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_aliases_jinja.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/byte_strings.baml
baml_language/crates/tools_semantic_tokens/src/main.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/dynamic_type_builder.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_let_else_defer.baml
baml_language/crates/baml_lsp2_actions/src/definition_at_tests.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/signature_variety.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_sentiment.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/throws_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.baml
baml_language/crates/baml_compiler2_tir/src/inference.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_literal_keys.baml
baml_language/crates/baml_lsp2_actions/src/describe.rs
baml_language/crates/baml_compiler2_ast/src/lower_expr_body.rs
baml_language/Cargo.toml
baml_language/crates/baml_fmt/src/ast/tokens.rs
baml_language/crates/bex_project/src/bex_lsp/multi_project/mod.rs
baml_language/crates/baml_compiler2_hir/src/item_tree.rs
baml_language/crates/baml_compiler2_tir/src/resolve.rs
baml_language/crates/baml_compiler_syntax/src/syntax_kind.rs

rossirpaulo · 2026-07-13T21:48:31Z

Disposition of the CodeRabbit review: the Critical finding is valid and now addressed — the EOF guard in find_token_after_hashes/parse_raw_string was a standalone bug fix riding inside #3867 (a real panic on incomplete input like a bare ## at EOF; reproduced: index out of bounds: the len is 15 but the index is 15). It is re-applied byte-for-byte with a new regression test in #4005, stacked on this PR, so the fix never leaves canary even while the feature is reverted. The four remaining comments are revert-inherent (restored code/fixtures that were green together on pre-#3867 canary, and improvements that depend on the reverted resolution index) — individual replies on each thread; all of it returns with the re-land.

@codeshaunted

…oundaryML#3867) (BoundaryML#4005) Stacked on BoundaryML#4002 (retargets to canary when it merges). Re-applies the one change in BoundaryML#3867 that was a standalone bug fix rather than part of the semantic-tokens feature, so the revert doesn't reintroduce it — addressing CodeRabbit's Critical finding on BoundaryML#4002. `find_token_after_hashes` could return the one-past-the-end index after a hash run at EOF, and `parse_raw_string` indexed `tokens` with it directly — a panic on incomplete input like a bare `##` at end of file, present on canary long before BoundaryML#3867. Under the new LSP core (BoundaryML#4000) a parser panic on the source path additionally poisons the project into a terminal broken state, so this class matters more now. Both hunks are byte-for-byte @codeshaunted's fix from BoundaryML#3867 (credited as co-author). New here: the regression test the original lacked — verified to panic (`index out of bounds: the len is 15 but the index is 15`) with the guards removed, and covering bare `#`/`##` at top level, inside expression bodies, and after `let`. Validation: `cargo test -p baml_compiler_parser` 140/140 (139 pre-existing + the new test), clippy `-D warnings` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01VEMxti5WQ8bCgJUuX9AQvo  ## Summary by CodeRabbit * **Bug Fixes** * Improved handling of incomplete raw-string syntax at end-of-file. * Prevented unexpected parser crashes when inputs end with standalone hash characters. * Malformed inputs (including truncated bare-hash sequences) now return parse errors instead of failing.  --------- Co-authored-by: Avery Townsend <avery@boundaryml.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…L#3867) on BoundaryML#4000 (BoundaryML#4011) Re-lands BoundaryML#3867 (reverted in BoundaryML#4002 to let BoundaryML#4000 land first) as a revert-of-revert on top of BoundaryML#4000, following the re-land guidance agreed in BoundaryML#4002. ## Byte-for-byte re-land The classification engine is unchanged from what BoundaryML#3867 shipped: the span-keyed resolution index, the CST classification walk, interface-member resolution, the contextual-keyword re-lexing (`as`/`type`/`true`/`false`/`null`), all 78 semantic-token fixtures, and the `tools_semantic_tokens` viewer. ## Adapted to BoundaryML#4000 (LSP boundary only) - **Negotiated encoding**: `encode_semantic_tokens` now encodes through the connection's `PositionCodec` — `delta_start`/`length` are negotiated code units (UTF-16 default, surrogate pairs handled), and multiline tokens split into per-line segments. This also fixes the byte-length miscount for non-ASCII identifiers that BoundaryML#3867 originally shipped (called out in BoundaryML#4002). - **Request idiom**: `semanticTokens/full`, `full/delta`, and `range` use `encoding_for_request` + `read_for_request` — bounded gate with ambient cancellation observed at entry / post-gate / completion — instead of `try_lock_db`. - **Connection-scoped delta cache**: `semantic_tokens_cache` is now per-connection (fresh in `connection_scoped_lsp_session`, next to the encoding negotiation), so concurrent clients can't cross-serve wrongly-encoded tokens or trample each other's `previous_result_id`. The `result_id` sequence stays process-global. - **`returns(ref)`**: the tracked `semantic_tokens` query keeps BoundaryML#4000's `returns(ref)` + `salsa::Update`, avoiding an O(tokens) clone per request. **Not re-landed**: BoundaryML#4002's incidental `pnpm-lock.yaml` refresh (fiddle-proxy entries for an untracked package) — unrelated to BoundaryML#3867 on both sides; the diff is strictly `baml_language/`-scoped. ## New tests - Negotiated-encoding token encoding: UTF-16 vs UTF-8 lengths, astral-plane surrogate pairs, multiline segment splitting (`request.rs`). - Connection-scoped token-cache freshness asserted in the session-scoping test (`mod.rs`). ## Validation - Full suite: 7486/7486 (nextest) + `baml_src baml_test` 2057/2057 standalone. - `bex_project` 52/52 incl. the new encoding + session tests. - clippy clean (workspace, all targets), `cargo check --target wasm32-unknown-unknown -p bex_project` clean.  ## Summary by CodeRabbit - **New Features** - Added richer, type-aware semantic highlighting, including booleans, escape sequences, declarations, properties, methods, enum members, and library symbols. - Added range-based and delta semantic-token updates for faster editor refreshes. - Added a local semantic-token viewer with fixture comparison, scratchpad editing, token inspection, and snapshot acceptance. - Improved navigation to interface fields and methods, including aliased members. - **Bug Fixes** - Improved type information and completion support for destructuring and catch bindings. - Corrected contextual keyword and literal parsing, formatting, and highlighting. - Improved hover results for caught values.

Revert "feat(lsp): resolution-first semantic tokens rework (#3867)"

3908c1c

This reverts commit a05ec5a.

vercel Bot deployed to Preview – beps July 13, 2026 21:20 View deployment

vercel Bot deployed to Preview – promptfiddle2 July 13, 2026 21:27 View deployment

coderabbitai Bot requested changes Jul 13, 2026

View reviewed changes

vercel Bot deployed to Preview – promptfiddle July 13, 2026 21:41 View deployment

rossirpaulo mentioned this pull request Jul 13, 2026

fix(parser): guard raw-string hash lookahead at EOF (re-applied from #3867) #4005

Merged

coderabbitai Bot approved these changes Jul 13, 2026

View reviewed changes

rossirpaulo added this pull request to the merge queue Jul 13, 2026

Merged via the queue into canary with commit 4a9ffe3 Jul 13, 2026
54 checks passed

rossirpaulo deleted the paulo/revert-3867-semantic-tokens branch July 13, 2026 22:10

codeshaunted mentioned this pull request Jul 14, 2026

feat(lsp): re-land resolution-first semantic tokens rework (#3867) on #4000 #4011

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

revert(lsp): temporarily revert #3867 to land #4000 first#4002

revert(lsp): temporarily revert #3867 to land #4000 first#4002
rossirpaulo merged 1 commit into
canaryfrom
paulo/revert-3867-semantic-tokens

rossirpaulo commented Jul 13, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jul 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rossirpaulo commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

rossirpaulo commented Jul 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Re-land guidance (agreed with Avery)

Sequence

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026

⏭️ Performance benchmarks were skipped

Uh oh!

coderabbitai Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

github-actions Bot commented Jul 13, 2026

Binary size checks passed

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rossirpaulo commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

rossirpaulo commented Jul 13, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jul 13, 2026 •

edited

Loading

coderabbitai Bot commented Jul 13, 2026 •

edited

Loading