feat(lsp): re-land resolution-first semantic tokens rework (#3867) on #4000 by codeshaunted · Pull Request #4011 · BoundaryML/baml

codeshaunted · 2026-07-14T00:22:33Z

Re-lands #3867 (reverted in #4002 to let #4000 land first) as a revert-of-revert on top of #4000, following the re-land guidance agreed in #4002.

Byte-for-byte re-land

The classification engine is unchanged from what #3867 shipped: the span-keyed resolution index, the CST classification walk, interface-member resolution, the contextual-keyword re-lexing (as/type/true/false/null), all 78 semantic-token fixtures, and the tools_semantic_tokens viewer.

Adapted to #4000 (LSP boundary only)

Negotiated encoding: encode_semantic_tokens now encodes through the connection's PositionCodec — delta_start/length are negotiated code units (UTF-16 default, surrogate pairs handled), and multiline tokens split into per-line segments. This also fixes the byte-length miscount for non-ASCII identifiers that feat(lsp): resolution-first semantic tokens rework #3867 originally shipped (called out in revert(lsp): temporarily revert #3867 to land #4000 first #4002).
Request idiom: semanticTokens/full, full/delta, and range use encoding_for_request + read_for_request — bounded gate with ambient cancellation observed at entry / post-gate / completion — instead of try_lock_db.
Connection-scoped delta cache: semantic_tokens_cache is now per-connection (fresh in connection_scoped_lsp_session, next to the encoding negotiation), so concurrent clients can't cross-serve wrongly-encoded tokens or trample each other's previous_result_id. The result_id sequence stays process-global.
returns(ref): the tracked semantic_tokens query keeps fix(lsp): resolve the 0.14 mutex/latency regression — Phase 0 containment + Phase 1 bounded ingress #4000's returns(ref) + salsa::Update, avoiding an O(tokens) clone per request.

Not re-landed: #4002's incidental pnpm-lock.yaml refresh (fiddle-proxy entries for an untracked package) — unrelated to #3867 on both sides; the diff is strictly baml_language/-scoped.

New tests

Negotiated-encoding token encoding: UTF-16 vs UTF-8 lengths, astral-plane surrogate pairs, multiline segment splitting (request.rs).
Connection-scoped token-cache freshness asserted in the session-scoping test (mod.rs).

Validation

Full suite: 7486/7486 (nextest) + baml_src baml_test 2057/2057 standalone.
bex_project 52/52 incl. the new encoding + session tests.
clippy clean (workspace, all targets), cargo check --target wasm32-unknown-unknown -p bex_project clean.

Summary by CodeRabbit

New Features
- Added richer, type-aware semantic highlighting, including booleans, escape sequences, declarations, properties, methods, enum members, and library symbols.
- Added range-based and delta semantic-token updates for faster editor refreshes.
- Added a local semantic-token viewer with fixture comparison, scratchpad editing, token inspection, and snapshot acceptance.
- Improved navigation to interface fields and methods, including aliased members.
Bug Fixes
- Improved type information and completion support for destructuring and catch bindings.
- Corrected contextual keyword and literal parsing, formatting, and highlighting.
- Improved hover results for caught values.

…4000 Revert of the temporary revert (4a9ffe3), re-landed on top of #4000's position-encoding negotiation, connection-scoped sessions, and bounded ingress. The classification engine (resolution index, CST walk, keyword re-lexing, tools_semantic_tokens viewer) re-lands byte-for-byte; only the LSP boundary is adapted, per the revert's re-land guidance: - encode_semantic_tokens now goes through the connection's negotiated PositionCodec: delta_start and length are code units (UTF-16 by default, surrogate pairs included), not bytes, and multiline tokens split into per-line segments. This also fixes the byte-length miscount for non-ASCII identifiers that #3867 originally shipped. - semanticTokens full/delta/range all use the #4000 request idiom: encoding_for_request + read_for_request (bounded gate, ambient cancellation observed at entry/post-gate/completion) instead of try_lock_db. - semantic_tokens_cache is connection-scoped (fresh per connection_scoped_lsp_session, next to the encoding negotiation): cached arrays are per-encoding and each client owns its delta baseline. The result_id sequence stays process-global. - The tracked semantic_tokens query keeps #4000's returns(ref) + salsa::Update, avoiding the O(tokens) clone per request. Not re-landed: the revert's incidental pnpm-lock.yaml refresh (fiddle-proxy entries for an untracked package; unrelated to #3867 on both sides). New tests: negotiated-encoding token encoding (UTF-16 vs UTF-8 lengths, astral-plane surrogate pairs, multiline splitting) and connection-scoped token-cache freshness in the session test.

vercel · 2026-07-14T00:22:39Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview, Comment	Jul 14, 2026 12:44am
promptfiddle	Ready	Preview, Comment	Jul 14, 2026 12:44am
promptfiddle2	Ready	Preview, Comment	Jul 14, 2026 12:44am

github-actions · 2026-07-14T00:23:00Z

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

Add RUN_CODSPEED=1 to the PR description, or
Include run-perf or /perf in the PR title or any commit message.

coderabbitai · 2026-07-14T00:25:07Z

📝 Walkthrough

Walkthrough

This PR updates contextual keyword parsing, compiler scope and interface metadata, LSP semantic-token classification, binding resolution, semantic-token transport, extensive semantic-token fixtures, and a new local semantic-token viewer with snapshot reconciliation.

Changes

Semantic token compiler and LSP pipeline

Layer / File(s)	Summary
Compiler and formatter token contracts `baml_compiler2_*`, `baml_compiler_parser`, `baml_compiler_syntax`, `baml_fmt`	Contextual keywords and literals receive dedicated syntax kinds, AST accessors and formatter token types are added, catch bindings gain distinct semantic-index entries, interface name spans are collected, and inference/resolution helpers expose scope bodies and namespace, enum, and field checks.
Semantic-token indexing and classification `baml_lsp2_actions/src/tokens*`, `completions.rs`, `definition.rs`, `describe.rs`, `type_info.rs`	Semantic tokens are generated through a document-order CST walker with resolution-backed classification, token modifiers, escape and boolean token types, range queries, interface-member definition navigation, and inferred types for pattern and catch bindings.
Semantic-token fixtures and robustness tests `baml_lsp2_actions_tests/**`	Fixtures and expected outputs cover contextual syntax, interfaces, generics, patterns, lambdas, literals, prompts, iterators, control flow, bindings, and partial-input robustness.
LSP transport and caching `bex_project/src/bex_lsp/multi_project/*`	Semantic-token capabilities now advertise modifiers, range, and delta support; full, range, and delta requests share encoding logic and use connection-scoped caches.
Local semantic-token viewer `tools_semantic_tokens/**`, `stow.toml`	A new CLI, Axum server, fixture analyzer, browser UI, snapshot acceptance flow, and source-staleness watcher provide local token inspection and fixture updates.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

BoundaryML/baml#3111: Related formatter token and AST infrastructure.
BoundaryML/baml#3188: Earlier semantic-token full-request and capability work.
BoundaryML/baml#3417: Related catch-pattern representation and binding behavior.

Suggested reviewers: rossirpaulo, sxlijin, 2kai2kai2, antoniosarosi

Poem

I hop through tokens, bright and new,
With scopes and catches in my view.
Snapshots bloom, deltas dance,
A viewer gives each span its chance.
The parser twinkles; tests agree—
A tidy burrow of syntax glee!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: re-landing the resolution-first semantic tokens rework on top of `#4000`.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch avery/sem-tok-reland

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-14T00:33:29Z

Binary size checks passed

✅ 7 passed

	Artifact	Platform	File	Gzip	Gated on	Baseline	Delta	Status
✅	`baml-cli`	Linux	🔒 22.1 MB	9.4 MB	file	21.5 MB	+569.2 KB (+2.6%)	OK
✅	`packed-program`	Linux	🔒 15.9 MB	6.7 MB	file	15.6 MB	+228.8 KB (+1.5%)	OK
✅	`baml-cli`	macOS	🔒 17.0 MB	8.2 MB	file	16.6 MB	+432.5 KB (+2.6%)	OK
✅	`packed-program`	macOS	🔒 12.2 MB	5.9 MB	file	12.1 MB	+133.9 KB (+1.1%)	OK
✅	`baml-cli`	Windows	🔒 18.6 MB	8.4 MB	file	18.1 MB	+449.5 KB (+2.5%)	OK
✅	`packed-program`	Windows	🔒 13.1 MB	6.0 MB	file	13.0 MB	+139.7 KB (+1.1%)	OK
✅	`bridge_wasm`	WASM	14.7 MB	🔒 4.1 MB	gzip	4.1 MB	+89.9 KB (+2.2%)	OK

🔒 = the size this artifact is GATED on (ceiling + delta). Binaries gate on file size (installed binary); WASM gates on gzip (download size). The other size is shown for information only.

Generated by cargo size-gate · workflow run

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (4)

baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs (1)
1104-1118: 🚀 Performance & Scalability | 🔵 Trivial

Cache entries are never evicted on didClose.

semantic_tokens_cache keeps the full encoded token array for every path touched during a connection, and nothing removes an entry when a document closes (ProjectRefreshMode::ClosedDocuments doesn't prune it). Growth is bounded by files opened per connection and freed when the session drops, so this is a mild retention concern rather than a leak. If long-lived connections open many files, consider dropping the entry when a document closes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs` around
lines 1104 - 1118, Update the semantic-token cache lifecycle around
cache_semantic_tokens to remove the cached entry when the corresponding document
is closed. In the didClose handling for ProjectRefreshMode::ClosedDocuments,
derive the same FsPath key and delete it from semantic_tokens_cache while
preserving existing caching behavior for open documents.
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml (1)
1-60: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate fixture: identical to if_else.baml.

This file's source and full expected token list are byte-for-byte identical to baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_else.baml. If the intent is to distinguish "if-statement" vs "if-as-expression" coverage, this file's source should exercise something if_else.baml doesn't (e.g. an if-expression used inline as an argument/return, not just via let x = if ...). Otherwise, consider dropping one of the two to avoid two fixtures drifting out of sync as the classifier changes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml`
around lines 1 - 60, Remove the duplicate semantic-token fixture, or revise it
to cover an if-expression scenario not exercised by if_else.baml, such as using
the expression inline in an argument or return. Update the source and complete
expected token list together so the fixtures are no longer byte-for-byte
identical and remain focused on distinct coverage.
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml (1)
1-46: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove duplicate test fixture.

The content of this file is an exact, line-by-line duplicate of implements_for.baml. Consider removing one of the files to avoid duplicate test execution and maintainability overhead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml`
around lines 1 - 46, Remove the duplicate test fixture represented by
interface_impl.baml, retaining the existing implements_for.baml fixture as the
single source for this semantic-token coverage.
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml (1)
19-43: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove leftover copy-pasted comments.

These // error: comments appear to be copy-pasted artifacts from expr/access.baml (as indicated by the path in the text) and are not actively part of the //- diagnostics test runner assertions below.

Consider removing this block to keep the fixture clean. Note that you will also need to regenerate the semantic token snapshots at the bottom of the file to account for the removed lines.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml`
around lines 19 - 43, Remove the leftover // error diagnostic comment block from
the semantic token fixture, leaving the active //- diagnostics assertions
intact. Then regenerate the semantic token snapshots at the bottom of the file
so their expected positions reflect the removed lines.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@baml_language/crates/tools_semantic_tokens/src/server.rs`:
- Around line 140-142: The async handlers perform synchronous fixture
resolution, loading, token computation, and acceptance on Tokio worker threads.
In baml_language/crates/tools_semantic_tokens/src/server.rs lines 140-142, wrap
resolve_fixture and analysis::load_fixture in tokio::task::spawn_blocking and
propagate both join and operation errors; at lines 161-163, wrap compute_tokens
likewise; at lines 180-182, wrap resolve_fixture and accept_fixture likewise,
preserving each handler’s existing response and error mapping.
- Around line 91-122: Update the fixtures handler to replace the
std::thread::scope computation with tokio::task::spawn_blocking, moving fixture
summary calculation into the blocking pool and awaiting its result. Preserve the
existing per-fixture parallelism, error-to-zero behavior, summary construction,
and panic handling while ensuring the async executor is not blocked.

---

Nitpick comments:
In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml`:
- Around line 1-60: Remove the duplicate semantic-token fixture, or revise it to
cover an if-expression scenario not exercised by if_else.baml, such as using the
expression inline in an argument or return. Update the source and complete
expected token list together so the fixtures are no longer byte-for-byte
identical and remain focused on distinct coverage.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml`:
- Around line 19-43: Remove the leftover // error diagnostic comment block from
the semantic token fixture, leaving the active //- diagnostics assertions
intact. Then regenerate the semantic token snapshots at the bottom of the file
so their expected positions reflect the removed lines.

In
`@baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml`:
- Around line 1-46: Remove the duplicate test fixture represented by
interface_impl.baml, retaining the existing implements_for.baml fixture as the
single source for this semantic-token coverage.

In `@baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs`:
- Around line 1104-1118: Update the semantic-token cache lifecycle around
cache_semantic_tokens to remove the cached entry when the corresponding document
is closed. In the didClose handling for ProjectRefreshMode::ClosedDocuments,
derive the same FsPath key and delete it from semantic_tokens_cache while
preserving existing caching behavior for open documents.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 67ac656b-705a-4643-9faa-9fdbe6b041c2

📥 Commits

Reviewing files that changed from the base of the PR and between 30eb1c1 and 68d04bf.

⛔ Files ignored due to path filters (66)

baml_language/Cargo.lock is excluded by !**/*.lock
baml_language/crates/baml_tests/snapshots/broken_syntax/optional_parameter_defaults/baml_tests__broken_syntax__optional_parameter_defaults__02_parser__function_type_defaults.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/pending_greaters_fix/baml_tests__broken_syntax__pending_greaters_fix__02_parser__extra_greater.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/test_invalid_contexts/baml_tests__broken_syntax__test_invalid_contexts__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/broken_syntax/type_annotation_errors/baml_tests__broken_syntax__type_annotation_errors__02_parser__type_malformed.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/config_dictionary/baml_tests__compiles__config_dictionary__02_parser__valid_dictionary.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/json_alias_basic/baml_tests__compiles__json_alias_basic__02_parser__json_alias_basic.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/json_map_literal/baml_tests__compiles__json_map_literal__02_parser__json_map_literal.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/lambda_advanced/baml_tests__compiles__lambda_advanced__02_parser__lambda_advanced.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/lexical_scoping/baml_tests__compiles__lexical_scoping__02_parser__lexical_scoping.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/literal_union_arithmetic/baml_tests__compiles__literal_union_arithmetic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/llm_parse_catchable_parse_error/baml_tests__compiles__llm_parse_catchable_parse_error__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/method_explicit_type_args/baml_tests__compiles__method_explicit_type_args__02_parser__method_explicit_type_args.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_auth_auth.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/namespaces_type_resolution/baml_tests__compiles__namespaces_type_resolution__02_parser__ns_llm_models.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/optional_function_parameters/baml_tests__compiles__optional_function_parameters__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/parser_expressions/baml_tests__compiles__parser_expressions__02_parser__precedence.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/parser_statements/baml_tests__compiles__parser_statements__02_parser__break_continue.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/patterns_new/baml_tests__compiles__patterns_new__02_parser__patterns_new.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/stream_crossfile/baml_tests__compiles__stream_crossfile__02_parser__file_a.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/test_expr_name_concat/baml_tests__compiles__test_expr_name_concat__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/test_raw_string_name/baml_tests__compiles__test_raw_string_name__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_basic/baml_tests__compiles__testset_basic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_dynamic/baml_tests__compiles__testset_dynamic__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/testset_nested/baml_tests__compiles__testset_nested__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_alias_interaction.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_annotation/baml_tests__compiles__type_annotation__02_parser__type_positions.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/compiles/type_builder_test/baml_tests__compiles__type_builder_test__02_parser__test.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/attr_disambiguation/baml_tests__diagnostic_errors__attr_disambiguation__02_parser__non_field.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/backtick_strict_types/baml_tests__diagnostic_errors__backtick_strict_types__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/basic_types/baml_tests__diagnostic_errors__basic_types__02_parser__types.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__io_ops.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/builtin_io/baml_tests__diagnostic_errors__builtin_io__02_parser__shell_ops.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/catch_arm_braceless_return/baml_tests__diagnostic_errors__catch_arm_braceless_return__02_parser__repro.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/catch_return_type_mismatch/baml_tests__diagnostic_errors__catch_return_type_mismatch__02_parser__catch_return_type_mismatch.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/duplicate_class_span/baml_tests__diagnostic_errors__duplicate_class_span__02_parser__repro.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__binary_expr.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__config_decls.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__loop_stmts.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__match_exprs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__other_exprs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/format_checks/baml_tests__diagnostic_errors__format_checks__02_parser__type_alias_decls.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/function_type_throws/baml_tests__diagnostic_errors__function_type_throws__02_parser__negative.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/function_types/baml_tests__diagnostic_errors__function_types__02_parser__function_types.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/generics/baml_tests__diagnostic_errors__generics__02_parser__classes.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/headers_edge_cases/baml_tests__diagnostic_errors__headers_edge_cases__02_parser__edge_cases.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/match_exhaustiveness/baml_tests__diagnostic_errors__match_exhaustiveness__02_parser__match_exhaustiveness.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/null_handling/baml_tests__diagnostic_errors__null_handling__02_parser__null_handling.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/optional_parameter_defaults/baml_tests__diagnostic_errors__optional_parameter_defaults__02_parser__unsupported_contexts.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__array_constructor.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/parser_constructors/baml_tests__diagnostic_errors__parser_constructors__02_parser__map_constructor.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/self_in_body/baml_tests__diagnostic_errors__self_in_body__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__complex.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/stream_types/baml_tests__diagnostic_errors__stream_types__02_parser__ns_foo_with_top_level_refs.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_name_type_error/baml_tests__diagnostic_errors__test_expr_name_type_error__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_expr_wrong_runner/baml_tests__diagnostic_errors__test_expr_wrong_runner__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/test_with_runner_ambiguity/baml_tests__diagnostic_errors__test_with_runner_ambiguity__02_parser__main.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/type_aliases/baml_tests__diagnostic_errors__type_aliases__02_parser__type_aliases_json.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/unknown_type_error/baml_tests__diagnostic_errors__unknown_type_error__02_parser__unknown_type_error.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__function_type_throws.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_function_type.snap is excluded by !**/*.snap
baml_language/crates/baml_tests/snapshots/diagnostic_errors/void_return_type/baml_tests__diagnostic_errors__void_return_type__02_parser__void_type_position_errors.snap is excluded by !**/*.snap
pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (122)

baml_language/Cargo.toml
baml_language/crates/baml_cli/src/paint.rs
baml_language/crates/baml_compiler2_ast/src/lower_expr_body.rs
baml_language/crates/baml_compiler2_hir/src/builder.rs
baml_language/crates/baml_compiler2_hir/src/item_tree.rs
baml_language/crates/baml_compiler2_hir/src/semantic_index.rs
baml_language/crates/baml_compiler2_mir/src/lower.rs
baml_language/crates/baml_compiler2_tir/src/inference.rs
baml_language/crates/baml_compiler2_tir/src/resolve.rs
baml_language/crates/baml_compiler_parser/src/parser.rs
baml_language/crates/baml_compiler_syntax/src/ast.rs
baml_language/crates/baml_compiler_syntax/src/syntax_kind.rs
baml_language/crates/baml_fmt/src/ast/declarations.rs
baml_language/crates/baml_fmt/src/ast/expressions.rs
baml_language/crates/baml_fmt/src/ast/tokens.rs
baml_language/crates/baml_fmt/src/ast/types.rs
baml_language/crates/baml_lsp2_actions/Cargo.toml
baml_language/crates/baml_lsp2_actions/src/completions.rs
baml_language/crates/baml_lsp2_actions/src/definition.rs
baml_language/crates/baml_lsp2_actions/src/definition_at_tests.rs
baml_language/crates/baml_lsp2_actions/src/describe.rs
baml_language/crates/baml_lsp2_actions/src/lib.rs
baml_language/crates/baml_lsp2_actions/src/tokens.rs
baml_language/crates/baml_lsp2_actions/src/tokens/classify.rs
baml_language/crates/baml_lsp2_actions/src/tokens/index.rs
baml_language/crates/baml_lsp2_actions/src/type_info.rs
baml_language/crates/baml_lsp2_actions_tests/src/lib.rs
baml_language/crates/baml_lsp2_actions_tests/src/range_tokens_test.rs
baml_language/crates/baml_lsp2_actions_tests/src/runner.rs
baml_language/crates/baml_lsp2_actions_tests/src/typing_robustness_test.rs
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/array_pattern.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/backtick_string.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/bigint_literals.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/block_attributes.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/break_continue.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/byte_strings.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/call_named_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/cancel_token.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_all_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_param.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/catch_throw.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/class.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/client.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/config_dictionary.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_binding.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/const_let_else_defer.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/constructor_spread.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/deep_method_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/dynamic_type_builder.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_decls.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/enum_member_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/exhaustive.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/extends_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/field_alias.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/for_loops.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/function.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generators.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_field_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/generic_function_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/header_comments.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/http_config.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_else.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_expression.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/if_let_chain.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/implements_for.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/index_access.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interface_impl.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_inferred_generic_type_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_iter_core.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/interfaces_sort_comparable.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/is_operator.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/iterator.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/jinja_control_prompt.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_map_literal.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/json_parse_stringify_intrinsics.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/lambda_advanced.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/llm_function.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_literal_keys.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/map_type_and_methods.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_expr.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/match_literal_types.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/method_explicit_type_args.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/multi_segment_path.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_agent_clients.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_ansi.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_game_clients.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_images_pipeline.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/ns_sentiment.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/patterns_new.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/remap_role.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/requires_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/retry_policy_valid_retry.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/self_qualified_call.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/signature_variety.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/stream_llm_inferred_typeargs.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/task_group.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/template_string_calls.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_block.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/test_expr_throwing_body.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/testset_vibes_nested.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/throws_clause.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/top_level_binding.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_alias.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/type_aliases_jinja.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/watch_let.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/while_loop.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/semantic_tokens/with_keyword.baml
baml_language/crates/baml_lsp2_actions_tests/test_files/syntax/hover/catch_binding.baml
baml_language/crates/bex_project/src/bex_lsp/multi_project/mod.rs
baml_language/crates/bex_project/src/bex_lsp/multi_project/request.rs
baml_language/crates/tools_semantic_tokens/Cargo.toml
baml_language/crates/tools_semantic_tokens/README.md
baml_language/crates/tools_semantic_tokens/src/analysis.rs
baml_language/crates/tools_semantic_tokens/src/index.html
baml_language/crates/tools_semantic_tokens/src/main.rs
baml_language/crates/tools_semantic_tokens/src/server.rs
baml_language/crates/tools_semantic_tokens/src/staleness.rs
baml_language/stow.toml

coderabbitai · 2026-07-14T00:48:17Z

+async fn fixtures(State(state): State<AppState>) -> Result<Json<FixturesResponse>, ApiError> {
+    let dir = state.fixtures_dir.as_ref();
+    let names = analysis::list_fixture_names(dir)
+        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;
+
+    // Each fixture builds its own ProjectDatabase + runs the compiler, so compute
+    // the diff badges in parallel (one thread per fixture) rather than serially.
+    let summaries = std::thread::scope(|scope| {
+        let handles: Vec<_> = names
+            .iter()
+            .map(|name| {
+                scope.spawn(move || {
+                    let diff_count = analysis::load_fixture(&dir.join(name))
+                        .map(|fx| analysis::diff_count(&fx.current, &fx.expected))
+                        .unwrap_or(0);
+                    FixtureSummary {
+                        name: name.clone(),
+                        diff_count,
+                    }
+                })
+            })
+            .collect();
+        handles
+            .into_iter()
+            .map(|h| h.join().expect("fixture summary thread panicked"))
+            .collect()
+    });
+
+    Ok(Json(FixturesResponse {
+        fixtures: summaries,
+    }))
+}


🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Avoid blocking the async executor and spawning unconstrained OS threads.

Using std::thread::scope inside an async fn blocks the Tokio worker thread until all scoped threads finish, starving the async executor. Additionally, since it spawns an OS thread per fixture concurrently, it can cause severe resource contention.

Please offload this parallel computation to Tokio's built-in blocking thread pool using tokio::task::spawn_blocking.

⚡ Proposed refactor

-async fn fixtures(State(state): State<AppState>) -> Result<Json<FixturesResponse>, ApiError> { - let dir = state.fixtures_dir.as_ref(); - let names = analysis::list_fixture_names(dir) - .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?; - - // Each fixture builds its own ProjectDatabase + runs the compiler, so compute - // the diff badges in parallel (one thread per fixture) rather than serially. - let summaries = std::thread::scope(|scope| { - let handles: Vec<_> = names - .iter() - .map(|name| { - scope.spawn(move || { - let diff_count = analysis::load_fixture(&dir.join(name)) - .map(|fx| analysis::diff_count(&fx.current, &fx.expected)) - .unwrap_or(0); - FixtureSummary { - name: name.clone(), - diff_count, - } - }) - }) - .collect(); - handles - .into_iter() - .map(|h| h.join().expect("fixture summary thread panicked")) - .collect() - }); - - Ok(Json(FixturesResponse { - fixtures: summaries, - })) -} +async fn fixtures(State(state): State<AppState>) -> Result<Json<FixturesResponse>, ApiError> { + let names = analysis::list_fixture_names(&state.fixtures_dir) + .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?; + + // Each fixture builds its own ProjectDatabase + runs the compiler, so compute + // the diff badges in parallel using Tokio's blocking thread pool. + let mut handles = Vec::with_capacity(names.len()); + for name in names { + let dir = Arc::clone(&state.fixtures_dir); + handles.push(tokio::task::spawn_blocking(move || { + let diff_count = analysis::load_fixture(&dir.join(&name)) + .map(|fx| analysis::diff_count(&fx.current, &fx.expected)) + .unwrap_or(0); + FixtureSummary { name, diff_count } + })); + } + + let mut summaries = Vec::with_capacity(handles.len()); + for handle in handles { + summaries.push(handle.await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?); + } + + Ok(Json(FixturesResponse { + fixtures: summaries, + })) +}

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

async fn fixtures(State(state): State<AppState>) -> Result<Json<FixturesResponse>, ApiError> {

let dir = state.fixtures_dir.as_ref();

let names = analysis::list_fixture_names(dir)

.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;

// Each fixture builds its own ProjectDatabase + runs the compiler, so compute

// the diff badges in parallel (one thread per fixture) rather than serially.

let summaries = std::thread::scope(|scope| {

let handles: Vec<_> = names

.iter()

.map(|name| {

scope.spawn(move || {

let diff_count = analysis::load_fixture(&dir.join(name))

.map(|fx| analysis::diff_count(&fx.current, &fx.expected))

.unwrap_or(0);

FixtureSummary {

name: name.clone(),

diff_count,

}

})

})

.collect();

handles

.into_iter()

.map(|h| h.join().expect("fixture summary thread panicked"))

.collect()

});

Ok(Json(FixturesResponse {

fixtures: summaries,

}))

}

async fn fixtures(State(state): State<AppState>) -> Result<Json<FixturesResponse>, ApiError> {

let names = analysis::list_fixture_names(&state.fixtures_dir)

.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;

// Each fixture builds its own ProjectDatabase + runs the compiler, so compute

// the diff badges in parallel using Tokio's blocking thread pool.

let mut handles = Vec::with_capacity(names.len());

for name in names {

let dir = Arc::clone(&state.fixtures_dir);

handles.push(tokio::task::spawn_blocking(move || {

let diff_count = analysis::load_fixture(&dir.join(&name))

.map(|fx| analysis::diff_count(&fx.current, &fx.expected))

.unwrap_or(0);

FixtureSummary { name, diff_count }

}));

}

let mut summaries = Vec::with_capacity(handles.len());

for handle in handles {

summaries.push(handle.await.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?);

}

Ok(Json(FixturesResponse {

fixtures: summaries,

}))

}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/tools_semantic_tokens/src/server.rs` around lines 91 - 122, Update the fixtures handler to replace the std::thread::scope computation with tokio::task::spawn_blocking, moving fixture summary calculation into the blocking pool and awaiting its result. Preserve the existing per-fixture parallelism, error-to-zero behavior, summary construction, and panic handling while ensuring the async executor is not blocked.

coderabbitai · 2026-07-14T00:48:17Z

+    let path = resolve_fixture(&state.fixtures_dir, &query.name)?;
+    let fx = analysis::load_fixture(&path)
+        .map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?;


🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Blocking I/O and CPU-intensive operations on the async executor.

These synchronous functions execute file I/O and run the heavy compiler pipeline. Executing them directly inside an async fn blocks the Tokio worker thread, preventing it from processing other requests. Offload these operations to Tokio's blocking thread pool.

baml_language/crates/tools_semantic_tokens/src/server.rs#L140-L142: wrap the resolve_fixture and load_fixture calls in tokio::task::spawn_blocking.

baml_language/crates/tools_semantic_tokens/src/server.rs#L161-L163: wrap the compute_tokens call in tokio::task::spawn_blocking.

baml_language/crates/tools_semantic_tokens/src/server.rs#L180-L182: wrap the resolve_fixture and accept_fixture calls in tokio::task::spawn_blocking.

📍 Affects 1 file

baml_language/crates/tools_semantic_tokens/src/server.rs#L140-L142 (this comment)

baml_language/crates/tools_semantic_tokens/src/server.rs#L161-L163

baml_language/crates/tools_semantic_tokens/src/server.rs#L180-L182

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@baml_language/crates/tools_semantic_tokens/src/server.rs` around lines 140 - 142, The async handlers perform synchronous fixture resolution, loading, token computation, and acceptance on Tokio worker threads. In baml_language/crates/tools_semantic_tokens/src/server.rs lines 140-142, wrap resolve_fixture and analysis::load_fixture in tokio::task::spawn_blocking and propagate both join and operation errors; at lines 161-163, wrap compute_tokens likewise; at lines 180-182, wrap resolve_fixture and accept_fixture likewise, preserving each handler’s existing response and error mapping.

…okens (BoundaryML#4017) Follow-up to BoundaryML#4011 (now on canary). Fixes three gaps that made `baml describe` highlighting look worse than the same code in an editor: 1. **Background-agnostic palette.** Colors are restricted to the terminal's named ANSI palette (slots 0–15, always theme-controlled) plus dim/bold attributes. The fixed 256-color-cube values and the forced-white variables are gone: ordinary names render at the default foreground, comments/operators are dim. Output is legible on light and dark terminals alike, styled by the user's own scheme. 2. **Semantic-token modifiers are honored.** Declarations render bold, `defaultLibrary` (stdlib) entities italic, deprecated ones struck through — the same signals editor themes derive from the LSP modifier bitset, previously discarded. Dim-base token types trade dim for bold at their declaration instead of stacking contradictory weights. 3. **Fragments use the exact LSP classifier.** Synthesized snippets (keyword-doc examples, interface member signatures) are parsed as a scratch file in a thread-local in-memory project and rendered from real `semantic_tokens` — replacing the hand-rolled lexer scanner that colored only keywords/primitives/literals. Names a context-free fragment can't resolve stay neutral, exactly like an editor. The scanner (`PRIMITIVE_TYPES`, `decl_keyword_kind`, `is_keyword`) is deleted and `Highlighter::range`/`highlight_str` now share one render loop. ## Tests - New unit tests: variables keep the default foreground; declaration/defaultLibrary modifier overlays; a rendered body contains no fixed-cube 256-color codes and no forced white/black; fragments classify via the compiler (bold declaration + italic stdlib type — signals the old lexer could never emit); unresolvable fragment names stay neutral. - `baml_cli` suite 343/343 (describe snapshots are colorless and unaffected); clippy clean. - Verified end-to-end on a scratch project and on the `describe defer` keyword page with forced color.  ## Summary by CodeRabbit * **Bug Fixes** * Improved syntax highlighting so both real source code and generated fragments use the same semantic-token-based styling. * Correctly honors semantic modifiers (e.g., declaration emphasis, default-library italics, deprecation strikethrough, and dimming for non-declarations). * Keeps unresolved names neutral and avoids highlighting errors/crashes. * Preserves terminal default colors—no forced black/white and no fixed palette indices—while ensuring modifier overlays render correctly.

vercel Bot deployed to Preview – beps July 14, 2026 00:23 View deployment

vercel Bot deployed to Preview – promptfiddle2 July 14, 2026 00:30 View deployment

codeshaunted enabled auto-merge July 14, 2026 00:38

codeshaunted added this pull request to the merge queue Jul 14, 2026

vercel Bot deployed to Preview – promptfiddle July 14, 2026 00:44 View deployment

coderabbitai Bot requested changes Jul 14, 2026

View reviewed changes

Merged via the queue into canary with commit 8607471 Jul 14, 2026
54 checks passed

codeshaunted deleted the avery/sem-tok-reland branch July 14, 2026 01:01

codeshaunted mentioned this pull request Jul 14, 2026

feat(cli): background-agnostic describe highlighting with LSP-exact tokens #4017

Merged

coderabbitai Bot mentioned this pull request Jul 15, 2026

feat(compiler2): require explicit throws on function types (E0151) #4034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(lsp): re-land resolution-first semantic tokens rework (#3867) on #4000#4011

feat(lsp): re-land resolution-first semantic tokens rework (#3867) on #4000#4011
codeshaunted merged 1 commit into
canaryfrom
avery/sem-tok-reland

codeshaunted commented Jul 14, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

vercel Bot commented Jul 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 14, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Uh oh!

coderabbitai Bot Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

codeshaunted commented Jul 14, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Byte-for-byte re-land

Adapted to #4000 (LSP boundary only)

New tests

Validation

Summary by CodeRabbit

Uh oh!

vercel Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 14, 2026

⏭️ Performance benchmarks were skipped

Uh oh!

coderabbitai Bot commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jul 14, 2026

Binary size checks passed

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codeshaunted commented Jul 14, 2026 •

edited by coderabbitai Bot

Loading

vercel Bot commented Jul 14, 2026 •

edited

Loading

coderabbitai Bot commented Jul 14, 2026 •

edited

Loading