Add composable test profiles and canonical test IDs

[hellovai]

Why

baml test needs a cheap default when a project also contains HTTP, LLM, or other expensive tests. BAML should not invent magic categories such as llm or integration; projects should choose their own grouping conventions.

A profile is a named preset argument vector for baml test, using the same syntax taught by baml test --help.

[test]
default = "regular"

[test.profiles.regular]
args = ["-x", "::integration::"]

[test.profiles.integration]
args = ["-i", "::integration::"]

baml test                                # use the default regular profile
baml test --profile integration          # opt into integration tests
baml test --profile integration -i hello # integration AND hello
baml test --no-profile                   # bypass the default profile

Selector model

Every test has a canonical, project-local ID beginning with root:

• root::smoke
• root.orders::integration::creates_order
• root.orders.ChargeCard::declined_card

Selectors are case-sensitive and operate on the full ID:

• A value without * is a substring filter, so -i hello finds every ID containing hello.
• A value containing * is an anchored glob, so -i 'root.orders::*' selects the orders namespace.
• Repeated includes are OR within one source. Exclusions accumulate and always win.
• Profile includes establish the initial candidates; explicit CLI includes narrow that set instead of reopening it.

This keeps common commands natural for humans and models while retaining explicit structural patterns for saved profiles. There is no special llm tag, test annotation, or selector function in this version.

Profiles and precedence

Profile args are parsed by the real baml test CLI parser without shell expansion. Direct scalar options such as --color override profile scalar options. Profile args cannot contain bootstrap options that resolve the profile itself: --profile, --no-profile, --from, or --help.

If [test].default is absent, plain baml test continues to run all tests.

IDs, output, and compatibility

Canonical hierarchy uses . for BAML namespace/function ownership and :: for testset/test nesting. baml test --list is the source of truth and its IDs are reused in PASS, FAIL, and TOLERATED result lines. Runner-only verdicts are labeled AGGREGATE.

This intentionally replaces legacy slash-shaped structural selectors. Unambiguous old selectors receive an actionable :: migration suggestion, while literal / remains legal inside names.

TestSetReport.result_names is optional so existing custom testset runners remain source-compatible. Older runners that execute only part of a selection without returning identities no longer fabricate PASS lines for skipped leaves.

Safety and performance

Lazy testsets are pruned before their collectors run whenever either the profile or explicit CLI filters prove that the subtree cannot contribute a selected test. This prevents excluded collectors from making HTTP/LLM calls or other side effects. Plain substring exclusions can also prune a subtree when the substring is already present in its canonical prefix.

Filtered discovery never repopulates an unfiltered cache, expansion failures are not cached, and invalid dynamic names containing reserved :: surface as discovery errors. The discovery-cache payload carries canonical IDs, intentionally invalidating old cached payloads through its serialized shape.

Review fixes

The follow-up review fixes:

• prevent legacy partial custom runners from assigning skipped names to aggregate PASS counts;
• require a bounded canonical root, root., or root:: prefix during slash-migration validation;
• make AST fallback namespace validation match HIR identifier rules;
• remove dead profile feature-merging state and the corresponding documentation claim.

The CLI discovery path already propagates invalid dynamic test names before execution. Re-throwing from the lazy test child thunk would violate its throws never contract, so that CodeRabbit suggestion was intentionally not applied.

Testing

• cargo test -p baml_cli --lib: 390/390
• cargo test -p baml_cli --test test_profiles_e2e -- --test-threads=1: 9/9
• custom legacy-runner identity regression
• AST/HIR namespace validation regression
• testing stdlib HIR/TIR/MIR/codegen snapshots: 8/8
• cargo clippy -p baml_cli --all-targets --all-features -- -D warnings
• repository pre-commit hooks, including workspace clippy, formatting, and Markdown validation

Review process

An implementation agent completed the first pass. A separate adversarial agent received only the specification and exercised the feature as a user. Its initial no-ship findings around lazy pruning, dynamic names, caching, literal slashes, parser/help behavior, and console identities were fixed and retested before this PR was opened.

Summary by CodeRabbit

• New Features

  • Added test profiles via baml.toml [test] / [test.profiles.*], including profile defaulting and args.
  • Introduced stable canonical root::... test identifiers across listing, filtering, execution, and caching.
  • Implemented layered include/exclude selector matching for profile + CLI combinations.

• Bug Fixes

  • Improved leaf-level reporting (PASS, TOLERATED, FAIL) with correct skipped-leaf and tolerant aggregation behavior.
  • Better handling of lazy expansion sentinels and retries after transient expansion failures.
  • Added clearer errors for invalid test names and legacy /-style selectors.

• CLI

  • Fixed baml test --color always behavior and refreshed help/keyword rendering.

• Documentation

  • Updated Markdown allowlist for long-term architecture content.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview, Comment	Jul 23, 2026 9:09pm
promptfiddle	Ready	Preview, Comment	Jul 23, 2026 9:09pm
promptfiddle2	Ready	Preview, Comment	Jul 23, 2026 9:09pm

[github-actions]

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

• Add RUN_CODSPEED=1 to the PR description, or
• Include run-perf or /perf in the PR title or any commit message.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The test system now uses canonical root::... identifiers, owner-aware registration, layered profile and CLI filtering, canonical discovery caching, and name-aligned pass/fail/tolerated reporting.

Changes

Canonical registration and registry execution

Layer / File(s)	Summary
Owner-aware registration baml_language/crates/baml_compiler2_ast/src/*, baml_language/crates/baml_compiler2_hir/src/lib.rs, baml_language/crates/baml_builtins2/baml_std/testing/registry.baml	Compiler lowering derives test owners from file ns_... namespace paths and emits scoped register_test_at/register_test_set_at calls; reserved :: names raise InvalidTestName discovery errors.
Layered selection and reporting baml_language/crates/baml_builtins2/baml_std/testing/*	Registry traversal applies profile and CLI selectors independently, prunes lazy testsets before expansion when excluded, emits canonical <testset>::(failed to expand) sentinels only if selected, and aligns executed leaf identities with passed/failed/tolerated outcome names via result_names propagation and flatten_with_tolerated.
CLI profiles, discovery, and filtering baml_language/crates/baml_cli/src/{manifest.rs,test_command.rs,test_filter.rs,bytecode_cache.rs,commands.rs}	TOML profiles define per-testset selector args and output overrides; CLI resolves merged TestInvocation state, qualifies legacy test IDs with namespace prefixes and derives canonical IDs, updates discovery caches with canonical records, and filters by canonical selectors via substring or glob matching.
Leaf and aggregate output baml_language/crates/baml_cli/src/test_command.rs	Per-leaf PASS, TOLERATED, and FAIL lines now use canonical IDs; aggregate outcomes report PASS/TOLERATED/FAIL counts and skipped leaves are never rendered as passed.
Testing and documentation baml_language/crates/baml_cli/tests/*, baml_language/crates/baml_builtins2/keyword_docs/baml_keywords.yaml, baml_language/.markdown-whitelist	E2E coverage validates profile parsing, selector pruning, reserved-name rejection, cache durability, canonical output identity, skipped-leaf reporting, and lazy expansion caching; keyword docs and Markdown whitelist updated for canonical IDs and modern test syntax.

Estimated code review effort: 4 (Complex) | ~65 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant resolve_invocation
  participant TestRegistry
  participant flatten_with_tolerated
  participant consume_flat_report
  CLI->>resolve_invocation: parse profile and merge selectors
  resolve_invocation-->>CLI: TestInvocation (merged filters + overrides)
  CLI->>TestRegistry: run_filtered with profile & CLI include/exclude
  TestRegistry->>TestRegistry: select_names_layered, run selected/all
  TestRegistry-->>flatten_with_tolerated: TestSetReport with result_names
  flatten_with_tolerated-->>consume_flat_report: passed_names, tolerated_names, failed_names
  consume_flat_report-->>CLI: per-leaf + aggregate output

Loading

Possibly related PRs

• BoundaryML/baml#3849: Continues the registry filtering and flattening changes updated here.
• BoundaryML/baml#3828: Extends the same test-reporting name propagation pipeline.
• BoundaryML/baml#3924: Modifies the bytecode cache plumbing for canonical test IDs introduced here.

Suggested reviewers: aaronvg, codeshaunted

Poem

A rabbit hops through root:: trails,
With profiles tucked in carrot pails.
PASS and FAIL now know their names,
Tolerated leaves join the games.
Lazy sets sleep when filters call—
Canonical carrots for all! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the two main changes: composable test profiles and canonical test IDs.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/baml-test-profiles

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@baml_language/crates/baml_builtins2/baml_std/testing/registry.baml`:
- Around line 716-724: Update the identities fallback around
classify_selected_names so selection-based classification is used only when
selected_names.length() equals counts.total, in addition to the existing
partition-count validation. For partial custom-runner execution without
result_names, avoid classifying uninvoked leaves as passed; preserve
runner-provided identities when available.
- Around line 17-22: Update both testset_child and testset_child_selected so
their catch-all handling does not intercept InvalidTestName; rethrow this typed
error on every execution path while preserving conversion of other execution
failures into the existing testset error leaf.

In `@baml_language/crates/baml_cli/src/test_command.rs`:
- Around line 578-584: Update the test command flow around the
`invocation.features` collection so the merged profile and CLI features are
passed through the existing feature-activation path before database/engine
compilation. Ensure repeated values from both sources are combined and honored,
and remove or otherwise resolve the unused `invocation.features` state.
- Around line 153-167: Update the root-prefix check in validate_selectors so
only a bounded canonical root prefix is exempt, rather than any selector
beginning with the characters “root”. Ensure selectors such as
“rooted::nested/case” still trigger the legacy separator diagnostic, while valid
root-prefixed selectors retain their current behavior.

In `@baml_language/crates/baml_compiler2_ast/src/lower_cst.rs`:
- Around line 2134-2137: Update the namespace validation in the path-derived
owner logic around the valid check in lower_file_with_path to match
file_package::extract_ns_name: require the namespace name to be non-empty, begin
with an ASCII alphabetic character or underscore, and allow only ASCII
alphanumeric characters or underscores thereafter. Preserve the existing
namespace push behavior for valid names.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 9efa8a8e-0d8f-46cb-8a34-34a606add46d

📥 Commits

Reviewing files that changed from the base of the PR and between 1ebf901 and b6179cb.

⛔ Files ignored due to path filters (5)

• baml_language/crates/baml_cli/src/snapshots/baml_cli__describe_command_tests__render_testing_package_listing.snap is excluded by !**/*.snap
• baml_language/crates/baml_tests/snapshots/compiles/__testing_std__/baml_tests__compiles____testing_std____03_hir.snap is excluded by !**/*.snap
• baml_language/crates/baml_tests/snapshots/compiles/__testing_std__/baml_tests__compiles____testing_std____04_5_mir.snap is excluded by !**/*.snap
• baml_language/crates/baml_tests/snapshots/compiles/__testing_std__/baml_tests__compiles____testing_std____04_tir.snap is excluded by !**/*.snap
• baml_language/crates/baml_tests/snapshots/compiles/__testing_std__/baml_tests__compiles____testing_std____06_codegen.snap is excluded by !**/*.snap

📒 Files selected for processing (16)

• baml_language/.markdown-whitelist
• baml_language/architecture/test-profiles.md
• baml_language/crates/baml_builtins2/baml_std/testing/registry.baml
• baml_language/crates/baml_builtins2/baml_std/testing/runners.baml
• baml_language/crates/baml_builtins2/baml_std/testing/types.baml
• baml_language/crates/baml_cli/src/bytecode_cache.rs
• baml_language/crates/baml_cli/src/commands.rs
• baml_language/crates/baml_cli/src/manifest.rs
• baml_language/crates/baml_cli/src/test_command.rs
• baml_language/crates/baml_cli/src/test_filter.rs
• baml_language/crates/baml_cli/tests/exit_code_e2e.rs
• baml_language/crates/baml_cli/tests/test_list_discovery_cache_e2e.rs
• baml_language/crates/baml_cli/tests/test_profiles_e2e.rs
• baml_language/crates/baml_compiler2_ast/src/lib.rs
• baml_language/crates/baml_compiler2_ast/src/lower_cst.rs
• baml_language/crates/baml_compiler2_hir/src/lib.rs

[github-actions]

Binary size checks passed

✅ 7 passed

	Artifact	Platform	File	Gzip	Gated on	Baseline	Delta	Status
✅	baml-cli	Linux	🔒 26.9 MB	11.4 MB	file	26.2 MB	+669.6 KB (+2.6%)	OK
✅	packed-program	Linux	🔒 17.4 MB	7.2 MB	file	17.0 MB	+392.9 KB (+2.3%)	OK
✅	baml-cli	macOS	🔒 20.8 MB	10.0 MB	file	20.3 MB	+529.9 KB (+2.6%)	OK
✅	packed-program	macOS	🔒 13.6 MB	6.3 MB	file	13.2 MB	+347.3 KB (+2.6%)	OK
✅	baml-cli	Windows	🔒 22.5 MB	10.2 MB	file	21.9 MB	+571.9 KB (+2.6%)	OK
✅	packed-program	Windows	🔒 14.5 MB	6.4 MB	file	14.2 MB	+342.2 KB (+2.4%)	OK
✅	bridge_wasm	WASM	16.4 MB	🔒 4.5 MB	gzip	4.4 MB	+71.8 KB (+1.6%)	OK

🔒 = the size this artifact is GATED on (ceiling + delta). Binaries gate on file size (installed binary); WASM gates on gzip (download size). The other size is shown for information only.

---

Generated by cargo size-gate · workflow run