Skip to content

fix(sandbox): make the code-source item check advisory - #172

Merged
M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/item-scan-advisory
Oct 22, 2026
Merged

M3gA-Mind merged 1 commit into
tinyhumansai:mainfrom
M3gA-Mind:fix/item-scan-advisory

Conversation

@M3gA-Mind

@M3gA-Mind M3gA-Mind commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

v0.8.29 (#166) made code_source_errors fail the dry run. That check is a token scan: it decides whether item is bound by matching a fixed list of binding forms, and every item outside that list counts as a read. Valid scripts that bind item any other way are flagged, the dry run returns ok: false, and the message tells the builder the script "fails every real run", so the builder rewrites working code. Forms it misreads include JS const { json: item } = input[0], input.map(({ json: item }) => …), method shorthand pick(item) { … }, and Python with … as item:, except E as item:, item := …, def f(*item):, for (i, item) in ….

Deciding whether item is bound is scope analysis over two languages, and some cases cannot be decided by a token scan ({ json: item } is a binding in const { json: item } = x but a read in return { json: item }). So this makes the check advisory instead of adding more heuristics:

  • AuthoringReport::is_failing no longer counts code_source_errors (its one non-test caller is the sandbox dry run).
  • The entry message is hedged: the node "appears to read" an undefined item; "if it does, every real run fails".
  • The dry-run summary names flagged nodes on a POSSIBLE UNDEFINED \item`line afterCODE NOT EXECUTED, in both the okand the failing branch, so anok: true` result does not bury the hint in a JSON field. The sentence about it is removed from the failing message.
  • The field name code_source_errors and the scanner are unchanged. The CODE NOT EXECUTED … UNVERIFIED line from fix(code): teach the input contract and flag unexecuted code in dry runs #166 still tells the builder no script ran.

API Or Behavior Changes

  • A flow whose code node the scan flags now dry-runs ok: true (with the hint and the entry in code_source_errors) instead of ok: false.
  • code_source_errors[].message wording is hedged. No field or type changes.

Tests

Regression tests: sandbox::tests::a_flagged_item_read_is_a_hint_and_does_not_fail_the_dry_run (new: valid rename-destructuring source must dry-run ok: true with the hint) and authoring_report::tests::a_code_source_reading_an_undefined_item_is_reported_but_does_not_fail_the_dry_run (renamed, now asserts !is_failing()).

CARGO_INCREMENTAL=0 cargo test -p tinyflows --features mock --lib -- a_flagged_item_read_is_a_hint a_code_source_reading_an_undefined_item_is_reported

  • Before the fix: FAILED. 0 passed; 2 failed (sandbox: left: Bool(false) right: true).

  • After the fix: ok. 4 passed; 0 failed (with a_clean_dry_run_says_its_code_nodes_never_ran and a_graph_without_code_nodes_gets_no_code_summary).

  • cargo fmt --check (touched files)

  • N/A: cargo clippy --all-targets -- -D warnings: left to CI

  • N/A: cargo clippy --all-targets --all-features -- -D warnings: left to CI

  • N/A: cargo build --all-targets: left to CI

  • N/A: cargo build --all-targets --all-features: left to CI

  • N/A: cargo test: only the targeted tests above ran locally; CI runs the suite

  • N/A: cargo test --all-features: left to CI

Documentation

CHANGELOG [Unreleased] entry added; the code_source module doc now says the scan can flag working code and never fails a dry run.

Summary by CodeRabbit

  • Bug Fixes
    • Possible undefined item references reported by the source scan no longer cause a dry run to fail.
  • New Features
    • Dry-run summaries now show these advisory hints alongside code nodes that were not executed, noting that a script may need to read input instead.
    • Reports describe these findings as possibilities, since some valid binding patterns may be flagged by the scan.

The token scan behind code_source_errors cannot see every binding form
(rename-destructuring, Python with/except as, walrus, *args), so valid
scripts were flagged and the dry run returned ok:false, telling the
builder the script fails every real run. Drop it from is_failing, hedge
the message, and name flagged nodes on a POSSIBLE UNDEFINED line in both
the ok and the failing summary.
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: e6c252eb-0cd5-4390-854f-9c7cd362ee0e
📥 Commits

Reviewing files that changed from the base of the PR and between 99fd391 and 48af4b4.

📒 Files selected for processing: 6
  • CHANGELOG.md
  • crates/tinyflows/src/authoring_report.rs
  • crates/tinyflows/src/authoring_report_tests.rs
  • crates/tinyflows/src/code_source.rs
  • crates/tinyflows/src/sandbox.rs
  • crates/tinyflows/src/sandbox_tests.rs

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Code-source scan findings remain in the report as advisory hints and no longer cause dry runs to fail. When code nodes were not executed, the sandbox summary can also identify nodes with source errors and advise using input instead of possibly undefined item.

Changes: Advisory code-source reporting

Layer / File(s) Summary
Make source-scan findings advisory
crates/tinyflows/src/authoring_report.rs, crates/tinyflows/src/authoring_report_tests.rs, crates/tinyflows/src/code_source.rs, CHANGELOG.md
The report describes scan findings as possible undefined-item hints, and is_failing() no longer treats them as failures. Documentation and tests reflect the advisory behavior.
Include source errors in sandbox summaries
crates/tinyflows/src/sandbox.rs, crates/tinyflows/src/sandbox_tests.rs
When code nodes were not executed, the summary retains its unverified-code message and adds source-error node names and input guidance. Tests verify that a flagged rename-destructuring script passes the dry run.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Suggested reviewers: senamakel

Merge Risk: ⚪ Minimal · up to 48af4

The advisory behavior is implemented consistently; no actionable merge-blocking risk was established.

Pre-merge checks: Passed 5
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: making the sandbox code-source item check advisory.
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 5 files. (1 skipped: 1…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 5 files. (1 skipped: 1 unsupported.)

A rabbit reads the scan’s small clue,
It says “perhaps,” not “certainly true.”
The dry run hops along its way,
While hints point out what scripts should say.
“Use input,” the rabbit sings,
Then bounds away on quiet springs.

Comment @coderabbitai help to get the list of available commands.

@tinysweeper

tinysweeper Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Tiny Sweeper review

Reviewed head: 48af4b46e2db15ef17765e6b0c7bc68172ab361b

Verdict: Ready for maintainer review · Priority: none
Coverage: Reviewed 6 of 6 files.
Pull request purpose (title): fix(sandbox): make t…

Diagnostics and supporting details

Full summary

Pull request purpose (title): fix(sandbox): make the code-source item check advisory. Tiny Sweeper completed its review; deterministic results follow.

Review snapshot

Change surface Files Review signal Count
Production 3 Active findings 0
Tests 2 Noted findings 0
Documentation 1 Resolved findings 0
Configuration 0 Pending checks/questions 0

Completeness: Complete
Test assessment: Test coverage is assessed from changed tests and lane evidence; execution is not claimed without trusted check data.

Features

  • Added — POSSIBLE UNDEFINED item summary line in the dry-run report: `code_summary` now takes the source errors and appends a `POSSIBLE UNDEFINED \`item\` in code node(s) (...)` line after `CODE NOT EXECUTED`, so an `ok: true` result surfaces the hint in its summary rather than burying it in a JSON field. The sentence about it was removed from the failing-branch message. (crates/tinyflows/src/sandbox.rs pub fn report(, crates/tinyflows/src/sandbox.rs pub fn report()

Tests

  • renamed unit test — `a_code_source_reading_an_undefined_item_is_reported_but_does_not_fail_the_dry_run` (renamed from `a_code_source_reading_an_undefined_item_fails_the_dry_run`) builds a report from a graph whose `code` source reads `item` without binding it, asserts the single `code_source_errors` entry with its `ReferenceError`/`NameError` message, and now asserts `!report.is_failing()` instead of `is_failing()`.: The tests lane (answered, 0 findings) reports this pins the advisory behavior: the entry is still produced while the report no longer fails. (crates/tinyflows/src/authoring_report_tests.rs fn a_code_source_reading_an_undefined_item_fails_the_dry_run() {)

Findings

No active actionable findings.

Before merge

None.

Agent review details

critique

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: Reviewed 6 files; 0 findings. _The code index is behind this pull request (indexed at `104f6da9a367`), so retrieved context may be out of date._

security

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The security lane answered with 0 findings across 5 security-reviewed files, noting `CHANGELOG.md` was not security-reviewed as prose/tabular data.
  • Lane summary: Reviewed 5 files; 0 findings. 1 file was not security-reviewed: CHANGELOG.md (prose or tabular data). _The code index is behind this pull request (indexed at `104f6da9a367`), so retrieved context may be out of date._

tests

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Lane summary: The change flips `code_source_errors` from a hard dry-run failure to an advisory hint, and both sides are pinned: `authoring_report_tests::a_code_source_reading_an_undefined_item_is_reported_but_does_not_fail_the_dry_run` asserts `!report.is_failing()` while still asserting the entry is reported, and `sandbox_tests::a_flagged_item_read_is_a_hint_and_does_not_fail_the_dry_run` asserts `ok == true` with the new `POSSIBLE UNDEFINED` line present in `note`. The new `source_errors` branch in `code_summary` is exercised by that sandbox test (non-empty and matching exact text), and the empty case by `a_graph_without_code_nodes_gets_no_code_summary`. One residual gap worth noting but not a defect: the `is_failing` branch in `sandbox.rs` still serializes `code_source_errors` but its message no longer explains that field — nothing in the diff claims otherwise, and the advisory text lives in `note`/`code_summary`, which the passing tests pin. _The code index is behind this pull request (indexed at `104f6da9a367`), so retrieved context may be out of date._

commits

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The commits lane answered with 0 findings, reporting no sensitive content in committed files.
  • Lane summary: No sensitive content in committed files. _The code index is behind this pull request (indexed at `104f6da9a367`), so retrieved context may be out of date._

description

  • Conclusion: Success
  • Scope reviewed: all assigned evidence
  • Positive: The description lane answered with 0 findings, reporting the diff is internally consistent with the described behavior and the new/renamed tests match it: the `note` key exists only in the `ok: true` branch while `code_summary` appends the hint only when the graph has `code` nodes.
  • Lane summary: The change makes the `code_source_errors` token scan advisory: `is_failing` no longer counts it, the message wording is hedged, and flagged nodes are surfaced on a `POSSIBLE UNDEFINED \`item\`` line in `code_summary` for both the `ok` and failing branches. The diff is internally consistent — the `note` key only exists in the `ok: true` branch but `code_summary` only appends the hint when the graph has `code` nodes, which also put the hint at the head of the failing branch's `message` — and the new/renamed tests match the behaviour described. I found no defect worth reporting. _The code index is behind this pull request (indexed at `104f6da9a367`), so retrieved context may be out of date._
Evidence and run details
  • Models: ladder/vectors-oai3, deepseek-v4.1-flash
  • Spend: $0.021827
  • Tokens: 607479 input · 9961 output · 220160 cached · 478 embedding
  • Updated: 2026-10-22T21:09:14Z
Head State Pass summary
48af4b46e2db ready for maintainer review 0 active finding(s), 0 resolved finding(s) (at 2026-10-22T21:09:14Z)

tinysweeper 0.1.2

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

Diagnostics

Total spend: $0.0218

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Oct 22, 2026
@M3gA-Mind
M3gA-Mind merged commit 62469d3 into tinyhumansai:main Oct 22, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant