[pull] main from langwatch:main - #279
Open
pull[bot] wants to merge 3047 commits into
Open
Conversation
rogeriochaves
force-pushed
the
main
branch
5 times, most recently
from
January 21, 2026 01:15
1e7b14c to
2209258
Compare
… a Lite Member seat (#6608) * fix(members): a personal workspace no longer blocks moving someone to a Lite Member seat An admin reducing an organization to fit its subscription could not move members to a Lite Member seat. The downgrade cascade that drops a member to Viewer everywhere swept in the personal workspace provisioned for them, whose only admin is its owner, tripped the last-admin guard, and rolled the whole transaction back, so the organization role never changed either. Trying the other route, setting the member to Viewer on that workspace, hit the invariant refusal instead. Reported by a customer with two members in that state. A personal workspace is not one of the organization's teams, and the member surfaces now treat it that way. - `findSharedTeamIds` is the set a seat decision applies to, so the cascade reaches the teams the organization shares and leaves the workspace that is only theirs alone. Lives beside the refusal it is the counterpart to, and both the router and the repository resolve through it. - The role-binding lists an admin manages access from drop personal scopes. They used to render an unremovable `ADMIN on <name>'s Workspace` row for every member of the organization, which is what invited the attempt. - The refusal itself is a handled error with a stable code, customer copy and the workspace's name in `meta`. It consolidates three shapes the same refusal took, one per entry point, including a `ValidationError` whose code and 422 both misreported it and a plain `Error` that reached the groups REST boundary as a 500. Nothing about the owner's admin binding changes, and nothing needs to: a member's organization role already caps what any of their non-custom bindings can do, so a Lite Member reads their own workspace and writes nothing, and getting their full access back restores writes with no repair. That cap had no test; it has one now, because this design leans on it. Two things the same journey needed: - The member list shows full and Lite seats in use against the plan, reusing the usage page's row. Both decisions it offers are refused once the matching allowance runs out, and the allowance was only readable off the refusal, after picking the person and clicking save. - Those refusals name which seats ran out and offer disabling a membership as the reversible way to free one. An admin who hits them is working down to their plan, so "upgrade" alone is the answer they came here to avoid. And two on how the workspaces got there: - An invited member of an organization with no shared project was shown "let's kick off by creating your organization", because onboardedness was read off `primaryIntent`, which is null for every organization predating ADR-038. Belonging to an organization is the test now. - Removing a membership left the personal team and project behind, owned by a non-member and still holding their one slot, while the archive refusal told admins these workspaces disappear with the member's access. They now do, and provisioning revives its own archived workspace so inviting the person back returns the same one rather than failing on a slot nothing can see. * test(members): route the lifecycle teardown's project delete through cleanupTestRows The raw deleteMany filtered on personalTeamId, a let assigned in beforeAll, so a setup that threw before the assignment left Prisma with an undefined filter value, which it drops rather than matching nothing: every project in the shared database. cleanupTestRows refuses an unidentified filter loudly instead, and the entry runs first so projects still go before their team. * fix(members): refresh the seat counts after a seat change, and address review Seat usage stayed on screen without being invalidated, so removing, disabling or reclassifying a member left the card showing the counts from before the action, on the one page where those numbers are the reason an admin is there. All three surfaces that change a seat now invalidate it. Also from review: - the archive on member removal narrows to isPersonal projects, matching what the reactivation restores, so the pair moves the same rows - the personal-scope helpers take named parameters, per the house rule - the scope resolution splits into a fetch and a pure fold, which keeps both under the cognitive-complexity gate - the seat teardown restores the org role and the shared-team binding by writing the rows, not by replaying the router after the app was reset, where it wrote nothing and the failure was swallowed
* perf(dev): queue typecheck runs behind a machine-wide slot A tsgo run on this codebase peaks around 3 to 4 GiB and saturates every core. One is fine. The three or four that a laptop driving several worktrees and agents produces are what make the machine unusable, and nothing about `pnpm typecheck` knew that another one was already running. The typecheck scripts now go through dev/scripts/typecheck-queue.mjs. It takes a slot from a counter shared by every worktree, terminal and agent on the machine, runs the real command, and releases. Occupancy is a directory of one small JSON file per run (pid, arrival, label, state), so a killed run frees its slot with no bookkeeping, and waiters are served in arrival order. With a slot free it prints nothing and is otherwise transparent: stdio inherited, exit code passed through, signals forwarded. Queued, it says what it is waiting behind and how long it waited, which is what tells an agent that the extra minutes were queueing rather than a hung typechecker. Nothing can block forever: a wait past 30 minutes runs anyway with a warning. TYPECHECK_SLOTS sets the limit and 0 turns the queue off. Unset, the limit comes from the machine (one per 6 GiB of RAM, capped at one per 4 cores) and CI does not queue at all, where one job runs one typecheck and a gate could only add risk. `--explain` prints the limit and who holds what. `haven typecheck` already held a RAM slot of its own, so it now passes TYPECHECK_SLOTS=0 to the run it spawns: counting a run twice would queue it behind itself, and the reaper's duration ceiling would be spent waiting rather than typechecking. * perf(dev): put lint under the same slot as typecheck, as CHECK_SLOTS biome saturates the machine the same way tsgo does, and for the same reason: 6,784 files, 38 CPU-seconds compressed into 4 seconds of wall clock across every core. That is the right trade for one run. Capping the tool's own threads is not the fix, RAYON_NUM_THREADS=2 does work on biome but spends the same CPU over 25 seconds instead of 5. Not running four of them at once is the fix. So the queue now covers lint and format too, against ONE counter: they compete for the same cores, and a lint that starts while a typecheck runs is exactly the pile-up this exists to stop. Renamed to match what it now governs: TYPECHECK_SLOTS is CHECK_SLOTS, TYPECHECK_QUEUE_* is CHECK_QUEUE_*, and the script is dev/scripts/check-queue.mjs. Wired: typecheck, typecheck:tests, typecheck:legacy, lint, lint:fix, lint:plugins, format. The first blocked message now names the holders rather than saving them for the heartbeat, so a waiter knows which worktree to go look at immediately: checks: 1 check is already active on this machine (limit 1, set CHECK_SLOTS to change). Queued at position 1, waiting for a free slot. Active: @langwatch/web typecheck (spicy-puzzling-rain) for 2s checks: slot free after 17s in the queue, starting now. * fix(dev): the check queue must never be the reason a check fails Five findings from review, all real. The queue promised in its own docblock that it could never block a check, and only honored that for the wait timeout. Any throw escaped: a read-only, full or foreign-owned /tmp made fs.mkdirSync reject out of main, so the typecheck exited nonzero without ever invoking tsgo. Proven against the previous commit, which fails with an unhandled rejection and never runs the command. Queueing now sits inside an error boundary that degrades to an unqueued run; the command itself stays outside it, so it still runs exactly once and its own failures are its own. readEntries accepted any JSON object with a live pid, and byArrival then read .token off it. Two entries sharing an arrival millisecond force that tie-break, so a worktree on a branch with a different entry shape crashed this one with a TypeError inside Array.sort. Also proven against the previous commit. Entries are validated for the fields the queue reads and dropped otherwise, which is the normal case for a directory that is machine-wide on purpose. The queue directory is created 0o700. The uid in its name made the path unique, not private, and the labels name worktrees and branches. In the test harness, runs resolve on `close` rather than `exit`: exit fires while the stdout and stderr pipes may still hold buffered data, and these tests compare that output exactly, so it would have flaked as a queue bug rather than a harness one. And waitForHolder's doc comment said it watched the queue directory when it reads the event log. Both degradations are now bound by scenarios that fail against the old code rather than asserting on strings.
…utdown timing (#5977) * fix(aigateway): parse duration env vars correctly, wire up graceful-shutdown timing Two gaps found while reviewing the #4806 heartbeat fix (PR #5963): 1. pkg/config.Hydrate parsed every time.Duration field as a raw nanosecond integer via strconv.ParseInt instead of time.ParseDuration, because time.Duration is a defined int64 and fell into the generic integer branch. AuthCacheConfig.SoftBump/ HardGrace/ConfigTTL are the fields this affects today — .env.example documented SOFT_BUMP=5m/HARD_GRACE=6h, a format that could never have actually parsed. Fixed by special-casing the exact time.Duration type ahead of the generic int64 branch. 2. charts/gateway/values.yaml documented shutdown.preDrainWait/timeout with a whole invariant-with-terminationGracePeriodSeconds comment block, but neither value ever reached the container — configmap.yaml never set an env var for either, and serve.go never called lifecycle.WithDrainDelay at all (true of all three pkg/lifecycle consumers: aigateway, nlpgo, langyagent). The chart values were fiction. Added DrainDelaySeconds to the shared pkg/config.Server (paralleling GracefulSeconds), wired it in aigateway's serve.go only — nlpgo/langyagent don't advertise this capability in their own charts, and langyagent has its own ADR-048 shutdown-budget hard-fail validation that's out of scope to touch here. Renamed the chart values to plain-integer seconds (preDrainWaitSeconds/timeoutSeconds) to avoid Helm-template string parsing, wired both through configmap.yaml, and added a CI assertion (matching the existing NetworkPolicy-assertion pattern in go-services.yaml) that greps the rendered template for the correct values. Also added a startup WARN when GracefulSeconds is shorter than the effective non-streaming heartbeat interval — not compared against the gateway's 14-minute upstream ceiling, since no sane GracefulSeconds ever approaches that (permanent noise, nothing actionable); the heartbeat interval is the deliberately-chosen boundary between "fast, typical" and "slow but legitimate," and isn't universally true today — the stock defaults (10s graceful, 45s heartbeat) already fail it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aigateway): convert AuthCacheConfig to plain int64 seconds Per @0xdeafcafe's review (pkg/config/config.go#L64): a defined-int64 time.Duration field parsed from a string reads as non-standard next to this same PR's plain-int Server.GracefulSeconds/DrainDelaySeconds. Renaming SoftBump/HardGrace/ConfigTTL to SoftBumpSeconds/HardGraceSeconds/ ConfigTTLSeconds (env LW_GATEWAY_AUTH_CACHE_*_SECONDS) aligns both halves of this PR on one convention. pkg/config.Hydrate's time.Duration fix stays — it's now preventative for the next Duration-tagged field added anywhere the shared hydrator is used, rather than fixing a currently-live caller. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(aigateway): close the goated-review findings on the shutdown-timing chart wiring - Add pkg/config/server.go and services/aigateway/config.go to the `chart` filter so the helm job's regression guards can't silently stop running on a Go-only rename of the env vars they check. - Add a CI assertion enforcing terminationGracePeriodSeconds actually covers preDrainWaitSeconds + timeoutSeconds + slack, matching the invariant the values.yaml comment already claimed (and correcting that comment's stated slack from 5s to the true 10s). - Update docs/ai-gateway/self-hosting/helm.mdx and health-checks.mdx off the renamed shutdown.preDrainWait/timeout keys — both self-hosting docs still referenced the pre-rename names, one with a worked example that would now silently no-op. - Update gateway-service.feature's pre-existing SIGTERM-drain scenarios off the same renamed keys, and add the Bindings note they were missing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(config): refuse env-tagged time.Duration fields outright Every service sharing pkg/config.Hydrate expresses an env-configurable time span as an int64 count of seconds on a _SECONDS-suffixed variable. A time.Duration field is a defined int64, so Hydrate routes it by reflect.Kind() into the generic integer branch, where the only value that sets it is a raw nanosecond count: one config surface, two incompatible notations, and which one applies depends on a field's Go type rather than on anything an operator can see. With the auth-cache knobs now on plain seconds, the repo has no time.Duration env field left. Hydrate refuses the declaration, not just a bad value, so the mistake surfaces on the first boot of the service that introduced the field instead of on the first deployment that tries to configure it. * fix(aigateway): keep the stock graceful window above the heartbeat interval warnIfGracefulShutdownTooShort fired on every stock deployment, because the Go default (10s) and the chart's timeoutSeconds (15s) both sat below the 45s non-streaming heartbeat interval. A warning every install emits is noise: operators learn to scroll past it, and the one deployment that really has narrowed its drain window looks identical to the rest. The default graceful window is now 60s and the chart's timeoutSeconds matches, with terminationGracePeriodSeconds raised to 75 to keep the preDrainWait + timeout + slack invariant the CI job asserts. The warning now marks a deployment that narrowed SERVER_GRACEFUL_SECONDS on purpose. Also here: - LoadConfig range-checks every seconds-valued field. Each becomes a time.Duration by multiplying by a billion, so a nanosecond count pasted into a seconds field wrapped int64 into a negative duration and read back as "disabled" rather than as a startup failure. - The chart refuses a values file carrying shutdown.preDrainWait or shutdown.timeout. Helm merges unknown keys in silently, so an operator who kept the duration-string names would install a release whose drain timing quietly ignored them. - A negative auth-cache hard grace is the documented way to disable stale-while-error. Zero was described as the opt-out but takes the 6h default, matching every other knob on that struct. * docs(aigateway): finish the auth-cache seconds rename and the drain-timing numbers The auth-cache env vars carry a _SECONDS suffix and an integer value, but the self-hosting config page, the Helm and troubleshooting pages, and the production runbook still named the duration-string variables and told operators to set values like 6h that no longer parse. Also corrected here: - HARD_GRACE_SECONDS=0 was documented as the way to disable stale-while-error. It selects the 6h default instead, so every page that offered it as the regulated-deployment opt-out was handing out a setting that does the opposite. A negative value is the opt-out. - .env.example gained the CONFIG_TTL_SECONDS line, which the other two auth-cache knobs had and it did not. - The chart's drain numbers follow the 60s graceful window, and the Helm page's four-phase drain named /livez where the chart probes /healthz. - The health-checks page told operators not to reach for a manual preStop sleep in one section and prescribed preStop: sleep 3 in its troubleshooting table two sections later. llms-full.txt regenerated with docs/llms.txt.cjs, which is what docs-ci diff-gates it against. * fix(aigateway): reject negative waits and retired duration env vars, enforce the chart's drain budget Review follow-ups on #5977. A negative SERVER_GRACEFUL_SECONDS or SERVER_DRAIN_DELAY_SECONDS was accepted and became an already-expired context deadline, so SIGTERM dropped every in-flight request at once instead of draining. Neither field reads a negative as "disabled", so both are now refused at startup with a message saying zero is the way to ask for no wait. The retired duration-string variables (LW_GATEWAY_AUTH_CACHE_SOFT_BUMP, _HARD_GRACE, _CONFIG_TTL) were simply unread, so a deployment still carrying one booted on the default. An operator who had set the hard grace to hard-fail at JWT exp would have come back up serving stale bundles for six hours with no signal. Startup now stops and names the replacement, the same way the chart already refuses the retired shutdown.preDrainWait / shutdown.timeout keys. terminationGracePeriodSeconds stayed at 75 no matter how wide the drain was configured, so shutdown.timeoutSeconds=90 rendered a pod the kubelet kills 30s into its own timeout. The chart now refuses that render and names the grace period the drain would need. The chart assertions move out of inline workflow YAML into charts/gateway/tests/shutdown-values.sh so their cases can carry @Scenario bindings; the feature-parity checker gains that directory as a shell-test root, and the two specs it binds are tagged. Docs: the runbook said a hard-cap eviction shows customers 401s, but the re-resolve hits the same unreachable control plane and answers 503 auth_upstream_unavailable; 401 is the separate bad-credential case. The auth-cache variables are spelled in full everywhere an operator might copy one, and the gateway's real 60s graceful default replaces the 5s the config reference claimed. Claude-Session: https://claude.ai/code/session_018dSRiwg1XGWVFNitNjkjZ4 * Settle the inbound body cap at 32 MiB in all three places The cap was stated three times and agreed nowhere: 128 MiB in the Go default, 10 MiB in the Helm value the chart renders, 32 MiB in the self-hosting docs a pod gets sized from. An operator provisioning from the docs got a cap the gateway did not enforce, and the only symptom is a 413 on a payload the docs promised would fit. The errors page also named GATEWAY_MAX_REQUEST_BODY_BYTES, which nothing reads. All three now say 32 MiB, which fits a 1M-context multimodal payload under a 512 Mi pod limit, and a test reads the chart and the docs rather than restating the number, so moving one alone fails the build. Claude-Session: https://claude.ai/code/session_018dSRiwg1XGWVFNitNjkjZ4 --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Rogério Chaves <rogeriochaves@users.noreply.github.com>
…undContent rely on (#6349) (#6364) * test(traces): pin the tab state preservation TraceDetails and PlaygroundContent rely on (#6349) tabs reach editable state a remount would discard. That reasoning was arrived at by reading the code and never confirmed by a test, so nothing failed if a later performance pass added `unmountOnExit` and silently threw a user's half-written work away. Each component gets a test that types into the draft, switches tab, switches back, and asserts the draft survived, plus a direct assertion that the panel is the same DOM node rather than a fresh one so a failure names which half broke. Both also assert lazyMount's own half: an unvisited panel is not mounted, and is once opened. The PlaygroundContent comment was wrong about what is at risk, and the test made that visible. Every field in SpanEditorPanel and the LLM/RAG/Prompt editors is controlled straight through to the traceStore on each keystroke, so all of them survive a remount. The only state that lives in React is AttributeEditor's `newKey`, an attribute name typed but not yet committed with Add. The comment now says that, and the test protects that draft specifically. For TraceDetails the lazyMount assertion is on the User Events panel, not Trace Details or Sequence: those two carry an additional inner `selectedTab === ...` gate, so they stay empty whether or not the prop is there and would not notice it being removed. Evidence the tests discriminate, per component, by mutating the source and re-running: - add `unmountOnExit`: the two draft/identity tests fail, the lazyMount ones pass - remove `lazyMount`: the unvisited-panel test fails, the draft ones pass The 11 pre-existing TraceDetails suites the ticket asked to run go from 56 to 63 tests, all green. Closes #6349 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(traces): action-based names and a positive lazyMount assertion (#6349) Review follow-up on the two tab-state tests. The `it()` titles now name the action and its result rather than the state after it, matching the repo convention. The lazy-mount case gains its other half. Asserting only that the Graph panel is absent before its tab is opened passes just as happily for a Graph view that never renders at all, so the precondition block now also opens the tab and asserts the panel appears. TraceDetails already had that pair; its blocks are restructured so the precondition reads as `given` and the opening reads as `when`, which is what the two are. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(traces): nest the Editor scenarios under their given precondition Both `when` suites depend on a span already being selected, which was carried by the helper name rather than by the structure. The Graph suite next to them already stated its given. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Ubuntu <ubuntu@ip-10-0-3-183.eu-central-1.compute.internal> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ruction (#6625) * fix(metrics): bound the rollup successor read's request size by construction Prod ClickHouse logged 20 SYNTAX_ERROR (code 62) exceptions in one minute on 2026-08-03. The rejected text was the rollup successor seek arriving as the request body in URL-encoded form - `query=...¶m_tenantId=...` - so the server parsed the encoding as SQL and failed at the first `&`. It stopped after #6493 reshaped the method, but nothing about that change bounded the request: the current shape builds larger ones than the one that failed, four parameters and a whole SELECT branch per point, which at SEEKS_PER_QUERY is 256 parameters and tens of kilobytes of SQL. The successor read now sends a fixed statement and nine parameters whatever the chunk holds. Every per-point value travels as an array parameter, and SEEK_SELECT is emitted once in a CTE the branches read from rather than once per branch. Two branches keep the reads bounded rather than trading a small request for a wide scan. Every chunk point is already stored when this runs, so within a series no chunk point's successor can be further away than the next chunk point; only the newest needs an open-ended look forward. So one branch reads the chunk's own span and the other is a single LIMIT 1 BY SeriesId seek past the end of it. `affectedBucketsBySeries` takes the minimum of the union, so the successors resolved are identical to the ones a seek per point returned. The read stays payload-free, as #6493 made it. The affected-bucket reads keep a seek per bucket on purpose. Their predecessor branch is bounded below by the retention window, and a joined form can only apply that bound after the rows are read, turning a single-row reverse index seek into a read of the series across the whole window - the memory class #6493 fixed. What they shed is the fan-out for bounds the server can derive itself: the bucket end and the retention floor now travel once as shared scalars, so a bucket costs two parameters rather than four. Tests pin the request as invariant rather than merely smaller: byte-identical SQL and the same parameter names across chunk sizes and series counts, the select list emitted exactly once, and the resolved buckets equal to those a seek-per-point read produces for a single seek, a full chunk and a chunk boundary. The integration suite gains a series whose chunk points have stored points between them, which is the case the span branch exists for. * perf(metrics): prune partitions on both halves of the folded successor read The folded read's only time bound was the shared lower one in the CTE, so the branch that reads within the chunk's spans scanned to the end of the table before the join discarded the rows past them, and the branch that reads past the spans scanned from the earliest span start. `metric_data_points` partitions by `toYearWeek(TimeUnixMs)`, and a bound that only reaches the column through the join cannot prune. Each branch now carries the one constant bound its own half can prove. A row within a span sits below its series' far end, so it sits below `spanEnd`, the furthest of them; a row past the far end sits above `spanStart`, the nearest. Both are pure upside - pushed down they prune, left in place they are a cheap filter over rows the branch was going to discard anyway. Range-filtering TimeUnixMs under FINAL is safe here because it is part of the dedup key, so no version of a row can drift out of the window. Two more parameters, both constant, so the request stays fixed-size and the invariance guards still hold. The equivalence tests now apply the same bounds in their model of the statement, so a bound that wrongly excluded a row would fail as a missing successor rather than pass unnoticed. Also records why the `LIMIT 1 BY` here is the narrow kind the ClickHouse query guide permits rather than the kind it bans - six sequence columns, nothing heavy - so a later reader does not "fix" it into the IN-tuple form. * fix(metrics): bound the successor read by encoded parameter bytes The successor read was bounded in parameter count, not in size. Eleven parameter names are fixed, but the arrays they carry grow with the series in the chunk, and those ride the URL as `param_*` entries: measured on the pinned client's own serialisation, one series encodes to ~530 characters and a full 64-series chunk to ~19,900 - larger than the seek-per-point shape it replaced, and far past the client's 4096-character `MAX_URL_BIND_PARAM_LENGTH` (inert here, since nothing sets `use_multipart_params_auto`). Chunking now splits on the encoded parameter bytes, against an explicit budget with headroom under the client's ceiling, keeping the seek cap as an upper bound on series. Three unit tests pin the quantity that actually travels, measured with the client's own `formatQueryParams`. Also in this change: - the equivalence fixtures were vacuous. Their points were gauges, and a gauge successor never pulls a second bucket into the affected set, so every assertion in that block held with the successor read deleted. They are cumulative now, and two fixtures were added that discriminate: an interior stored point in a bucket no chunk point occupies, and two series whose spans sit hours apart. All four mutation probes - dropping the within-span branch, and flipping each of the three constant bounds min<->max - now fail a test. - the integration tier folds a two-series chunk for the first time, each series staggered and holding stored successors past its own span, and its point-at-a-time reference series are now seeded entirely through the per-point entry point rather than half-built by a chunk. - `spanStart` / `spanEnd` renamed to `earliestSpanEnd` / `latestSpanEnd`; both were derived from span ends, and `spanStart` next to `spanEnd` read as "earliest span start", which is `scanFrom`. - doc corrections: eleven parameters not nine, ~2.4 KB of SQL not ~1 KB, the `series_points` CTE executes twice rather than once, what `SEEKS_PER_QUERY` now bounds, what `orderedAfter` actually qualifies, and the affected-bucket divisor's real justification. - the per-series scan bound both branches lack is documented at the site rather than fixed here. - test mocks discriminate on a bound parameter instead of a CTE name. - spec: the Rule promised a fixed-size request for a read that is deliberately batch-proportional; narrowed to a bounded one, with the encoded-budget property as its own scenario. Two scenarios retitled around the invariant they actually assert, and the metric spec's "Storage trouble" scenario disambiguated from the log spec's identical title, which the parity checker indexes by title alone. - specs/README described a `WATCHED` list that no longer exists. * test(metrics): assert the split asks about each series once, prefix the boolean Two review points on the byte-budget commit: the split assertion counted series rather than identifying them, so a chunker that asked twice about one and never about another would have passed it; and formatParamValue's new boolean parameter needed the repo's is/has/should prefix. * test(metrics): pin the split against the series the chunk carried Comparing the asked-about series to the chunk's own set, not just to a count of 64: a chunker that asked about a series nobody sent would have passed the cardinality check.
…t was refused (#6630) * fix(members): let a seat change take a team's last admin, and say what was refused An admin moving somebody to a Lite Member seat was refused whenever that person was the only admin of a shared team. The correction to Viewer tripped the last-admin guard, which threw inside the transaction carrying the organization role change, so nothing was saved: the seat stayed as it was, and clearing the member's team access first could not help either, because the seat change is applied before it and always read the roles as they still were. The guard is there so a team is never left with nobody who can administer it, and that is not the state this produced. An ORGANIZATION-scoped ADMIN binding grants team permissions in every shared team, so the organization's admins still administer a team whose last team-scoped admin is gone. A seat decision is the organization's to make, so it goes through, and the save reports the teams it changed so the admin who made it is not left to find out. Editing one team's own members is a team-local decision and keeps the guard. The refusals themselves were unreadable. Each was a bare ValidationError: a real sentence in the message, nothing in meta. The wire message is the code slug now, so the registry had only validation_error to key off, which reads its copy out of meta and fell through to "Some of the values aren't valid" for a failure the server could name exactly. Three codes now carry it, each naming the team and the one step that clears it: - team_last_admin_required - cannot_remove_self_as_last_admin - lite_member_viewer_only * test(members): guard the seat-change reset, split the suite, and match the new success shape Three CI failures, all from this branch: - The per-test reset used a raw deleteMany filtered on ids assigned in beforeAll, so a setup that threw would have matched every role binding in the shared database. It goes through cleanupTestRows now, which refuses an unidentified filter instead. The teardownScan gate caught it. - The dialog's mocked updateMemberRole resolved undefined while the real one now answers with the teams left without a team admin, so the save read a field off nothing and never reached the bindings mutation. The mock answers what the mutation answers. The component defaults the field too: mid-rollout this code can reach a server that omits it, and by then the save has already succeeded, so reading it eagerly would report a successful save as failed. - The plan-limit suite asserted the success shape exactly, which the new field changes. The seat-change suite also passed 300 source lines, so it splits along the line the behaviour already had: what a seat correction may do, and what a team-local decision still may not, over one shared fixture. * test(members): cover the omitted-response field, and stop the dialog suite passing through its own errors The suite mocked `api.useContext()` without `roleBinding.listForOrg` or `limits.getUsage`, both of which the save invalidates, so every save threw after its mutations and landed in the error toast. Nothing noticed, because no test asserted the toast: each one checked a mutation call that happens earlier. The new case asserts the success toast, which is what surfaced it. Both are mocked now, so the saves these tests describe are saves that finish. Also from review: - a case for the response arriving without the affected-teams field, which is the rollout window the component's default exists for and nothing executed - the per-test default for updateMemberRole answers the shape the mutation answers, in beforeEach where the suite already sets its defaults, rather than in the hoisted block where beforeEach overrode it anyway - the fixture returns each created team's row, so the name a refusal is asserted against has one source instead of a restated template - the fixture reinstalls the app per test: it is a process singleton, and a neighbouring suite's teardown landing between two of these tests would leave the null repository in place, holding every assertion on a mutation that wrote nothing * test(members): compose the seat-change fixture from named helpers The fixture was one 220-line factory that seeded rows, installed the app, and closed over five read and reset helpers, so reading it meant holding three abstraction levels at once. Each responsibility is now its own module-level function and the factory composes them. Drops the fixture's `bindTeamRole`, which no suite called. The dialog's omitted-field case moves its precondition into a `given` block and its save into a `when`, matching the rest of the seat-change suites.
…erdict stops posing as a passing zero (#6632) * fix(traces-v2): the Events column reads real events, and a category verdict stops posing as a passing zero Three defects on the trace list and its drawer. The Events column was hardcoded empty. #4205 moved event derivation off the trace-summary fold (the per-span hoist grew the fold state O(span-count)) and onto a read-time `stored_spans` query, but only wired that read into the drawer. `mapToTraceListItem` was left returning `events: []` under a comment claiming the list never surfaced events, which the line it replaced disproves. The column ships in the default All and Errors lenses, so every row read "—", and the Conversations lens event counter was dead the same way, silently, behind a `> 0` guard. Events now come from `tracesV2.listEvents`, one batched read per visible page, grouped by name in ClickHouse. Its own query rather than part of `list`: events live in a different table from the summary, and a page whose columns and grouping never mention events pays nothing. A row shows one badge per name with a repeat count, because an agent turn that retries a tool 237 times has 237 `tool.output` events and one thing worth saying about them. A categorising evaluator (`langevals/llm_category`) answers with a label and neither a score nor a pass/fail. The UI substituted a `pass` status and a score of `0` for the fields it never filled, so its card claimed a run that passed scoring zero and buried the category under Show details. The category now leads the card, and the same fix lands on the header chip and the list chip through `getEvalChipDisplay`, which is where all three already agreed on everything else. An evaluator that produced a real verdict keeps its badge and score and gains its label beside them. "View definition" opened the evaluator drawer and lost it a moment later. The eval hover card was left open behind the drawer it had just opened, so the pointer leaving dismissed the card, the drawer read that as an interaction outside itself, and closed — taking the URL back with it. The card now closes when its own action navigates. * test(traces-v2): bind the Events column fetch scenarios to a real hook test The four unbound scenarios in trace-list-events-column.feature all describe when the events read fires and what a row shows before it answers, which is what `useTraceListEvents` decides. Covers the merge, the pending state, a failed read leaving the list intact, the column hidden, an empty page, the column being switched on, and the Conversations lens needing the read without the column. evaluations.feature enforces scenarios now, so it leaves LEGACY_INERT. * fix(traces-v2): the Events column says when it cannot answer, and a boolean verdict stays a verdict Review follow-ups on the Events column and the category-verdict chip. A failed events read used to fall back to the empty marker, which reports a trace that has events as having none. Rows now carry that failure and the column reads as unavailable. The same applies mid-page-turn: React Query hands back the previous page's rollups with the load already finished, and none of them are keyed by a trace on the new page, so those rows stay pending until their own answer arrives. Onboarding sample traces ship with their own events and have ids that exist nowhere else, so the preview no longer looks anything up for them. A boolean score is a verdict whatever it is labelled, so it no longer reads as a category and keeps its Pass/Fail. A category-only card also keeps its label when the label reads the same as the placeholder score it displaced ("0"), which was leaving such a card with no verdict at all. Also splits the rollup read into its query and its mapping, gives the new multi-argument helpers named parameters, nests the new suites under given/when, and rewrites the events spec in terms of what the user sees. * test(traces-v2): give the events hook test's mock helper a named parameter * test(traces-v2): assert the exact rollup on a trace id two tenants share * chore: drop unrelated generated-file churn from this branch * style(traces-v2): format the tenant-isolation assertion and document the Events cell states * docs(traces-v2): say what the events test fixtures stand for
…f all spans (#6640) * fix(observability): stop tracing every Redis command, which was 93% of all spans BullMQ runs every queue operation through Redis, so instrumenting ioredis in a process that owns a job queue traces the queue's own bookkeeping rather than the work. Measured in production: langwatch-service-worker emitted 317 job spans/sec and roughly 10,116 Redis command spans/sec, about 32 Redis spans for every one span describing the job. `evalsha` alone ran at 1,771/sec, which is BullMQ executing its own Lua scripts. Across the platform that was 11,177 spans/sec, 966 million spans/day and 370 GiB/day into Tempo, of which the worker was 93%. None of it answers a question: it records that a queue was being a queue. ioredis instrumentation is now opt-in behind OTEL_TRACE_REDIS_COMMANDS, and its require moves inside the factory so the package is not loaded when it is off, matching how the rest of this file gates its dependencies. The configuration is unchanged when enabled, including requireParentSpan and the db.statement truncation. This is the app-side half of the Tempo cost work. The infrastructure half cuts trace retention from 90 days to 3 and stops replicating traces cross-region. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z * refactor(observability): extract the Redis tracing policy so it can be tested Addresses review on the ioredis opt-in. The decision the flag encodes is worth protecting: a regression that registers the instrumentation unconditionally is silent and only shows up on a bill. instrumentation.node loads its dependencies through gated `require` calls, and vi.mock does not intercept `require`, so testing the flag in place would have meant building a boot harness around the whole OTel SDK. The two things worth pinning are pure instead: whether tracing is on, and what a Redis span records. Those move to instrumentation.redis with 17 tests covering the default, the exact-match parsing, and the serializer's key-only truncation. Also from review: the boolean now reads as a state (redisCommandTracingEnabled, matching langwatchTracingEnabled beside it), the .env.example wording says spans need an active parent span because requireParentSpan is set, and the procedural comment restating the instrumentations array is gone. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z * fix(observability): bound the Redis db.statement, freeze the instrumentation config Second review pass on the ioredis opt-in. Dropping the command's values was not enough on its own. The first key is caller-controlled and can be arbitrarily long, so an unbounded key reintroduced exactly the large span attribute this serializer exists to prevent. The statement is now capped at 256 characters including a visible "..." marker, because a silently shortened key reads as a real key and sends whoever is debugging after a span that never existed. 256 is far beyond any key this codebase builds, so truncation should be a symptom rather than routine. redisInstrumentationConfig is also frozen with a const assertion, since it is module-level, exported, and shared. Four boundary tests cover the cap: over, exactly at, one under, and the marker. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z * test(observability): name the cap preconditions with given/when Follows the house BDD convention: 'given' for a precondition, 'when' for the action. The oversized-key block was naming a precondition with 'when', and the three cap cases now read as separate preconditions (over, exactly at, one under) rather than one block holding all three. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z
* feat(clickhouse): one client, bounded where it can be seen A connection pool caps sockets, not statements. Work arrives from queues whose concurrency is set somewhere else, so a process will try to run far more statements than it has sockets and the surplus waits inside the pool with no timeout, no metric and no ceiling. That is the shape of the 2026-07-31 overload: the server admitted past its limit, rejected, and the retries went back into the same wall. #6604 wrote the limiter that fixes this and left it unwired - the app used its retry policy, its logging and its pool sizing, and nothing else. This wires it, at the one place a client is now built. - `managedClient.ts` is the single construction site. Default settings, statement limit, resilience, driver - in that order. The limit sits outside retry so a slot is held for the whole statement rather than taken per attempt, which is what stops a small overload becoming a persistent one. - The per-organization private-instance client is built there too. It previously set no pool size at all, so it ran the driver's default of 10 with nothing bounding its statements - the weakest limits in the system on its smallest servers. - `clickhouse_statements_in_flight`, `_queued`, `_shed_total` and `clickhouse_statement_wait_seconds` report what the pool used to hide. A refusal is a typed `ClickHouseOverloadedError`, distinct from "unavailable" because only one of the two is fixed by looking at ClickHouse. - The limit is the pool size, so capacity is unchanged on the day this lands. What changes is where the queueing happens. `DEFAULT_CLIENTS_PER_PROCESS` drops 2 -> 1: the app-layer factory it paid for had no callers, so it was halving every derived ceiling for a pool nobody opened. The factory and its barrel are deleted. A boundary test holds both rules - one construction site, and access only through a repository from `getApp()`. Forty files predate the second rule and are named in a ratcheting backlog that can only shrink. Spec: specs/clickhouse/single-client-access.feature * refactor(filters): read filter options through a repository on the App The first entry off the access backlog, and the shape the rest follow. `FilterService` resolved a ClickHouse client and wrote the query itself, which put storage decisions - which table a filter reads, how it scopes to the tenant, how its rows decode - in a service, where no boundary can enforce them. They move to `FilterOptionsClickHouseRepository`. The service keeps what is its own: validating the caller, tracing the call, and refusing to leak a raw ClickHouse message to the browser. `dataForFilter` now takes the service from `getApp()` instead of constructing one, so the route reaches ClickHouse the only way it may. The boundary test's backlog is one line shorter. It failed until that line was removed, which is the ratchet working: a file that no longer needs an exemption cannot keep one. * refactor(gateway): take the ClickHouse repositories from the App Four more entries off the access backlog, in three shapes. DEAD. `src/server/traces/span-storage.service.ts` resolved a client and built a repository per call. Nothing imported it - every one of the fifteen `span-storage.service` imports resolves to the app-layer service of the same name. Deleted. DUPLICATED. `gateway/clickhouseRepos.ts` existed to be "the single construction point for the gateway's ClickHouse-backed repositories", with a header warning that building them per surface is how REST came to serve stale spend for budgets the UI showed live (#6248). `auth-cli.ts` had its own copy of the same constructor anyway - which is what a convention gets you when nothing enforces it. Both are gone. The budget ledger and the virtual-key spend repository are built once in presets and handed out as `getApp().gateway`, so the tRPC routers, both REST apps and the CLI route now share the instances rather than each minting their own. `auth-cli.ts` no longer imports ClickHouse at all. PASSED IN. `applicableEndUserCaps` built a repository inline from its own resolver; it now takes one. The caller supplies `getApp().gateway.budgets` and answers `ClickHouseUnavailableError` when the deployment has no ClickHouse - the ledger is the only store spend accrues in, so there are no figures to report without it. 18,860 unit tests across src, app and ee pass. * refactor(clickhouse): take traces/event-sourcing/scenarios repositories from the App Seven more entries off the access backlog. SHAPE C (repository extraction): - orgBillableEventsMeter.store.ts: the ClickHouse insert moves to BillableEventsClickHouseRepository, taken from getApp().billing.events. The store keeps org resolution and orphan-project handling. - orphaned-run-reconciliation.clickhouse.ts + scenario.processor.ts: the two boot-time orphan-reconciliation sweeps now take their shared client and ClickHouseOrphanedRunFinder from getApp().scenarios.orphanReconciliation instead of resolving getSharedClickHouseClient() themselves. - pipelineRegistry.ts: the inline experimentId lookup query moves to ExperimentIdLookupClickHouseRepository, injected via PipelineRepositories like every other store the registry consumes — it resolves nothing itself. SHAPE B (duplicated resolver closures): - trace-blob-resolution.deps.ts and replayPreset.ts each rebuilt the same tenant->client resolver presets.ts already builds. Both now read it from a new getApp().clickhouse.{enabled,resolveClient} slot instead. - log-record-storage.service.ts's createDefaultLogRecordStorageService() duplicated the LogRecordStorageService presets.ts already assembles for app.traces.logRecords; TraceService's lazy fallback now takes that instance directly instead of building a second one. New App slots: clickhouse.{enabled,resolveClient}, billing.events, scenarios.orphanReconciliation.{client,finder} — wired in dependencies.ts, app.ts and both presets.ts assemblies. Two integration tests (trace-detail-blob-recall-endpoint, export-summary-blob-resolution) exercised buildTraceBlobResolutionDeps()'s no-arg path without an initialized App; both now seed a real getApp() singleton via createTestApp() so the mocked ClickHouse client is still what the resolver dials. clickhouse-trace.service.ts (18 statements, 3396 lines) is left on the backlog — extracting it is a much larger, riskier change than this batch. 569 unit test files / 11941 tests pass across the touched trees. * refactor(clickhouse): route analytics, evaluations, automations and workers through repositories Six entries off the ClickHouse access backlog, in three shapes. SHAPE B (wiring only). analytics.service.ts resolved a client purely to construct repositories and the legacy backend; construction moves to presets.ts, and the shared AnalyticsService instance is handed out as getApp().analytics.service instead of each of its ~6 callers (four routers, one REST route, one dispatch closure) building its own. startWorkers.ts resolved the shared client to boot storage-stats collection; that now happens inside clickhouse/metrics.ts (already an allowed-to-resolve location) via a new startStorageStatsCollectionFromSharedClient() export. graph-trigger-heartbeat.ts already took an injected resolver everywhere except its own default-deps factory, which now points at a new shared defaultClickHouseClientResolver export on clickhouseClient.ts instead of hand-rolling the same throw-if-unavailable wrapper. SHAPE C (query extraction). clickhouse-analytics.service.ts's three legacy reads (filter options, top documents, feedbacks) move into LegacyAnalyticsBackendClickHouseRepository. evaluation.service.ts's per-trace evaluation reads (including the memory-limit degrade-to-light- projection retry) move into TraceEvaluationsClickHouseRepository, both under app-layer/*/repositories/. collectUsageStats.ts's two org-wide counts move into InstanceUsageStatsClickHouseRepository — org-scoped (TenantId IN across the org's project ids), not per-tenant, so it takes the organization resolver directly. clustering.ts's single client resolution now goes through defaultClickHouseClientResolver. None of these needed new App slots except AnalyticsService: the others are either self-contained (default resolver lives in an already-exempt location) or already dependency-injected, so construction-time code that runs before the App singleton exists (buildAutomationDispatchPorts, TraceService via EvaluationService.create()) never calls getApp(). Left in the backlog: experiment-run.service.ts. Four public methods with three different no-client behaviors (throw / throw / return null), two private helpers sharing an already-resolved client across call sites, and no existing unit test isolating the resolution point — a correct extraction needs more care than this batch's remaining budget allowed. 5164 unit tests across analytics, app-layer, evaluations, workers and experiments-v3 pass (run in scoped batches — vitest's worker pool cannot hold all ~4300 in one invocation without crashing a few workers, unrelated to this change). * test(clickhouse): guard the App-resolver escape hatch, take agent 2's entries off the backlog * refactor(clickhouse): move ee/governance and ee/billing off direct client access Nine of the ten assigned files migrated onto repositories the App hands out; activityMonitor.service.ts skipped (see below). WIRING (Shape B). cliBootstrap.service.ts and pullerWorker.ts each built their own GatewayBudgetClickHouseRepository / GovernanceOcsfEventsClickHouseRepository per call; both now take the instance from getApp(). governance.ts's recordWorkspaceView did the same for its OCSF write. QUERIES (Shape C). Five new repositories carry the SQL, tenant scoping and row decoding that used to live in the services: - GovernanceTraceActivityClickHouseRepository (trace_summaries reads shared by GovernanceSetupStateService's activity probe and QuarantineFillEvaluator's per-source breakdown) - GovernanceKpisClickHouseRepository gained findSpendTotals (the spend-spike evaluator's current/baseline window comparison), joining its existing insertContribution write side - GovernanceOcsfEventsClickHouseRepository gained findAll (the SIEM export read), joining its existing insertEvent write side - PersonalUsageClickHouseRepository (five queries behind the /me dashboard: summary, top model, daily buckets, model breakdown, and the PRINCIPAL-ledger union for ingestion-source traffic) - BillableEventsClickHouseRepository (billing-month rollups); the exported billableEventsQuery.ts functions stay the public API so every existing caller — including the one threaded through pipelineRegistry.ts as a plain function reference — needed no change, but now reach ClickHouse only via getApp().billableEvents internally Each service keeps validation, merge-two-sources business logic and fail-safe error handling; a query-time failure still degrades to zeros the same way it always did, and "no ClickHouse repository at all" still throws where the original client lookup would have thrown. App gained: governance.{ocsfEvents,traceActivity,kpis,personalUsage} and a top-level billableEvents slot, all undefined on a deployment without ClickHouse. spendSpikeAnomalyWorker.ts and user.ts (existing callers of the migrated services) updated to pass the App's repositories through. SKIPPED: activityMonitor.service.ts (10 statements across ~1100 lines of department-attribution and sort-whitelisting business logic interleaved with SQL) — too large and too risky for a single-session migration alongside the rest of this batch. Left on the client-access-boundary backlog for a follow-up. The governanceOcsfExport integration test that could previously only assert on a raw SELECT (its comment explained the service's client resolver couldn't be overridden) now drives GovernanceOcsfExportService.list end-to-end against the test ClickHouse client, since the constructor injection this refactor adds is exactly what unblocks that. * fix(traces): resolve the blob-resolution client lazily, not while building deps Routers build these deps per request, including on paths that never resolve a blob. Reading `getApp()` while assembling them therefore made merely constructing the deps require an initialised App, and every such route 500'd wherever one is not - seven unit tests in the traces REST route, and any caller that builds deps before boot completes. The App is still the single source of the resolver. It is consulted when the resolver is called rather than when it is handed over. * refactor(clickhouse): route ClickHouse access through the App for routes/routers Eight more entries off the access backlog: the gateway-spend and webhooks REST apps, the gatewaySpendEvents and user tRPC routers, the ingestion and gateway-internal Hono routes, ops.ts, and the stored-objects cross-tenant lookup. DEAD RESOLVER, LIVE REPOSITORY. gateway-spend/app.ts and webhooks/app.ts each built their own GatewaySpendEventsRepository / WebhookEventsClickHouseRepository inline. Both now come from getApp().gateway, alongside the budgets and virtualKeySpend repositories already there. gateway-internal.ts and user.ts had the same duplicated GatewayBudgetClickHouseRepository construction; both now take getApp().gateway.budgets. NEW SERVICE LAYER. gatewaySpendEvents.ts (tRPC) and gateway-spend/app.ts (REST) called repository read methods directly, which the routing layer must never do. GatewaySpendEventsService wraps the repository so both surfaces call a service instead. NEW REPOSITORIES. stored-objects-cross-tenant-lookup.ts fanned out to every ClickHouse instance itself; the fan-out and per-instance query move to StoredObjectOwnerClickHouseRepository (kept under src/server/stored-objects/ — AC16's no-retention-GC scan confines every stored_objects reference to that module). ops.ts resolved the ops/shared client and ran the EXPLAIN query inline; OpsExplainClickHouseRepository now owns client resolution and query execution, OpsExplainService owns the fail-closed/fallback-warning decision the route used to make itself. ingestionRoutes.ts's budget-debit path already had `getApp()` in scope from an earlier commit; it just wasn't using it for its own inline GatewayBudgetClickHouseRepository construction. Every graceful-degradation branch is unchanged: a deployment without ClickHouse still returns the same empty/zero/503 responses it did before, now gated on repository presence instead of `isClickHouseEnabled()`. Tests updated to mock getApp() instead of the ClickHouse client module, per the pattern gatewayBudgets.perPerson.unit.test.ts established. ops-explain.integration.test.ts and gateway-internal.config-route both needed a getApp() stand-in they didn't need before — verified against a real ClickHouse (testcontainers), 12/12 and passing. The boundary test's backlog still lists all eight files; that's the ratchet working as designed, not a miss — removing backlog entries is the merge orchestrator's job across all four parallel batches. * fix(app-layer): repair a comment lost in the merge, drop agent labels The four parallel migrations were merged one at a time, and resolving the dependencies.ts conflicts by keeping both sides swallowed a `/**` opener, leaving an orphaned comment block. Every unit test still passed and tslsp reported nothing; biome's parser was the only thing that caught it, which is worth remembering about type-only modules. Also renames the write-side billable-events repository to `BillableEventsMeterClickHouseRepository`. Two of the migrations independently created a class of the same name - one the meter projection's write path, one the billing read path - and presets.ts importing both was a duplicate-identifier parse error. They are genuinely different repositories; now they read that way. * test(clickhouse): tag and bind the single-client-access scenarios CI's feature-parity gate refuses a spec that enforces nothing, and an untagged file enforces nothing: seven scenarios, zero measured. Tagged @Unit and bound to the tests that prove them. Two of the seven had no test to bind to, which is the useful part of this: - "a slot is held across retries" is the core claim of the composition order, and nothing asserted it. Now a stand-in client retries inside one call and the test proves a statement queued behind it cannot start between those attempts. Composed the other way round this fails. - "every client is built the same way" was the claim the private-instance client had been quietly breaking. Now asserted directly: both clients get the same pool size and driver settings, and each registers its own metrics label. * test(clickhouse): tag and bind the single-client-access scenarios CI's feature-parity gate refuses a spec that enforces nothing, and an untagged file enforces nothing: seven scenarios, zero measured. Tagged @Unit and bound to the tests that prove them. Two of the seven had no test to bind to, which is the useful part of this: - "a slot is held across retries" is the core claim of the composition order, and nothing asserted it. Now a stand-in client retries inside one call and the test proves a statement queued behind it cannot start between those attempts. Composed the other way round this fails. - "every client is built the same way" was the claim the private-instance client had been quietly breaking. Now asserted directly: both clients get the same pool size and driver settings, and each registers its own metrics label. * fix(clickhouse): stop logging a URL parse failure that can carry credentials Review catch (CodeRabbit), and the right catch. A ClickHouse URL carries a password, and Node attaches the offending string to an ERR_INVALID_URL as `input` - which a structured logger serialises straight into the log line. The instance label is all an operator needs to know which client failed. Three more from the same review: - The construction rule matched only the driver's root specifier, so an import from `@clickhouse/client/web` would have built a client the guard could not see. It now matches any entrypoint of the package; probed with a deliberate subpath import, which the guard names. - The exemption doc block described three entries for a set of five. - The statement-limit tests copied the queue-depth floor by hand, so changing it in the module would have left them asserting a stale number. They import it now. The abandoned-request test also accepted any rejection, which an overload refusal would satisfy; it pins AcquireAbortedError. * fix(clickhouse): keep the overload error, and stop the gauges outliving their limiters Three more from review, and the first is the one that mattered: FilterService caught every repository error and rewrapped it as a generic "Failed to fetch filter options". That flattened ClickHouseOverloadedError - the typed 503 this branch adds so a shed statement can tell a caller to retry in a moment. The error existed and the one service reading through it threw it away. Handled errors now pass through; everything else is still swallowed, because a raw ClickHouse message embeds the failing SQL. `unregisterClickHouseLimiter` had no caller, which is both a YAGNI break and a leak: a closed client left its probe registered, so the gauges kept publishing a bound that no longer fronted anything. The shared client drops its probe on close and the private-instance cache drops each on eviction. The gauges also reset before publishing - `labels().set()` only ever writes, so a removed probe would otherwise leave its last value up for the life of the process. `close` and `ping` are not statements, so they were left to the prototype chain - but the default-settings proxy forwards with `receiver`, so they ran with `this` bound to the facade rather than the driver. A driver that uses `this` for teardown would break on exactly the cleanup path. Both are now applied to the real client. * fix(clickhouse): type the retry test's release queue CI's typecheck caught what the test run could not: TypeScript narrows a `let` that is only ever assigned inside a closure back to its initialiser, so `releaseAttempt?.()` was "not callable" even though it is called at runtime. A queue of resolvers instead of a single reassigned slot. Same test, same assertions, no narrowing to fight. Worth recording why this reached CI: `tsconfig.tsgo.json` excludes test files, so `pnpm typecheck` never sees them and `tslsp diagnostics` on the file was clean. `pnpm typecheck:tests` is the separate CI step that covers them, and it is the one to run after touching a test. * fix(clickhouse): rebind the meter repository, and two type errors from the merge CI's typecheck found three things the 18,876 passing tests did not. The important one is mine. Renaming the write-side repository to `BillableEventsMeterClickHouseRepository` was meant to separate it from the billing read repository of the same name. The rename ran while presets.ts still had the duplicate-identifier parse error, so it could not resolve every reference - and both `billing.events` construction sites were left pointing at the READ repository, which has no `insert` and takes a different constructor. The write path would have failed at runtime; only the type checker saw it. Two from the parallel batches: - `NullExperimentIdLookupRepository.findExperimentId()` declared no parameters. That satisfies the interface, but a caller holding the concrete type cannot pass the arguments the contract defines. - The ops-explain repository typed its per-query guardrails as `Record<string, unknown>`, which is wider than the driver's own `ClickHouseSettings`. `pnpm typecheck` and `pnpm typecheck:tests` both pass; the tests still do too. * test(clickhouse): give the integration fixtures an App to read repositories from Eight integration suites failed with "App not initialized". Not a bug in the routes - a consequence of the rule working. Those paths no longer resolve a ClickHouse client of their own; they take a repository from `getApp()`. A fixture that boots a real HTTP app but no App singleton therefore 500s, and the failure points at the route rather than at itself. `installClickHouseTestApp` builds a test App whose ClickHouse repositories are real and dial the test's own container. `createTestApp` alone cannot do this: its ClickHouse slots are null, which is right for a unit test and useless for one asserting on rows. Every slot is wired rather than only the one each caller happens to need. A test about ingestion should not have to know which repository the route reaches for, nor start failing because a later change made it read one more. The resolver may answer null - the container helpers do - and the helper wraps it into the throws-if-unavailable contract the repositories expect, so no fixture repeats that check. Each suite drops the singleton in `afterAll` so it cannot leak into the next file sharing the worker. Verified as far as this machine allows: `pnpm typecheck`, `pnpm typecheck:tests`, biome and 18,876 unit tests all pass. The integration suites themselves need containers this host does not have, so CI is the check. * refactor(clickhouse): decompose what the extraction made too complex The house-rules gate counts new lint violations against the base, and moving query logic out of services into repositories carried its complexity with it. The complexity was not new; the files were, so it counted as new. Splitting is the better answer than an override: - `TraceEvaluationsClickHouseRepository` had a retry nested inside a catch inside a method holding two closures. Now the read, the grouping, the light-projection retry and the failure report are each their own method. Behaviour is unchanged, including the part that matters: only a memory-limit error earns the second attempt, and everything else still fails immediately rather than spending the same budget to fail identically. - The legacy analytics row mapping is a function rather than a closure inside a `.map`, and the vote/score attribute spellings are a lookup rather than six inline comparisons. - `startScenarioProcessor` grew past the line limit when the orphan sweeps gained a guard. The sweeps are now `startOrphanedRunSweeps`. scenario.processor is back to its pre-existing count; the two repositories are at zero. * test(clickhouse): fix the last two integration fixtures, and the trap behind one `evaluation-payload-offload` read null where it expected the inputs, and the reason is worth writing down because it will catch someone else. `defaultClickHouseClientResolver` lives inside clickhouseClient.ts and calls `getClickHouseClientForProject` as a local binding. A test that mocks that export - spreading `...actual` and replacing one function - does NOT intercept it: the resolver keeps the module-internal reference and quietly reaches the real one. Every repository built from the default resolver therefore ignores the mock. The fix here is to inject the repository rather than rely on module mocking, which is also the more honest test. `end-user-spend-routing` was my own doing. It mocks `~/server/app-layer/app`, so the fixture helper I added imported `globalForApp` and `resetApp` from the mock, where they do not exist. It needs its existing `getApp` mock extended with the gateway repositories instead - built per call, since the client only exists once the containers are up. Checked the other seven wired suites for the same conflict; none has it. * test(clickhouse): assert the shared driver settings are present, not just equal Two clients agreeing that a setting is absent is not the property the test exists to hold. Pin presence on the first before comparing the second, and drop a comment that only restated the mock beneath it. * refactor(clickhouse): remove the exported client resolver, inject the root's `defaultClickHouseClientResolver` was a third door into ClickHouse: any module could import it and hold a client without going through `getApp()` or a repository. The resolver TYPE stays exported; the one resolver VALUE is built in `presets.ts` and travels from there by injection. The three non-test holders now take it from the composition root: - topic clustering takes the resolver as a parameter of `clusterTopicsForProject`. `presets.ts` binds it once and hands the bound page to both the event-sourcing run port and the App, so the manual task runs it through `getApp().topicClustering.runPage` instead of wiring its own (and now boots the App it already needed for `recordTopics`). - `defaultGraphTriggerHeartbeatDeps` requires the resolver instead of defaulting to the import; `buildAutomationDispatchPorts` threads it from `presets.ts`. - the per-trace evaluations repository is built in `presets.ts` like every other repository and handed out as `getApp().evaluations.traceEvaluations`; `EvaluationService` reads it lazily, since it is constructed inside `TraceService` on paths that never query evaluations. The topic-clustering tests inject a resolver through the new parameter rather than module-mocking `clickhouseClient`. The access ratchet drops the resolver from its name list, loses the two files that no longer resolve directly (backlog 5 -> 3), and gains a rule that fails if `clickhouseClient.ts` ever exports a resolver value again. * refactor(clickhouse): named dependency objects, handled errors, a service at the route Review pass over the client-access migration. The substantive ones: - the access ratchet now checks an exhaustive list of `clickhouseClient.ts`'s VALUE exports instead of matching a type annotation. `export const resolveClient = async (tenantId: string) => ...` is structurally a resolver with no annotation to match, so the annotation check could not see the exact thing it exists to keep out. - `findManyByTraceIds` resolved its client outside the try, so the one call that could not reach ClickHouse at all failed with the resolver's own error while every call that reached it and failed got the repository's. - `buildTraceBlobResolutionDeps` decided `clickhouseEnabled` from `isClickHouseEnabled()` while its resolver comes from the App, which gates on `!!config.clickhouseUrl || isClickHouseEnabled()`. On a `CLICKHOUSE_URL`-only deployment the two disagreed and ADR-022 full reads degraded to previews. - the gateway-spend surface threw a plain `Error` for a missing repository, which skips the `clickhouse_unavailable` code and the 503 contract. - `/api/ops/clickhouse/explain` read `getApp().opsExplain.repository` and built its service in the route. The App hands out the service now. The rest are the repo's own rules on code this PR wrote: named dependency objects for the constructors and factories it reshaped (evaluations, analytics, CLI bootstrap, governance setup/OCSF/spend-spike, billable-events insert, collectUsageStats), `isRetry`/`hasStarted` boolean names, `when` blocks in the suites that had none, one shared evaluations test helper instead of the same ten lines twice, and two comments that had stopped describing what the code does. * test(billing): assert the named insert argument the meter store now passes * refactor(clickhouse): the instance-usage repository from the App, and two exhaustiveness fixes - the export rule claimed to catch anything `clickhouseClient.ts` exports and did not parse `export default` or `export * from`, both of which can carry a resolver. Neither can hold a name an allowlist could match, so they are reported as themselves and always fail. - `createDefaultInstanceUsageStatsRepository` built the usage-stats repository over an imported `getClickHouseClientForOrganization` — the same side door this branch is closing. It is built in `presets.ts` now and handed out as `getApp().usageStats.instance`, alongside `billing.events`, which is organization-keyed for the same reason. - `findExperimentId(tenantId, runId)` takes a named object, through the interface, both implementations, the pipeline registry and the tests. * fix(app-layer): expose usageStats on App and give the test app a throwing resolver * refactor(clickhouse): split the export parser into one function per export form * test(clickhouse): read exports regardless of indentation, and pin the reader Every export pattern allows leading whitespace. A top-level export sits at column 1 today, but a rule that claims to see everything a module exports must not be one reformatting away from missing one. The reader is now exercised rather than assumed: a fixture carrying all four value-export forms, two of them indented, plus the two type-export forms it must not mistake for values.
… route coverage (#6605) * feat(api-docs): publish the experiments REST API and gate route coverage A customer evaluating LangWatch read the API reference, found no way to create an experiment, and concluded the REST API could not do it. It could: POST /api/experiment/init is the call every SDK makes first, and it had been serving traffic the whole time. It had simply never been annotated, so the spec generator skipped it, so no reference page existed. Publishes the whole experiment round trip, and adds a gate so the next endpoint cannot go missing the same silent way. Experiments in the reference: - POST /api/experiment/init create, or return the existing slug - GET /api/experiments list, with run counts - POST /api/experiments/{slug}/run start a run - GET /api/experiments/runs runs for one experiment - GET /api/experiments/runs/{runId} poll one run - GET /api/experiments/runs/{runId}/results per-row results execute and abort stay unpublished: both authenticate with a browser session, so an API-key caller cannot reach them. execute needed an explicit hide, because its body validator was metadata enough for the generator to publish it by accident. The gate, scripts/check-openapi-route-coverage.ts, compares every registered route to the document. Each of the 124 absences is now either internal, a deprecated alias, or a named gap with the reason it is still open, and the list is ratcheted so an entry that stops explaining anything fails. That is also the answer to "can we generate the spec for everything": yes, mechanically, and the gate now says what is left. Also drops path entries left empty by a hidden route, and moves the shared Hono route-table parsing into scripts/lib so both gates read it. * fix(api-docs): document the poll endpoint's real summary shape The SDK's generated client compiles against the spec and refused to build: `GET /api/experiments/runs/{runId}` was documented with the ClickHouse aggregate summary (datasetCost, evaluations) when the handler actually returns the run-state one (totalCells, completedCells, failedCells, duration, runUrl, plus the per-target and per-evaluator CI breakdown). Two different objects both called `summary`, one per endpoint. The list response really does carry the aggregate, so that stays; only the poll response was wrong. Both schemas now name the other so the next reader does not have to rediscover the collision. Also picks up the regenerated SDK OpenAPI client, which now has typed access to all six experiment endpoints. * fix(api-docs): document the targets a run's results resolve against `GET /api/experiments/runs/{runId}/results` returns `targets` alongside the rows, and the schema left it out. Every dataset row and evaluation carries a `targetId`, so without it a reader has the ids and no way to tell which prompt, agent, or evaluator each one names. * fix(api-docs): document domainError on the failure paths that return it Both the poll response and a failed dataset row carry `domainError`, the serialised handled error, and neither schema mentioned it. That is the field an integrator should branch on: `error` is the engine's own string and will change, `domainError.code` is the stable discriminant (ADR-045). Leaving it undocumented pushes callers toward string-matching the one field we tell everyone else not to match on. * fix(api-docs): address review, and correct three documented shapes The lint gate flagged one new Biome violation: formatReport had grown to cognitive complexity 17. Split into formatUnexplained + formatStale, the same shape check-openapi-completeness.ts already uses. Three documented shapes were wrong, all caught in review: - the experiment/init body declared `oneOf` on slug-or-id. The handler's refine only asks for at least one, and accepts both, so `oneOf` — which means exactly one — documented a rejection that does not happen. - 401 and 404 on the experiments routes were documented as `{error}`. A 404 is a thrown HandledError the boundary serialises, so it carries the code in `error` plus the error's meta spread alongside it. Both now use one open envelope schema that covers the handled and fallback forms. - GET /api/experiments reads page and pageSize and declared neither. Nothing caught it: the completeness gate only covers /api/gateway/v1 and /api/webhooks/v1. Also: POST /api/experiments/{slug}/run answers three hand-rolled 400s and documented none. They are documented as sent, with a note that converting them to HandledError is the right end state but turns `error` into a code slug and would break CI scripts matching the current text. The UNPUBLISHED entry for POST /api/admin/{resource} excused nothing — the /api/admin prefix already covered it — and the shadow guard that was supposed to catch that had a hole: it built its probe as `GET ${match}`, so an entry already in operation form became `GET POST /api/...` and matched no prefix. Guard fixed and pinned, entry deleted. Experiments pages now follow the lifecycle their overview describes. The default page sort is CRUD-shaped, which is right for a resource and wrong for a sequence of steps, so a group can declare its own order and anything it does not name keeps the old sort behind it. * fix(api-docs): match the app-derived prune to whole path segments, and document the run body Six findings from review, all real. The prefix match that decides which paths are app-derived was a substring test, applied at every level of the merge. It happened to be right on today's 124 paths, but "contains /api/experiment" is not the question being asked, and a future /api/experimental-runs would have been pruned by a rule written for a different surface. One `isAppDerivedPath` helper now answers it at a path-segment boundary, for both the prune and the replace-instead-of-merge, and the singular and plural experiment prefixes are listed separately because they really are two surfaces. `POST /api/experiments/{slug}/run` accepts an optional body -- inline rows, a dataset id, constant parameters, a row subset -- and documented none of it, so the reference read as though the endpoint took no input at all. The generated clients now type it. `POST /api/experiment/init` documented one 400 and one 403 while answering two of each: a body that is not JSON answers `message` where a body that fails validation answers `error`, and the API key ceiling refuses with the handled envelope alongside the plan limit's own. Both are documented as sent. Plus three house rules: `as const` on the two exported config arrays, object destructuring for `excludes` and `joinRoutePath`, and a schemas header that no longer under-claims what the file holds. * refactor(api-docs): pass named parameters to the two new multi-arg helpers * refactor(evaluations): move the evaluator display-name map next to the catalog `evaluatorTempNameMap` lived in a Chakra component, and a legacy route handler imported it from there to name evaluators in its catalogue response. That is a server file reaching into the UI tree for a plain `Record<string, string>`, and it is what stops that route's Hono app from being wired into the OpenAPI spec task: importing it would pull Chakra, react-hook-form and the tRPC client into a task bundle that renders nothing. The map is a display override on the evaluator catalog, so it belongs beside the catalog, in the pure module all six consumers already import. The drawer's integration test mocked the component purely to stub this map. With the map in a module that costs nothing to import, the mock has nothing left to do and is gone; the test now runs against the real one. * feat(api-docs): publish the seven legacy REST routes that were serving traffic unlisted Seven endpoints that customers and SDKs call every day were absent from the API reference for the same reason experiments were: nobody had annotated them, so the generator skipped them and no page was ever written. They are not deprecated and they are not internal. They are the ones an older SDK still sends to. POST /api/analytics timeseries, legacy path POST /api/track_event event tracking, legacy path POST /api/dspy/log_steps optimizer step reporting POST /api/trigger/slack Slack alert triggers POST /api/workflows/{workflowId}/run synchronous workflow run POST /api/workflows/{workflowId}/{versionId}/run version-pinned run POST /api/optimization/{workflowId}/{versionId} the run's legacy path Each documents its real body, its success shape and every refusal it sends, including which of `message` and `error` carries the sentence: these predate ADR-045 and answer prose rather than a code, and a reader guessing the wrong field gets `undefined`. `resolver()` only types against `responses`, and every one of these parses its body by hand rather than through `zValidator`, so there was nothing for the generator to read a request body off. `requestBodySchema` converts the zod schema the handler already uses, which keeps the documented body and the parsed body the same object rather than two descriptions that can drift. The Slack trigger's schema had been written twice for that reason; it is now written once. The docs generator's second ratchet caught all four unowned paths on the first run, as designed. Workflows takes the optimization spelling, Experiments takes the DSPy steps, Triggers takes the singular Slack form, and Events becomes a group of its own -- which also gives `POST /api/events/track` the page it had been waiting on, so the canonical spelling is no longer the undocumented one. The experiments reference test asserted on a substring of the generator's prefix list, which went red the moment that list legitimately grew. It now asserts what the customer actually needed: that every experiment operation has a page. * feat(api-docs): publish the evaluate family, the endpoint the customer went looking for "I was at the REST API, and it doesn't seem like there's a way to..." was about creating an experiment, but running an evaluator had the same hole, and it is the call most people reach for first. GET /api/evaluations/list the catalogue POST /api/evaluations/{evaluator}/evaluate run one evaluator POST /api/evaluations/{evaluator}/{subpath}/evaluate two-segment ids POST /api/guardrails/{evaluator}/evaluate the same call, gating The three evaluate routes share one handler and one envelope, so they share one set of schemas. What varies is the id in the path, and that is deliberately not enumerated: it can be a built-in id, a monitor slug in the caller's own project, or `evaluators/{slug|id}` for a saved one. Two of those three name rows we cannot know, so the description says so rather than publishing a closed list that is wrong for everybody. Per-evaluator detail stays in openapi-evals.json under Built-in Evaluators. That document declared every evaluate response as an array, which is LangEvals' own FastAPI shape: it evaluates a batch, we evaluate one input and answer one result. Left as-is it would have contradicted the operation being added here. The generator is corrected and the 39 committed operations are patched in place, rather than regenerating -- the generator fetches from the langevals repo and a full regen would pull four new evaluators and unrelated churn into this change. Two things publishing forced into the open, both fixed here: `GET /api/evaluations/list` declared `credential: "apiKey"` and never resolved a token, because `handlerManaged` applies no middleware. A declaration nothing enforces is worse than none -- it reads as a credential requirement to anyone auditing the registry while letting an unauthenticated request through. The handler genuinely returns the same compiled-in list to everybody, and the SDKs read it before they have a key, so the honest fix is to declare it public and say why. The operation restates `security: []` so the document stops claiming a key is needed. That endpoint also ran `zodToJsonSchema` over ~40 settings schemas on every request, for an answer that never changes. Unauthenticated plus recomputed is a CPU amplifier; it is built once now. Also drops two orphaned reference pages for `/api/evaluations/v3/*` operations that left the spec long ago -- no nav entry, no operation, and the group they sat in was being skipped entirely because it matched nothing. * feat(api-docs): finish the evaluations family, and sort the deliberate exclusions Publishes the two remaining evaluations routes: POST /api/evaluations/batch/log_results the second half of an SDK batch run POST /api/dataset/evaluate one evaluator across a saved dataset `log_results` is the call an SDK makes after `POST /api/experiment/init`, so the pair now reads end to end in the reference instead of stopping halfway. The exclusion list grows a fourth category, `elsewhere`: public, and deliberately documented outside the API reference because an operation is the wrong shape for it. `POST /api/collector` and `/api/ingest` are guides rather than request schemas, and `/api/otel/v1` implements someone else's protobuf contract. Filing them as gaps implied a debt that should shrink; they are settled answers, and mixing the two made the gap count meaningless. The two export downloads move to `internal`: both authenticate a browser session, so an API-key holder cannot reach them at all. They back the dashboard's download button, and `/api/traces/search` is the programmatic equivalent. `/api/annotations/trace/` named its segment `:trace` while the document called it `{id}`, so the route read as undocumented to anything comparing the two. The URL is identical either way; the handler now spells it the way the document does. Gaps: 22 at the start of this work, 3 now -- SCIM, which lands with the management APIs, and the two project API-key reads, which need their hand-authored operations moved into the app first. * feat(api-docs): publish the project API key endpoints, and move their family into the app Reading and regenerating a project's API key were the last two routes a customer could call but not find. They were unpublishable for a structural reason rather than an oversight: every other operation in the projects and api-keys families was authored by hand in the JSON while the routes carried a one-line `describeRoute`, so the generator could not produce them and the document survived only by never being regenerated over. Adding two operations meant either hand-writing two more, or finishing the move. This finishes it. The eight existing operations move verbatim, `operationId` and `security` included -- the ids become the Python SDK's function names, and this family takes an organization admin key rather than the project key the document requires at its root. Diffed operation by operation against the previous document to prove the move lost nothing. What the diff did surface is that two hand-written request bodies had drifted from the code that validates them: PATCH /api/projects/{id} documented `piiRedactionLevel`, which the schema does not accept, and omitted `teamId`, which it does -- while the operation's own description told you to use `teamId` to move a project between teams POST /api/api-keys omitted `keyType` entirely, so the choice between a personal and a service key was invisible Both are right now, and cannot drift again: the bodies come from the same zod schemas the handlers validate with. The per-field descriptions the hand-written entries carried move onto those schemas rather than being lost, which is also where they should have been. Gaps: 22 at the start of this work, 1 now -- SCIM, which lands with the management APIs. * fix(cli): call endpoints that exist for --wait and template cloning Two commands addressed routes nothing serves. Both are broken in main, and neither is caused by this branch -- they surfaced while checking whether the API reference covers everything the CLI does. `langwatch suite run --wait` and `langwatch scenario run --wait` polled `GET /api/scenario-events?batchRunId=`. That app registers two POSTs and a DELETE, and no GET at all. So every poll 404'd, the consecutive-failure budget ran out, and the wait ended by reporting the status endpoint as down -- exit 1 on a run that may well have passed. The flag has never worked. `GET /api/simulation-runs?batchRunId=` is the endpoint that answers, and it is already documented. It returns the runs rather than a tally, so the counting moves into one helper both commands share: a batch that reads "done" in one command and "still running" in the other would be worse than either. Two things the old shape got for free and this has to do deliberately: follow the cursor, because a suite can hold more scenarios than the 100 a page returns, and take the dispatched job count as a floor, because a batch whose runs are still being created would otherwise read as finished on the first poll. `langwatch governance ingestion-templates clone-from-platform` posted to `/ingestion-templates/clone-from-platform`. The route is `/ingestion-templates/clone`, which is what both the OpenAPI document and the governance guide say; only the CLI disagreed. The regression tests execute the real code path against a stub that serves exactly the routes the app registers and 404s everything else, so a caller reaching for a path that does not exist fails there the way it failed in production. Reverting either fix turns them red, which was checked rather than assumed. No coverage gate could have caught these: the route-coverage check compares registered routes against the document, and a CLI calling a route that was never registered is invisible from both sides of that comparison. * fix(api-docs): close the coverage gate's own blind spots The gate claimed to compare every registered route against the document. It was reading 283 of 317, and the 34 it could not see were invisible for two reasons of its own making. Seven `app.v1.ts` files export a register function that a sibling `app.ts` calls against an app it constructed, so they declare no basePath themselves. Reading only what a file declares skipped every route in them -- most of the prompts, traces, evaluators and scenarios v1 surfaces. The basePath now falls back to the sibling's. A parameter carrying a Hono regex constraint templated with the constraint attached: `:id{.+?}/versions` became `{id}{.+?}/versions`, which matches nothing any document would ever contain. The constraint is Hono's routing detail and has no OpenAPI equivalent, so it comes off. Nothing was hidden by either: all 34 newly-visible routes were already documented, which is why the count moved from 176/283 to 210/317 with no new failures. That is the point. A gate that cannot see a route cannot notice one going missing, and this one exists precisely because an endpoint went unnoticed for long enough that a customer concluded the REST API could not do the thing it had always done. * fix(feature-map): name the MCP tools that exist for experiments The experiments entry pointed agents at `platform_run_evaluation` and `platform_evaluation_status`. Neither has ever been registered: the tools are `platform_run_experiment` and `platform_experiment_status`. The feature map is embedded into the CLI at codegen time and shipped inside the npx server package, so a name in it is a name an agent will try, and these two were dead ends. The entry now lists the whole experiment surface the server registers, so the platform column mirrors the CLI column above it: list, run, status, list-runs, results. Nothing compared the map's names against the server's registrations, which is why the two drifted unnoticed. The guard added here does, and fails with the offending names rather than a count. * fix(api): forward experiment_id on init, and stop putting step payloads in telemetry Four findings from review of the last few commits, all real. `POST /api/experiment/init` accepts either identifier and forwarded only the slug. An id-only request passed validation and then hit "Either experiment_id or experiment_slug is required" as a 500 -- the one code path where documenting the endpoint made the gap visible, since every other caller of `findOrCreateExperiment` forwards both. `POST /api/dspy/log_steps` logged and reported the whole step on failure. A DSPy step carries the caller's dataset examples and the prompts and completions of every LLM call in it, so a validation error shipped customer content to the log and to PostHog. The run and step ids identify it; for the timestamp check, the offending timestamp is the whole diagnosis. The same route returned the caught error's message to the caller on 500. That failure comes from a dependency the caller cannot see or act on, and its text can name internals. It answers a generic body now, with the detail kept in the log line, and the 500 is documented rather than undeclared. `evaluatorTempNameMap` was typed `Record<string, string>`, which widened its entries and left the exported object mutable. It is `as const` now, and private: the seven call sites that indexed it with an arbitrary name go through `evaluatorDisplayName`, which is the operation they were all open-coding anyway.
* docs(observability): the queue is groupQueue, not BullMQ The comments added in #6640 credited the Redis span volume to BullMQ. BullMQ is not a dependency of this repo: it is absent from every package.json, absent from the lockfile, not installed, and imported nowhere. The measurement was right and the attribution was wrong. The traffic comes from groupQueue, the in-house Redis queue under src/server/event-sourcing/queues/groupQueue. Confirmed against the span names: `evalsha` is cachedLuaScript executing the scripts in scripts.ts, `lpush` is groupQueue itself, `zrange` is metricsCollector, and the ready-set and lease upkeep account for the rest. Four nearby comments claiming BullMQ is in use are corrected too, since they are what made the wrong attribution look supported. One of them read "the same BullMQ queues and GroupQueue streams", naming both the dead library and the real one in a single sentence. Comment-only; no behaviour changes. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z * docs: finish the BullMQ sweep, name groupQueue where the queue is meant BullMQ is not a dependency of this repo: absent from every package.json, zero hits in pnpm-lock.yaml, not in node_modules, never imported. The real queue is the in-house groupQueue under server/event-sourcing/queues/groupQueue. Stale comments claiming BullMQ is in use now name groupQueue, or just "the queue" where the implementation does not matter. Historical clauses ("replaces the deleted BullMQ pair", "parity with the BullMQ worker this replaces") are dropped, keeping the substantive facts they carried, such as the 3-attempt retry policy. Three test names that read BullMQ are renamed to describe the same behaviour. The nine remaining mentions are deliberate: five state correctly that BullMQ is NOT used (the signal that stops the misattribution), and four are comparative or sample data, including the "Why a Custom Queue" section of groupQueue/ARCHITECTURE.md and the foundry conversation fixture. Comment, doc and test-name changes only. No executable code touched. Claude-Session: https://claude.ai/code/session_0126RRNA7dpcKCdgcWApqY2Z
* feat(charts): alert when a ClickHouse backup stops succeeding The clickhouse-serverless chart schedules backups and nothing watched whether they worked. Adds Prometheus alerting rules, off by default, built entirely on kube-state-metrics so the chart ships no exporter. The primary rule is freshness rather than failure: a CronJob that is suspended, deleted, or silently rescheduled produces no failure event at all, so a failure-only alert stays quiet while backups stop happening. It carries a second branch for a backup that has never succeeded even once, which has no kube_cronjob_status_last_successful_time series and so is invisible to the obvious form of the query. Each target also gets a faster failure alert scoped to recently created Jobs, a suspension alert, and an absent-metrics alert so that losing the CronJob or kube-state-metrics is noisy rather than silently resolving every other rule. * fix(charts): let backup alerts target a CronJob this chart does not create Adds cronjobName for an exact CronJob name, alongside cronjob for a suffix the release fullname is prepended to. The README already promised targets could name CronJobs the chart does not create; only the prefixed form existed, so such a target selected a name that does not exist and ClickHouseBackupCronJobMissing would have fired forever. Setting both or neither now fails the render. Extends the tests to cover the PrometheusRule output mode and to assert the disabled cases by the manifests that come out rather than by how helm reports an empty --show-only selection.
…e assistant tabs disagreeing with the docs (#6653) * fix(ui): give the onboarding accent surfaces a dark mode, and stop the assistant tabs disagreeing with the docs A customer evaluating the product hit two things on the first screens they saw. The onboarding intent card's title became unreadable the moment they picked it: the selected background was `orange.50`, a raw palette step that renders the same near-white in both modes, under a title on `color="fg"`, which flips to near-white in dark mode. Same untreated value sat on the icon chips of three onboarding screens. The fix is one accent-surface module naming both sides of every accent, so there is a single place to change it. SelectableIconCard already had the right pair hardcoded and now reads it from there too. Separately, the Token Created dialog offered two coding assistants while the docs publish the MCP server for more than that, and the same file listed five *different* editors for its config-path chips. Both surfaces now come from one CODE_ASSISTANTS list. Assistants with a published installer (Claude Code, Codex, Gemini) get their terminal command; the rest name the config file they read instead of being handed a command that does not exist. Regression coverage reads the injected stylesheet rather than computed style, because jsdom cannot resolve Chakra v3's emitted rules and returns a transparent background in both modes — the obvious assertion proves nothing. The wider defect class — no written rule, no lint check — is #6652. * fix(api-keys): drop the Gemini tab rather than ship an mcp add command that does not run Review caught that the Gemini builder emitted gemini mcp add langwatch --env LANGWATCH_API_KEY=… -- npx -y @langwatch/mcp-server which Gemini's CLI does not parse: its options must precede the server name, and `--` separates Gemini's own flags from arguments passed to the MCP server process rather than introducing the command. It was modelled on the Codex builder, where `--` genuinely is the separator. That string sits behind the "you won't be able to see it again" warning, so a user who copies it spends their one look at the token on a line that fails. Removing the tab is the honest state until the command is checked against a real CLI; the whole fix is #6654. Codex loses its config-path chip for a related reason: the chip row is rendered from every entry carrying a configPath, and Codex reads TOML while the block above it renders JSON. The base branch deliberately omitted it. Also tracked in #6654. Claude Code and Codex commands are unchanged and were proven byte-identical to main across every flag combination. * test(api-keys): stop the Gemini-exclusion guard claiming to cover a spec scenario The guard added alongside the removal carried `@scenario "An assistant without an install command points at its config file"`, which it does not exercise — it asserts an entry's absence by key. The scenario was already correctly bound by the test above it, so parity stayed green while an annotation pointed at the wrong test: exactly the vacuous-binding shape the repo warns about, in miniature. Dropped the annotation and renamed the test to say what it actually does, including the issue that retires it. * fix(api-keys): bind the spec annotations that bound nothing, and test the rendered tabs check:feature-parity was FAILING and the earlier runs missed it — the summary was never read, only one feature's line was grepped. Four annotations referenced scenarios that do not exist: - Three in the onboarding colour-mode test invented titles for scenarios never written to a .feature file. This is a bug fix, and the repo does not open feature files for bug fixes, so the annotations go rather than the scenarios arriving. - The fourth was self-inflicted: a comment saying the test carries no "@Scenario" annotation contains the literal token, so the parser bound the prose that followed it. Reworded. Also from review: - The unselected-card assertion only checked that Chakra emitted a class, which a raw background does too. It now asserts the background reaches for a semantic `bg` variable and names no palette step. That surface is mode-aware by a different route than the selected one — a semantic token's value flips at :root, so it has no element-scoped `.dark` rule to find. - The assistant tests poked the registry from a file named `.integration.`. They now render the dialog, click a tab and read what the user is shown; the registry's own shape moved to the unit test where it costs no render. - accent-surface's pairs use `as const satisfies`, keeping their literal types per the repo's constant guidance. * test(api-keys): pin each installer's exact command, and make the tab tests selection-sensitive CodeRabbit reached the same conclusion the review did: asserting a command merely contains the token and the package name lets a builder swap its executable or move a flag to the wrong side of the `--` and stay green. That is exactly how the Gemini command shipped wrong (#6654). Each installer now has its full expected string pinned, across cloud and self-hosted and with and without a project id, plus a test that fails if an installer is added without pinning one. The tab tests were also passing vacuously: Claude Code is the default tab and also has a command, so "select Codex, expect a terminal heading" held whether or not the click did anything. They now start from a config-only assistant so the selection is what is actually measured, and the config-only case checks two different assistants so the message is shown to follow the selection. The command string itself cannot be asserted in the rendered dialog — ShikiCommandBox is `dynamic(..., { ssr: false })` and renders nothing in jsdom — so the exact strings live in the unit test and the integration tests prove which treatment the tab selects. Both files say so. * test(api-keys): cover the project-id and self-hosted flags independently The four pinned command cases paired a project id only with cloud mode and self-hosting only with no project id, so the combination where BOTH flags are present was never built. That is the densest line either builder emits, and Claude Code's is the awkward one: the project id goes before the `--` while the endpoint goes after the api key at the very end. Nothing checked it. Now a full 2x2 per assistant, driven by a combo table so the matrix is visible rather than implied, with every expected string still written out in full. Eight cases, up from four. The table is `as const` — it is a static fixture and nothing should mutate it.
…3178) (#5812) * refactor(experiment-runs): extract the shared deduped-items filter (#3178) The per-evaluator breakdown and the per-run cost summary in enrichRunsWithBreakdownAndCosts carried the same scope predicate and the same IN-tuple dedup subquery, differing only in projection and GROUP BY. The cost query's own comment pointed at the breakdown query for its rationale, which is the drift risk this ticket describes. Extract buildDedupedRunItemsWhere into clickhouse-experiment-run.queries.ts, alongside the OccurredAt range math it pairs with, and compose both queries around it. Option B from the ticket: the dedup key now exists once rather than in the six places it had to agree across the two queries. extraFilters narrow the outer read only. The dedup subquery has to see every version of a row to resolve max(OccurredAt), so pushing a filter into it would resurrect superseded rows; the helper makes that boundary explicit instead of leaving it implicit in two hand-written copies. The dedup OOM-safety guard matched source text, so extraction would have quietly narrowed it to the one remaining inline query. It now also asserts on the rendered filter: the max(OccurredAt) GROUP BY, the IN-tuple shape, tenant and OccurredAt bounds on both sides, and that extraFilters stay out of the subquery. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * style(experiment-runs): reformat the touched lines for the post-ADR-076 Biome config The restructure moved these files under a Biome config that formats an inline object type annotation and a long expect() chain differently. Both hunks are lines this branch introduced; origin/main is clean on both files, and the four complexity warnings on experiment-run.service.ts are unchanged at four. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: langwatch-agent <agent@langwatch.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Ubuntu <ubuntu@ip-10-0-3-183.eu-central-1.compute.internal>
…ders (#6606) * feat(auth): support AWS Cognito and OneLogin as self-hosted SSO providers Both speak OpenID Connect, so both are wired the way Okta already is: the operator supplies a client id, a client secret and the issuer, and every endpoint is read from the issuer's discovery document. For Cognito that discovery document is what carries the hosted-UI domain, so the operator never has to find and copy that domain separately, and the three COGNITO_* variables the app already validated are enough on their own. Cognito was previously advertised without working. The chart README documented six cognito values, _helpers.tpl validated them, and env-create.mjs accepted COGNITO_CLIENT_ID/SECRET/ISSUER, but values.yaml had no cognito key and the Deployment never emitted the variables, so setting NEXTAUTH_PROVIDER=cognito logged "cannot mount" and silently fell back to email mode. The chart now renders both providers, and a render test asserts what the container receives rather than what the template says. Verified against a real Cognito user pool on lw-dev. * docs(auth): correct the stale next.config.mjs reference on the pinned callback path The Next rewrite it named has not existed since the Vite migration. The path works because the genericOAuth plugin registers each config in ctx.socialProviders, which is what the core callback route resolves against. * style(auth): drop em dashes from the comments added here * feat(auth): route legacy callbacks for every generic-OAuth provider, and add a generic OIDC one Two review findings, both real. The legacy-callback rewrite table in api-router.ts only listed auth0 and okta, while cognito and onelogin pinned the same legacy path. That did not 404: the request fell through to the /api/auth/* catch-all and better-auth's core social callback picked the provider out of ctx.socialProviders, so sign-in worked while quietly taking a different code path from the two that are rewritten. The table is now derived from the provider list itself, so the two halves cannot drift, and legacyCallbackParity fails if they do. A reviewer pointed out they had already run LangWatch against their own OIDC identity provider by claiming it was auth0. Naming a provider should buy documented setup steps, not capability, so `oidc` joins the list: any issuer publishing a discovery document now works without pretending to be something else. The named entries stay because operators look for their provider by name and Cognito's issuer-vs-hosted-domain distinction is worth spelling out. Also from review: the chart test renders once per flag set instead of eight times and surfaces helm's error text, its provider count was wrong and its third case had no scenario binding, the duplicated auth0 @PARAM annotations are gone, and the .env.example provider comments are complete sentences. Verified against the real Cognito pool through the rewritten path, and again with NEXTAUTH_PROVIDER=oidc pointed at that same pool. * test(auth): assert the PKCE challenge reaches the identity provider The config carries pkce: true and the unit test checks the flag, but only the emitted authorization URL shows whether it arrived, and the integration suite is the only place that looks at that URL. * fix(auth): a trailing slash on NEXTAUTH_URL no longer breaks the redirect URL Identity providers compare redirect URLs by exact string, so a deployment URL written as `https://host/` produced `https://host//api/auth/callback/cognito`, which does not match the `https://host/api/auth/callback/cognito` the operator registered, and the provider refuses the sign-in. NEXTAUTH_URL is written by hand in a values file or an env var, which is exactly where a trailing slash comes from. Verified the new unit test fails without the fix rather than passing vacuously. Along with it, from the same review pass: - The parity test read the router source through an inline `import()`, which the repo bans outside the SDK CLI boot path. - The env schema comment said the generic provider is "configured by issuer alone" directly above the three variables it needs. - The docs said a discovery document returning `authorization_endpoint` meant the issuer was correct, which reads as a compatibility check when it only proves the issuer resolves. They now name the three endpoints LangWatch reads and the flow the client has to allow. - Two comments in the chart test counted providers, so adding one made them wrong. They no longer count. - The comments in `auth-client.tsx` listed the generic-OAuth providers by name, which is the same drift that put the legacy-callback rewrite out of sync. Generic OAuth is the fall-through there on purpose, so a new OIDC provider links correctly without being listed, and the comments now say that instead of enumerating. * test(auth): prove the callback URL an identity provider actually receives The unit test pins what the provider config carries, but an identity provider only ever sees the emitted authorization URL, and the redirect it compares against its registration is a query parameter on that URL. This asserts it there, through real BetterAuth, which is the only place the value can be observed the way the provider sees it. Verified it fails without the normalization: `redirect_uri` comes through as `http://localhost:5624//api/auth/callback/cognito`. * test(auth): gate every OIDC provider from the table, not a copy of it The license-gate tests listed cognito and onelogin by hand, so the generic oidc provider added later was never gated by them. The gate is provider-agnostic, so nothing was broken, but these tests are what would notice if that ever stopped being true for one provider, and one of them was outside their reach. They now take the list from PLAIN_OIDC_PROVIDERS, the same source the legacy-callback rewrites take it from, so a provider added there is covered without anyone remembering to come back. * refactor(auth): stop restating the provider list where it can go stale Two more copies of it, found by sweeping the diff for the drift shape that produced the legacy-callback bug: a doc comment on `buildGenericOAuthConfigs` enumerating provider ids, written before the generic one existed and so missing it, and a hardcoded triple in the callback-URL test. Both now come from `PLAIN_OIDC_PROVIDERS`. Places that legitimately enumerate providers, the chart templates, the env example and the docs, are left alone: there the list is the content, not a copy of it. * fix(auth): say on screen when an identity provider could not be started A licensed deployment that mistyped its provider name or left a client secret unset lands in email mode, which is the no-lockout guarantee working as designed, but nothing on screen said so. The gate logs it once at startup and the sign-in page looks identical to a deployment that never configured single sign-on, so an operator could believe federation was being enforced when it was not. The authentication settings page already explains the other cause of the same symptom, a deployment that is configured but unlicensed, with the reasoning that nobody reads server logs to explain a login screen. That reasoning covers this case too, so it now reports both. They are kept apart because they are fixed in different places: one by activating a license, the other by correcting the provider name or its credentials. Verified both new assertions fail when the notice is restricted back to the license case. This deliberately does not change what happens at sign-in. Falling back to email on a mount failure is a bound guarantee, and failing startup instead would lock every user out of a licensed install over one typo.
… TTL (#6680) * docs: correct the Monitor poll-cadence rule for the real prompt-cache TTL The rule assumed a flat 5 minute prompt-cache TTL and capped every poll cycle at 4.5 minutes on that basis. Measured across local transcripts over the last 24 hours, main sessions get a 1 hour TTL (44653 nonzero 1h cache writes vs 24 5m) while subagent sessions get 5 minutes exclusively (13280 nonzero 5m, zero 1h). Cap on the TTL that actually applies: ~15 min in a main session, 4.5 min inside a subagent. * docs: mark the cache-TTL split as measured, not contractual The split is observed harness behaviour over 414k local API calls, and it has flipped before: subagents ran at 1h as recently as 2026-02 (n=4142 on opus-4-6, 95.2% 1h) before moving to 5m. Date the measurement and say what to do if it stops holding. * docs: trim the cache-TTL measurement caveat
… updates (#6682) Bumps the github-actions group with 12 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/github-script](https://github.com/actions/github-script) | `7.1.0` | `9.0.0` | | [actions/checkout](https://github.com/actions/checkout) | `7.0.0` | `7.0.1` | | [pnpm/action-setup](https://github.com/pnpm/action-setup) | `6.0.9` | `6.0.10` | | [actions/setup-node](https://github.com/actions/setup-node) | `6.4.0` | `7.0.0` | | [actions/setup-go](https://github.com/actions/setup-go) | `6.5.0` | `7.0.0` | | [actions/setup-python](https://github.com/actions/setup-python) | `6.3.0` | `7.0.0` | | [github/codeql-action/init](https://github.com/github/codeql-action) | `4.37.0` | `4.37.6` | | [github/codeql-action/analyze](https://github.com/github/codeql-action) | `4.37.0` | `4.37.6` | | [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) | `6.2.2` | `6.2.3` | | [docker/login-action](https://github.com/docker/login-action) | `4.4.0` | `4.6.0` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `8.3.2` | `9.0.0` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.14.0` | `1.14.2` | Updates `actions/github-script` from 7.1.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v7.1.0...3a2844b) Updates `actions/checkout` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@9c091bb...3d3c42e) Updates `pnpm/action-setup` from 6.0.9 to 6.0.10 - [Release notes](https://github.com/pnpm/action-setup/releases) - [Commits](pnpm/action-setup@0ebf471...0977fd9) Updates `actions/setup-node` from 6.4.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@48b55a0...8207627) Updates `actions/setup-go` from 6.5.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@v6.5.0...b7ad1da) Updates `actions/setup-python` from 6.3.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@ece7cb0...5fda3b9) Updates `github/codeql-action/init` from 4.37.0 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@99df26d...5595cca) Updates `github/codeql-action/analyze` from 4.37.0 to 4.37.6 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@99df26d...5595cca) Updates `aws-actions/configure-aws-credentials` from 6.2.2 to 6.2.3 - [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases) - [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md) - [Commits](aws-actions/configure-aws-credentials@517a711...e6de054) Updates `docker/login-action` from 4.4.0 to 4.6.0 - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@af1e73f...dbcb813) Updates `astral-sh/setup-uv` from 8.3.2 to 9.0.0 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@11f9893...c771a70) Updates `pypa/gh-action-pypi-publish` from 1.14.0 to 1.14.2 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@cef2210...dc37677) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: pnpm/action-setup dependency-version: 6.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: actions/setup-node dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-go dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/setup-python dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: github/codeql-action/init dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: github/codeql-action/analyze dependency-version: 4.37.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: aws-actions/configure-aws-credentials dependency-version: 6.2.3 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: docker/login-action dependency-version: 4.6.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-actions - dependency-name: astral-sh/setup-uv dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…or (#7080) * perf(ci): run integration tests in the lane their dependencies call for The integration matrix took 12-13 minutes a shard, and measuring where that time went contradicted the obvious explanation. It is not slow tests: per-file durations on shard 4 are two outliers (90s, 40s) and then a cliff, with everything else between 3 and 13 seconds. It is per-file fixed cost multiplied by 1024 files, plus a lane that provisions three datastores for tests that never open a socket. Measured across the six shards of run 31977691802: import 1,664s 43% tests 1,408s 37% environment 381s 10% setup files 261s 7% transform 134s 3% Loading modules cost more than running the tests. SPLIT THE LANE. 546 of the 1024 `.integration.test.*` files declare jsdom and name no database, queue or cache. They now run on `test-component`: no service containers, no Prisma migration, no goose, no ClickHouse replay, no Helm, files concurrent, module registry shared — the unit lane's configuration, which is what those files always wanted. The datastore lane keeps the other 478 and drops from six shards to four. Both configs derive their file list from partitionIntegrationFiles, so the lanes are a total and disjoint cover by construction rather than by a list someone has to maintain. Default is the datastore lane, so a new test is never silently run without what it needs. REUSE THE MODULE REGISTRY. `isolate: false` on the datastore lane. This is not the concurrency knob that was tried and reverted: `fileParallelism` stays off, files still run strictly one at a time, and only the per-file rebuild of the module graph goes away. The unit lane has run this way across 1,688 files. STOP REBUILDING THE NLPGO BINARY. The cache was restoring correctly and the build ran anyway, every time, because staleness was decided by mtime — and git records none, so checkout stamps every source with the current run's time while the restored binary carries the time it was built in an earlier one. Every source looked newer than every cached binary. Now a stamp beside the binary records the digest of the sources it was compiled from. BALANCE BY MEASURED COST. The sequencer weighed file size and the spread showed it: 547-766s across near-identical file counts, with the matrix paying 766. Weights now come from a committed manifest, falling back to size for files it does not know. Committed rather than cached because every shard computes the split independently and must reach the identical answer. Refresh by dispatching the workflow and committing the artifact. CACHE THE GENERATED FILES. `start:prepare:files` is 30-35s of pure function of committed inputs, paid by every job. Now cached on a hash of those inputs, with a verifier so a key that stops covering a generator fails at the step that owns it instead of as a missing module minutes later. CUT THE JOBS PER PUSH. One push fans out to 27 runs and 98 jobs against a Team-plan allowance of the same order, which is why shards were measured waiting 286-590s for a runner while their only dependency finished in 8 seconds. Draft gating goes from 8 workflows to 18 via a `heavy` output on the shared detect-changes action, and CodeQL's matrix now follows the languages that actually changed instead of running every analyser whenever any of six trees is touched. Verified: 546+478=1024 with zero overlap on the real tree; 162 component-lane files (1336 tests) pass with no datastore; 50 new unit tests; all three specs fully bound; actionlint and biome clean; scoped typecheck of the changed configs clean. Note: `src/pages/[project]/` made this nearly a silent disaster — handed to a glob engine unescaped, `[project]` is a character class, and those twelve files would have been selected by neither lane while both reported a clean pass. Include patterns are escaped, with a test. * fix(navigation): wait on a drawer warm-up promise from any realm The component lane's first CI run failed one test: a warmed drawer rendered its spinner instead of opening at once. It reproduced locally under `pool: vmForks` and passed under `pool: forks`, which named the cause. `primeLazyComponent` settles a drawer's ready-state by reading its `lazy()` wrapper outside render. A wrapper that is not ready reports that by THROWING the promise it is waiting on, so the warm-up has to recognise a promise to know it must wait — and it asked `pending instanceof Promise`. `instanceof` asks which realm made the value, not whether it behaves like a promise. A browser has one realm, so this held everywhere it had ever run. Give each test file its own VM context and the thrown promise comes from the other realm, fails the check, and the warm-up returns WITHOUT waiting: the drawer is reported warm while still pending, and paints the spinner after all. Duck-type the thenable instead. `then` is what React itself looks for and what the promise contract specifies; realm identity was never the question being asked. Same class of bug as the `dedupe: ["zod"]` note in CLAUDE.md. Not a production bug — one realm in a browser — but the check was wrong on its own terms, and it was only ever going to be found by something that runs the code in more than one place. Pinned by a unit test that builds a genuinely foreign promise with `runInNewContext` and asserts both halves: that `instanceof` rejects it, and that it is awaited anyway. * fix(ci): keep the component lane on vmForks, and quarantine the location mockers Three corrections from the first CI run and a full local pass of the lane. TYPE ERROR. `isThenable` narrows to PromiseLike, and `PromiseLike.then` returns PromiseLike — not the `Promise<void>` primeLazyComponent declares. Adopt the foreign thenable with `Promise.resolve()`, which is both what the signature wants and the right semantics: a bare PromiseLike carries no catch/finally. POOL. `pool: "forks"` looked right on the reasoning that these files ran on forks under the integration config for their whole history, so a move should not change the environment. Measured, that is backwards: forks + isolate:false failed 87 of 236 files in src/components, against 4 under vmForks. They ran on forks with isolate:TRUE — a fresh registry per file. Reusing one registry inside a plain process is what leaks module state between files; a VM context is what makes reuse survivable, and it is why the unit lane runs 1,688 files that way. The two settings are a pair, not independent knobs. Reverted, with the measurement written down so the next person does not retry it. WINDOW.LOCATION. Five files replace `window.location` wholesale via defineProperty. Measured directly in a VM realm, jsdom defines it as a NON-CONFIGURABLE ACCESSOR: it cannot be deleted ("Cannot delete property 'location'") and cannot be redefined ("Cannot redefine property: location"). The setter means assignment still works; replacing the object never will. That is incompatible with a VM context wherever it runs — these files would fail the same way on the unit lane — so it is not something the split caused. They are held in the datastore lane by a named list that says exactly why and how to convert them, so 541 files move while the incompatibility stays attributed. Also fixed, both real bugs on their own terms: - test-setup.ts's matchMedia polyfill omitted `configurable`, so a test could assign it but never redefine it — which is how you swap matchMedia to simulate prefers-reduced-motion. A stand-in for a missing browser API has no business being less replaceable than the API it imitates. Local state: 453 of 455 component-lane files pass (3,201 tests). The two that fail pass in isolation, so they are order-dependent under the shared registry — the documented isolate:false risk, and the reason CI's own sharding is the place to judge it rather than a local run with different file ordering. `pnpm typecheck` is clean; `typecheck:tests` was OOM-killed locally twice with no diagnostics, so CI is the authority on that one. * refactor(navigation): put full-page navigation behind a seam, and delete the quarantine Replaces the "hold these five files back" list with the fix it was standing in for. The list is gone; all five run in the component lane. The five replaced `window.location` wholesale to observe a navigation. Measured in a jsdom VM realm, every way of doing that fails: Object.defineProperty(window, "location", ...) -> Cannot redefine property vi.spyOn(window, "location", "get") -> Cannot redefine property vi.spyOn(window.location, "reload") -> Cannot redefine property vi.stubGlobal("location", ...) -> Cannot redefine property `location` is a non-configurable ACCESSOR and its methods are non-configurable and non-writable. Assigning `href` is the one thing that works, and jsdom then logs "Not implemented: navigation" and records nothing — so a test cannot observe a navigation a component performs directly, by any means. The tests that appeared to were relying on a pool where jsdom left `location` replaceable. So the fix is a seam, not a workaround: src/utils/browserNavigation.ts exports hardNavigate / replaceLocation / reloadPage, the three call sites use it (auth/error, auth/signin, LegacyTracesDeprecationBanner), and the tests `vi.mock` the module and assert the call. A module import is substitutable in any environment, which a global accessor is not. It is better production code on its own terms too — a component writing `window.location.href` directly is untestable by construction. LangyEvalRunCard needed none of that: it only READS `location.origin`, so it now declares the document's URL with `@vitest-environment-options` and reads the real origin. A true URL is a better fixture than a stand-in object. useLicenseActions keeps its regression guard. The hook must NOT reload — a reload used to tear the restart toast off the screen — and the guard now asserts `reloadPage` was not called rather than spying on an unspyable global. Also from review: the draft gate in detect-changes checked only `pull_request`, so a draft `pull_request_target` still ran everything heavy. Both event names carry the same payload; both are checked now. Verified: all ten files across the five previously-quarantined suites pass in the component lane (45 tests); partition back to 546/478 of 1024 with no overlap; biome clean. * fix(ci): isolate the component lane, so a passing run means the same thing twice Measured over the full 546-file lane, twice each way: isolate: false -> 545/546. A DIFFERENT file fails each run — GlobalUpgradeModal and LangyComposerRecordedTurn one run, SavedChartsToolbar the next. All pass in isolation. isolate: true -> 546/546, 3,987 tests, no failures. The files this lane took are React component suites, and a shared module registry turns "has this lazy chunk resolved yet" and "what is in this zustand store" into global state that the FILE ORDER decides. The result is a rotating one-file flake: the most expensive kind of red, because it accuses an innocent file and does not reproduce when you run it. Registry reuse came in by inheritance from the unit lane, not by choice, and nothing about this lane's purpose needs it. The win is not booting a Postgres, a ClickHouse and a Redis, not running two migrations and a Helm setup, and running the files concurrently rather than one at a time. All of that stands. A deterministic suite is worth more than the import time reuse saves. The pool stays inherited (vmForks), and the note explaining why is kept: forks with a shared registry was measured worse still, 87 of 236 files failing. * fix(ci): address review — CodeQL gate, cache key, empty manifest, draft e2e Seven findings from review, six of them real bugs. CODEQL COULD SKIP ITSELF. `relevant` is the outer gate, and it had drifted from the language filters in both directions: `packages/**` was in codeql-javascript but NOT in `relevant`, so a packages-only change skipped CodeQL entirely; and `sdks/go/**` was in `relevant` with no analyser to match, so a Go-only change produced an empty matrix and skipped it too. Both read green. `relevant` is now exactly the union of the language filters, with a comment saying it must stay that way, and `sdks/go/**` is dropped — this workflow analyses javascript-typescript and python, so a Go path could never produce a leg. CACHE KEY MISSED THE TOOLCHAIN. prepare-generated-files hashed the generators' INPUTS but not the generators. Bump Prisma or the SDK bundler with no source change and every hashed file is identical, so the cache would restore artifacts from the previous toolchain and skip the regeneration the bump existed for. `pnpm-lock.yaml` is in the key now. THE VERIFIER PASSED ON EMPTY DIRECTORIES. `require_dir` cannot tell a restored directory from a complete one, so a partial cache cleared the step and failed minutes later inside a test worker — the exact thing the script exists to pre-empt. It now names the entrypoints the packages' own main/exports/bin fields point at. THE MANIFEST JOB COULD PUBLISH `{}`. If every test job skips or dies before its upload, the merge produced an empty object and the job published it as an artifact the instructions tell a human to commit — which would delete every weight in the repo. It fails on a zero count instead: an absent manifest is a safe state, a blank one is not. The collision comment was also wrong and now says the real reason `add` is safe (the reporter merges over the committed manifest, so keys collide with identical values). COMPONENT COVERAGE WAS COMPUTED AND DISCARDED. Both sibling lanes upload to Codecov; this one paid ~45% instrumentation on schedule and dispatch for output nobody read. It uploads under a `component` flag now. DRAFT PRs STILL RAN THE SDK E2E. sdk-javascript-ci deferred `ci` on drafts but not `e2e`, which boots the application and three datastores — the most expensive job in that workflow. Gated on `heavy`. Counts corrected to 546 throughout (548 was the jsdom-docblock count, not the partition), and the stale "reused module registry" descriptions of the component lane updated to match the isolation fix. Not changed: the missing `--reporter=json --outputFile=test-results.json` on the component lane. `test-unit` and `test-integration` do not pass it either, and extract-failures.sh handles the missing file by design, so the extract step is already a no-op in all three lanes. Fixing one lane would make them inconsistent; it belongs in its own change. * fix(ci): keep isolation on in the datastore lane too The full CI matrix returned the verdict: with `isolate: false`, three of four integration shards went red, and shard 2 alone failed 30 of its 120 files. The errors name the cause rather than hinting at it — "Cannot resolve ClickHouse client", "App not initialized", ECONNREFUSED. These suites build and tear down an application container per file, and that container lives in module scope. Share one registry across files and the first file's teardown takes the next file's client with it. That is not the `fileParallelism` hazard — nothing here runs at once — but it has the same root: this suite keeps real per-file lifecycle state in module scope. The unit and component lanes can share a registry precisely because they build no containers. So R2 comes out. It was the second-largest projected win (1,664s of import against 1,408s of test execution, 43% of integration runner time), and it is simply not available while a fresh module graph is what gives each file its container. Reclaiming it means giving the app container an explicit reset between files — a real change to the harness, and its own PR, where the failures it causes are the subject rather than collateral. What the measurement leaves standing, and what CI has now confirmed: - test-component (1) and (2) both PASS, ~5 min each, with no service containers, no migrations, no goose and no Helm. The lane split works. - The datastore lane carries 478 files instead of 1024, on 4 shards not 6. Both lanes now run isolated, which is also the honest resolution of the component lane's rotating flake: this suite is not ready for a shared registry, in either lane, and saying so is worth more than the import time. * fix(ci): emit per-shard duration deltas, so a fresh measurement cannot be lost The reporter merged its measurements over the committed manifest and emitted the whole thing, which put the full baseline in every shard's artifact. For a file measured by shard A, A's artifact then held the NEW value and every other shard's held the OLD one — and `jq -s add` lets the last input win, so whether the fresh measurement survived came down to the order `find` happened to list the artifacts in. Silently, and green. My earlier note claimed the colliding keys "hold the same committed value in every artifact". That is true only for files nobody measured; it is exactly wrong for the files the refresh existed to update. Each shard now emits ONLY what it measured, to its own gitignored file, and the aggregation lays the deltas over the committed manifest — baseline first, deltas on top. Deltas cannot fight each other: the lanes partition the files and a lane's shards partition its own, so their union is the same in any order, and the baseline is applied once underneath. Writing to its own file matters as much as the delta. A manifest is a COMMITTED artifact, and writing a partial one over vitest.durations.json would delete the weights for every file the run did not execute — which is what a developer running a single lane locally would have done. Also rebased onto main (TypeScript 7 workspace move, #7081). Two conflicts: CLAUDE.md, where main rewrote the check-queue paragraph this change inserts next to and both belong; and pnpm-lock.yaml, taken from main and regenerated so the picomatch devDependency is recorded rather than silently dropped — a lockfile resolved by hand would have failed --frozen-lockfile in CI. * fix(ci): address review — go build inputs, realm test, behaviour-level specs - Hash every go build input. watchDirs missed the root go.mod/go.sum/go.work and sdks/go, which the engine imports through a `replace` — so a dependency bump or an SDK change reused a stale binary while the digest claimed to cover 'every Go build input'. Both are watched now, with regression tests. - The realm test never drove the code it guarded: it awaited a foreign promise directly, so it passed whether or not primeLazyComponent recognised one. It now drives primeLazyComponent with a wrapper that reports itself pending, and asserts the warm-up stays unfinished until that promise settles. - Both feature files described internals — digests, stamps, mtimes, realms, instanceof. Rewritten as observable behaviour per the house rule, with the mechanics left in the unit tests where they belong. - Removed readManifestText, which nothing called. * chore(ci): refresh the lane counts after the legacy trace UI removal main's #6902 removed seven integration files with the legacy trace UI, so the partition is 540 component / 477 datastore of 1017. Still exact, still no overlap — the numbers in the comments are a measurement, and this is the current one.
…7093) userId, organizationId and projectId go into log lines unredacted. That was already true in practice -- the log context mixin has always injected them -- but nothing said so, and two linters said the opposite. So the rule got relitigated one review comment at a time, and the answer depended on who was reading. They are opaque internal identifiers, not personal data. What they are is the only thing that makes a log line attributable: they are how an incident gets filtered to one tenant, and how a support question becomes a query instead of a guess. Redacting them costs that and protects nothing, because the identifier was never the sensitive part. States it in logging-and-tracing.md, with what genuinely must never be logged (credentials, customer content, personal data proper) kept separate from it so the distinction is the readable part. Then makes the tooling agree: - semgrep's pii-in-logger-call drops its two userId patterns. This was the binding one -- it runs in CI, not just in review. - .coderabbit.yaml stops listing userId and organizationId as fail conditions and says why, so the bot has the reason rather than just the exception. The semgrep fixture keeps userId, organizationId and projectId as `ok:` cases, so re-adding an identifier pattern fails the config-check rather than quietly reinstating the false positive. That check asserts an exact fixture match count, which drops 5 -> 4 with the userId case removed. projectSlug stays flagged: a slug is a human-readable name, which is a different question from an opaque id and not one this change answers.
* perf(gateway): bound the standard provider worker pools GetConfiguredProviders advertises the whole bifrost standard provider list, because a virtual key may name any of them and bifrost resolves provider config by provider key alone. Only the URL-derived compat endpoints set ConcurrencyAndBufferSize, so every standard provider fell through to CheckAndSetDefaults and took bifrost's own 1000 workers with a 5000-slot queue -- sized for a deployment where one provider fronts the whole gateway, and paid 23 times over here. Production profiles put that at ~21,000 permanently parked bifrost.requestWorker goroutines per pod, 99.85% of every goroutine in the process, on both the gateway and nlp (which reaches the same code through the aigateway dispatcher). They served no traffic; their only measurable effect was the GC rescanning 21,000 stacks on every mark cycle and the profiler serializing them on every 15s upload. Together those were ~86% of the service's CPU and 58% of its heap allocation. Sizes them at the figure the compat path already settled on, for the same reason: the pool bounds in-flight upstream requests, and a burst past it queues rather than fails, since the gateway leaves DropExcessRequests off. The guard walks every advertised provider rather than sampling one, because the list grows on a bifrost upgrade without anyone here touching a file. * perf(observability): memoise loggers by name createLogger built a fresh pino instance on every call. It has 400+ call sites, many of them per-instance class fields and a few inline in catch blocks that construct a whole logger to emit one line, so production profiles put it at 2.3% of the app's wall time -- nearly a quarter of that inside pino's getCallers, which captures a stack trace on every construction to work out who called it. Measured over a window that excludes boot, so this is steady-state work, not startup. Sharing an instance is safe because nothing request-scoped is bound at construction. Name, service and service.version are process-wide; the per-request fields arrive through the mixin, which pino invokes on every log call and which reads the async-local context at that moment. The transport was already shared for the same reason. The tests assert that property against records the logger actually wrote, rather than against the implementation, since it is the whole reason this is allowed. disableContext is part of the key: otherwise whichever caller arrived first would decide, silently, whether every later line under that name carried its request fields. resetLoggerCache exists for tests that mutate the environment a logger reads at construction between cases. Production reads it once at boot. * perf(redaction): skip recognizers whose required literal is absent The PII recognizers are unanchored and scanned with matchAll from every starting position. On text that cannot match they still pay for the scan, and the email pattern is the expensive case: [A-Za-z0-9._%+-]+ consumes a long alphanumeric run, fails to find @, and backs off a character at a time. Production profiles put that one pattern at 2.6% of the worker's wall time, inside a redaction pass that is 7.4% in total. Three patterns cannot match without a literal that is visible in the pattern itself -- @, 0x, and : -- so they are skipped on text that does not contain it. String.includes is a native substring scan and settles the same question in one pass. This only ever removes a scan that would have found nothing, so no span that was redacted before stops being redacted. Deliberately not extended to the phone pass. libphonenumber's candidate pattern admits a single digit and MIN_LENGTH_FOR_NSN is 2, so there is no cheap precondition that provably covers everything it could match, and a prefilter that is merely almost right stops redacting real phone numbers. That pass stays at 3.3% rather than risk it. Getting an annotation wrong fails open, so the tests are built around that rather than around the happy path: every sample in the mis-annotation case is personal data of another kind, written to contain none of the claimed literals, and the fixtures assert their own emptiness so a sample that accidentally carried one cannot pass vacuously. * docs(observability): document trace sampling and report it at boot Nothing configures a sampler, so the fleet runs OpenTelemetry's default parentbased_always_on and records and exports every span. The SDK reads OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG itself, and NodeSDK only overrides the sampler when one is passed explicitly, so those variables already work end-to-end and no code was needed to make them. What was missing is that anyone could tell. An unset variable and a misspelled one produce the same silence and the same full-rate export, and the cost lands on the collector and on trace storage rather than anywhere an app owner would look. One line at boot answers "is this fleet sampling" without reading a chart. Documents what sampling is and is not for, since the obvious assumption is wrong: span creation is ~1.4% of the app's wall time and a sampled-out span still pays the context propagation around it, so this is a lever on collector and storage cost, not on application CPU. No default is changed here -- how much of our own tracing to keep is a deployment decision. * perf(ops): stop re-validating an unchanged ops snapshot on every read The live and detail readers each poll one fixed Redis key on an interval, so between writes they hand the byte-identical JSON string to the same schema over and over. Re-running the full schema parse each time was 2.7% of the app's wall time in production -- the single largest zod cost in the process, 84% of it inside zod itself -- spent proving that our own serializer works. Caches the last successful parse per schema and returns it when the raw string is unchanged. This keeps the validation rather than trading it away for a version check: a string that has not changed cannot describe a shape different from the one already checked, so nothing is now trusted that was not verified before. Bounded by construction -- one entry per schema, and the schemas are module constants. Only successes are stored, so an unreadable snapshot behaves exactly as it did and a key that is later rewritten correctly is picked up. The cached value is shared, so callers must treat it as read-only. mergeSnapshots projects it into a fresh object rather than mutating it, which is the arrangement this relies on and the reason it is safe here. The risk this buys is staleness, and staleness here is invisible: a reader that missed a rewritten snapshot would hold the dashboard on old numbers while the platform moved underneath it, reading as a quiet platform rather than a broken reader. That case is pinned by its own test, which fails (along with five others) if the cache is made to always hit.
…home (#7085) * feat(ops): give the event-sourcing page a sidebar and dead letters a home The page stacked four independently dense sections — projections, subscribers, processes, schedules — on one scroll, which meant an operator mid-incident scrolled past three healthy subsystems to reach the one that was wrong. ops-dashboard.md's rule is that space is proportional to trouble, and one page cannot honour that for four subsystems at once. Each section is now a route under a local sidebar, the same shape /ai-gateway uses. The main sidebar is untouched: it already matched /ops/event-sourcing by prefix, so this is a sub-nav inside one entry rather than five new top-level links. The Overview leads with what is wrong and links to it. Dead letters get the treatment they were missing entirely. The fleet table reported a dead COUNT and nothing else, and the only read that returns messages needs a full process ref — name, project AND process key — so a dead message could only be reached by an operator who already knew where it was. A number nobody can act on is an unfinished feature. - `findDeadMessages` / `countDeadByProcessName` read the whole fleet, newest retirement first, each row carrying the ref needed to redrive it straight from the list. - A Dead Letters route lists them, filters by process, expands to payload, and redrives without opening an instance first. - The dashboard's existing DLQ card now also reports the process outbox, so "what has stopped?" is one heading rather than two mechanisms an operator has to know apart. The outbox row does not record WHY a message died — the dispatcher puts that on the span and the log line — so each row carries its trace id and the reason stays one hop away rather than duplicated into Postgres. * fix(ops): read dead letters through raw SQL and drop high-entropy fixtures Two CI failures. The multitenancy guard rejects any Prisma query on ProcessManagerOutbox without a projectId, and a dead-letter sweep has no single project to name — it is cross-tenant by definition. Both reads now go through raw SQL with the same explicit `-- @tenancy` marker every other fleet-wide read in this repository already carries. Every returned row still states its own project, and every write stays on a guarded query. The secret scanner read the test fixtures' realistic ULID and trace id as generic API keys, on entropy. They are now repeating patterns, which is the same reasoning as the sequential-hex HMAC fixture in the spend-ingest suite, and the lengths still exercise the middle elision. * fix(ops): address review — loading gate, keyboard access, and invalid markup Three findings from the review, all real: - The DLQ card now renders for either source, but its loading gate and its heading still assumed queue groups. It waited only on the queue query, so the process-outbox row flipped in a moment after mount — which on an ops surface reads as a new incident rather than a page finishing load — and the heading printed "0 groups" above a red count. Both gate on both sources now. - The dead-letter row expander was mouse-only: no tabIndex, no key handler, no aria-expanded. The expanded region holds the trace id, which is this page's only route to WHY a message died, so the diagnosis sat behind a pointer. It is a button now, in the accessibility sense. - A Chakra Text inside the filter chips renders a <p>, which a <button> may not contain; the browser closes the button early and the chip stops being one control. Rendered as a span. Not taken: the suggestion to give `openDrawer` an object parameter. It is an existing API with 185 positional call sites, so changing it here alone would make this the odd one out. * fix(ops): stop the redrive test mutating shared fixtures, and scope row keys Three more from the review. The redrive case flipped a row out of `dead` that four other assertions counted and ordered on, and the memoised seed never restored it. It only passed because Vitest happened to run the blocks in declaration order — a retry, a `.only`, or one more case after it would have broken the others. It seeds and redrives its own row now, under its own process name. The keyboard handler I added for the row expander also fired for Enter or Space on the nested Redrive button, because the event bubbles: one press would have redriven the message AND toggled the row. It now ignores anything that did not originate on the row itself. On indexes for the dead-letter reads: not added, and the reasoning is written where the queries are. The existing index leads with `status`, so it narrows to the dead rows; the unindexed part is the sort over that subset, and the dead population is bounded by operator attention rather than traffic. If that ever stops being true, the schema already states the answer for this table — CREATE INDEX CONCURRENTLY from the runbook, never a deploy-time migration that locks the highest-volume write path.
…work is subscribers and process managers (ADR-098) (#6956) * refactor(event-sourcing): custom evaluation sync is a subscriber, and the dead Customer.io reactors are gone customEvaluationSync moves from withReactor to a fold-bound withSubscriber on traceSummary, keeping its registration name so jobs staged before a deploy dispatch into the new registration, and keeping the reactor-era delay/ttl/dedup semantics byte-for-byte. The three Customer.io reactors (trace, evaluation, simulation) were implemented but never registered — the counting strategy they waited on never landed — so they are deleted rather than migrated. * docs(adr): retire the reactor vocabulary — post-event work is subscribers and process managers (ADR-094) The spec supersedes reactors.feature and, unlike it, is tagged — the old file's scenarios reported bound while binding nothing. Test bindings land with the conversions. * refactor(event-sourcing): every reactor is a subscriber declaration on its pipeline The subscriber spec grows what the conversions needed, none of it changing existing registrations: a state-aware `when` (the committed fold state is in hand at guard time, so fold-dependent guards keep rejecting pre-enqueue), state-aware dedup/group keys, runIn/disabled pass-through, an event-shaped throttledWindow helper, and dedup only when asked for — a spec without dedup/ttl means every event dispatches its own job, which lifecycle syncs like suiteRunSync depend on. Batch collapse and queue dedup now share one key function, the throttleWindow doctrine applied to the sugar. Trace, simulation, coding-agent, governance (EE) and the global billing dispatch all keep their registration names, so queue jobs staged before the deploy dispatch into the new registrations. * refactor(event-sourcing): drop withReactor — the builder's one reaction primitive is withSubscriber The remaining callers were tests composing noop registrations; they compose noop subscriber specs now. throttledPerWindow's payload-shaped variant keeps its tests until the internal rename retires it. * fix(event-sourcing): reconcile the retirement with the simulation process-manager migration The simulation pipeline keeps the migration branch's shape wholesale — the execution process manager, its subscriber specs, and the revived Customer.io simulation subscriber. The CRM debounce constant moves into that subscriber, since its old home (the trace-side Customer.io reactor) is deleted. The cross-cutting throttle-policy test reads registrations off the real pipelines now that the throttle lives on the pipeline declaration. * docs(spec): bind post-event work, renumber the retirement ADR past the collision, and keep the Customer.io family migrated The retirement ADR takes 095 — the simulation process-manager migration already shipped as 094 on its branch. post-event-work.feature binds 12/12 tagged scenarios (role gating declared @unimplemented — only config presence has a test today), and the stale reactors.feature exemption leaves the parity list. The never-registered Customer.io trace and evaluation reactors come back as subscriber specs rather than deletions — the nurturing spec's scenarios are enforced and the simulation sibling is actively maintained — offered as optional pipeline deps and still unregistered pending the counting strategy. * fix(ci): restore the generated files main committed, and drop the app's dead scenarios references An early sweep committed langySkills/evaluators regeneration drift, which is what biome's format gate was failing on. The two app.ts lines referenced the scenarios dependency the simulation migration removed; nothing consumed the field server-side. * fix(ci): clear the error-level biome backlog and the base branch's test-type errors The nine biome errors were organize-imports and one unused import in this PR's own files, hidden below the display cap behind three and a half thousand warnings. The five test-type errors were the simulation migration's — an unexported view type, two casts tsgo refuses, a null into an optional string, and a vi.fn that cannot satisfy an intersection the spread widens — surfaced here because this stack runs typecheck:tests first. * chore: retrigger CI * refactor(event-sourcing): pay down the complexity the retirement's file moves re-exposed The biome delta gate counts violations per (file, rule), so a renamed file re-presents its whole pre-existing backlog as new. Rather than an override list, the flagged handlers get the decomposition the simulation Customer.io subscriber already models: guard helpers with the incident context on their doc comments, fire-and-forget CRM steps as named functions, payload builders out of dispatch loops, and the subscriber sugar split into compile helpers. Net: the branch now removes twenty violations and adds none. * docs(adr): renumber the retirement ADR to 098, past main's tsgo-governor collision * refactor(event-sourcing): retire the reactor vocabulary from the internal names too The retirement stopped at the public surface: `withReactor` was gone, every `*.reactor.ts` file was gone, but the machinery underneath still called itself reactors — `ReactorDefinition`, `foldReactors`, `reactorsForFold`, a whole `reactors/` directory. A reader arriving at the dispatch plane still had to learn the retired word to follow it. The dispatch-plane types now say what they are: `SubscriberDispatchDefinition`, `SubscriberDispatchContext`, `SubscriberDispatchOptions`, and `shouldReact` becomes `shouldDispatch`. `reactors/reactor.types.ts` merges into the existing `subscribers/` directory as `subscriber.types.ts`. Two families shared one name once the rename landed, so they are now spelled apart: the pipeline-level live event consumers keep `subscriber*` (`initializeSubscriberQueues`), and the fold/map-attached work becomes `projectionSubscriber*` (`initializeProjectionSubscriberQueues`). The pipeline option that was `reactors` is `foldSubscribers`, alongside `mapSubscribers`. Deliberately NOT renamed, because each is a contract someone outside this repo reads: - `es_reactor_total`, `es_reactor_duration_milliseconds`, `es_reactor_collapsed_total` and their `reactor_name` label — prod Grafana dashboards and alerts key on them. Their emitter helpers keep the matching name, so the function still tells you which metric it writes. - the `reactor` queue-kind literal and the `fold/<projection>/reactor/<name>` job path — the path IS the routing key, so renaming it strands jobs staged by old pods across a rolling deploy. - `reactor_dispatch` ops stage and the `reactor.name` span attribute. - `ReactorOutbox` in `prisma/migrations/**` — deployed migrations are immutable. ADR-098 records the kept set so the next reader finds a decision, not a miss. * docs(event-sourcing): retire the reactor vocabulary from the docs and specs too The code stopped saying "reactor" two commits ago; the prose had not caught up, and in three places it was actively lying to the reader: - `best_practices/event-sourcing-reactions.md` told people `withReactor` still exists for "current plain post-projection reactors". It does not exist at all. That section is now "There is no third primitive". - `event-sourcing/README.md` documented `.withReactor(foldName, reactorName, definition)` in its builder API table and gave a `ReactorDefinition` example. Both are replaced with the real `.withSubscriber(name, spec)` shape, including the `when` pre-enqueue guard and a pointer to `.withProcessManager`. - `queues/groupQueue/README.md` opened with a `.withReactor(...)` snippet. Customer-facing docs get customer-facing words rather than a swap of one piece of internal jargon for another: the governance pages now say "detector" and "detection", not "reactor" and not "subscriber". `llms.txt` / `llms-full.txt` are regenerated from those sources and now contain zero occurrences. Specs, testing guides, the Go services' comments about their TS counterpart, and the ClickHouse-adjacent prose follow the code's vocabulary. Feature parity stays green at 7,118 bound scenarios — the `@scenario` annotations moved with the titles because both sides renamed in the same pass. Historical ADRs keep their text, because an ADR is a record of what was decided at the time and rewriting it would falsify that record. ADR-023 and ADR-030 already carried supersede banners; ADR-026 was still "Accepted" while naming a predicate that no longer exists, so it gains an amendment mapping its vocabulary onto ADR-098's without touching the decision itself. ADR-098 gains the table of four deliberately-kept names, and its claim that the Customer.io subscribers were deleted is corrected — they are migrated and present, unregistered, with their spec bindings intact. Left alone on purpose: `prisma/migrations/**`, which is immutable history and is now the only place in the tree whose filename still carries the word. * fix(docs): repair the two links the vocabulary rename broke The blanket rename rewrote a filename inside a reference, not just prose: `subscriber.types.ts` pointed at `dev/docs/adr/026-subscriber-should-react-predicate.md`, which does not exist — the ADR is still `026-reactor-should-react-predicate.md` and, being history, keeps that name. It now links the real file and says why the two spellings differ. ADR-026 itself linked twice to `specs/event-sourcing/reactors.feature`, which this PR deletes. Both now point at its successor, `post-event-work.feature`. Checked the whole diff for the same class of defect: every ADR and spec path in an added line resolves to a file that exists. * fix(event-sourcing): repair what the vocabulary pass broke, and the contracts it left inaccurate The blanket rename rewrote "reactor" inside sentences that existed to contrast reactors with subscribers, which turns the claim into a tautology: a spec header announcing it supersedes itself, a dejaview test asserting subscribers are not subscribers, a queue test naming a jobType that does not exist. Swept the diff for the whole class rather than the flagged instances. It also left several contracts describing something the code no longer does. recordSpanCommand said reserved attributes survive the strip for "downstream subscribers", then named a fold two lines later. dispatchToSubscribers took foldName and was called with map projection names at three sites, so map dispatch failures logged under a fold-shaped field. throttleWindow accepted (event, state?) but declared its returned makeId as (event), while staticBuilder forwards the state. processManagerDefinition documented one queue lane where there are two. Pins the "reactor" jobType with the reason it outlives its vocabulary: it is the GroupQueue routing key, so renaming it strands in-flight jobs across a rolling deploy. ARCHITECTURE.md now names all three places a subscriber attaches, which is the ambiguity that forced the projectionSubscriber* spelling in the first place. Customer.io's trace and evaluation siblings were described as deleted. They were migrated, and both are present and unregistered. * refactor(event-sourcing): apply the repo's named-parameter rule to the genuinely new subscriber files git reports these files as added, not renamed, even at a 40% similarity threshold: converting a reactor definition into a subscriber spec rewrote enough of each one that they are new code by the repo's own reckoning. So the named-parameter standard applies to them, and the earlier claim that they merely "came across in a move" was wrong for this cohort. The real moves — pullRequestMapping.*, projectionRouter.*Collapse — keep their signatures, as do metrics.ts and coding-agent pipeline.ts, which are modified rather than new. Two of the flagged helpers took a second parameter no call site ever passed, so they lose it and become single-parameter rather than growing an options object for a value nobody supplies. _originGuardedSubscriber's handler re-check now fails open the way the router does for `when`. That re-check exists precisely so a fail-open `when` stays safe; a throwing guard inside it would otherwise be the one thing that loses the side effect it was added to protect. It logs and runs rather than dropping the work. Drops `hasRedis` from both broadcast subscriber deps — declared on new interfaces, read by neither handler. * refactor(zod): move the three genuinely leaf schemas to zod/v4, and name the record key types Continues the zod/v4 migration already begun in packages/langy. Of the 12 files in this PR that import zod, only three can move: the unit of migration is the contract, not the file. A v4 schema carries `_zod` / `$strip` internals that v3's `ZodType<any, ZodTypeDef, any>` and `ZodTypeAny` do not, so a schema can only move once everything typed against it has moved. Moving `processManagerDefinition.ts` alone — one type-only import — broke about 40 typecheck errors across ~30 files this PR never touches, because every process manager in ee/ and in six pipelines hands a v3 ZodObject to `IntentSpec<Schema extends ZodTypeAny>`. The command registry and hono-openapi's `resolver()` pin the rest the same way. tRPC 11 itself is fine — it takes Standard Schema — but a v4 input type then propagates into v3-typed React consumers. So the process-manager, command-schema, tRPC-router and Hono files stay on v3 until their contract moves, each of which is its own PR. The six `z.record(valueType)` calls now name their key type. v4 requires it, v3 accepts it, so that correction holds either way and is the one part of this that survives regardless of sequencing.
…project integration (ADR-093) (#6901) * docs(adr): ADR-093 — one automation flow with a source, and Slack as a project integration Records the frozen 2026-08-12 decisions for the #6717 design track: merge automation and alert into one flow behind a source (trace search vs graph metric), keep kind as the wire discriminator with source as a derived alias, re-step the composer as a wizard that edits from the review overview, unify the list table, move the Slack bot token to a per-project integration with most-specific-first resolution, and defer multi-channel delivery to a future ADR. Ships the @unimplemented behavioural spec and the reference-implementation plan (one flagged reference PR, then six fan-out units with strict file ownership). Refs #6717, #6716, #6896 * docs(adr): fold in Alex's three answers — the subject choice replaces any source picker The type card is deleted, not renamed: the wizard opens on "What should this automation watch?" (a trace filter or a graph) with the subject configured inline, collapsing to three steps — Watch, Delivery, Review — with edit still opening on Review. "Source" survives only as the derived wire alias; the wire values and the kind-discriminator compat design are unchanged. Token precedence stays most-specific-first, with the rotation gap handled by visibility (settings count plus a per-automation nudge on the row and in the drawer). Cadence seating in the Delivery step is decided: when and where it sends is one decision; ADR-043's Cadence facet is preserved at the data and draft level. Refs #6717 * docs(adr): fold corpus-audit findings into ADR-093 The false 'bound specs keep binding' claim becomes a per-file rebinding table naming which units rebind, supersede, or retire the affected scenarios (list-pages delete-nouns and Overview menu, the cadence-gallery regroup trigger, the cap-advice seats, the slack-bot-delivery token scenarios in the second specs root). Records the partial supersessions of ADR-037's stepper rejection and ADR-044's three-card picker, names the component being retired vs the action picker that survives, seats the cap advice (Review at create, Watch on edit), adopts ADR-021's single-scope-per-row shape for the SlackIntegration table with tenancy registration, narrows the never-store-a-token claim to creates and newly configured deliveries, pins template-graph atomicity to one Prisma transaction, records save-time-pinning-not-OAuth, bulk-clear semantics, and the three-state token read; adds the vocabulary-by-layer table, the missing cross-links, and the ADR-042 comment-citation fix. The spec gains the superseding and consolidation scenarios with their provenance noted. Refs #6717 * docs(adr): rename the third concept to Report, in ADR-093 and its spec Alex's 2026-08-12 pick after re-checking the Aug-4 transcript (Rogério: 'it should be like a report instead of a schedule'). Customer noun: Reports tab, New report; 'sends on a schedule' stays the description. Wire source value becomes report — free while the alias is unshipped, an API break later — aligning wire, storage (kind: REPORT), and the customer noun on one word. ADR-044 is superseded on the noun; the rebinding table now covers list-pages' schedule-delete copy too. Refs #6717 * docs(adr): mirror R0's amended-scope block into ADR-093 Verbatim from 2c92f99 on feat/automations-source-r0 so the branches merge clean: the View drawer's vocabulary joins R0 scope (its nouns are the merge's own copy; behaviour and the #6899 surfaces stay out), and templates seed rather than lock the Watch step until F5 ships their graph — locking an unanswered step would trap the author. Refs #6717 * docs(adr): align ADR-093 with the as-built R0 and fix the moved Slack spec path * chore(specs): declare ADR-093's spec inert until the reference PR binds it * docs(adr): label the diagram fences and settle two spec-prose threads The four ASCII diagrams carry a text language tag for MD040, the cap-advice feature reference reads as one unbroken filename, and the no-token scenario states what the author sees — no bot-token prompt, delivery through the project integration — instead of the row's storage layout. * docs(adr): align the deferred Slack-widening bullet with section 5's scoped table * docs(adr): call the scope widening what it is — an enum migration, no table-shape rework
…dates (#7090) chore(deps): bump the minor-and-patch group Bumps the minor-and-patch group in /sdks/python with 9 updates: | Package | From | To | | --- | --- | --- | | [python-liquid](https://github.com/jg-rp/liquid) | `2.3.0` | `2.3.1` | | [litellm](https://github.com/BerriAI/litellm) | `1.95.0` | `1.96.0` | | [ruff](https://github.com/astral-sh/ruff) | `0.16.1` | `0.16.2` | | [anthropic](https://github.com/anthropics/anthropic-sdk-python) | `0.120.2` | `0.121.0` | | [streamlit](https://github.com/streamlit/streamlit) | `1.60.0` | `1.61.1` | | [json-repair](https://github.com/mangiucugna/json_repair) | `0.61.7` | `0.62.0` | | [langsmith](https://github.com/langchain-ai/langsmith-sdk) | `0.10.15` | `0.10.17` | | [pypdf](https://github.com/py-pdf/pypdf) | `6.14.2` | `6.15.0` | | [nltk](https://github.com/nltk/nltk) | `3.10.1` | `3.10.2` | Updates `python-liquid` from 2.3.0 to 2.3.1 - [Release notes](https://github.com/jg-rp/liquid/releases) - [Changelog](https://github.com/jg-rp/liquid/blob/main/CHANGES.md) - [Commits](jg-rp/liquid@v2.3.0...v2.3.1) Updates `litellm` from 1.95.0 to 1.96.0 - [Release notes](https://github.com/BerriAI/litellm/releases) - [Commits](BerriAI/litellm@v1.95.0...v1.96.0) Updates `ruff` from 0.16.1 to 0.16.2 - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.16.1...0.16.2) Updates `anthropic` from 0.120.2 to 0.121.0 - [Release notes](https://github.com/anthropics/anthropic-sdk-python/releases) - [Changelog](https://github.com/anthropics/anthropic-sdk-python/blob/main/CHANGELOG.md) - [Commits](anthropics/anthropic-sdk-python@v0.120.2...v0.121.0) Updates `streamlit` from 1.60.0 to 1.61.1 - [Release notes](https://github.com/streamlit/streamlit/releases) - [Commits](streamlit/streamlit@1.60.0...1.61.1) Updates `json-repair` from 0.61.7 to 0.62.0 - [Release notes](https://github.com/mangiucugna/json_repair/releases) - [Commits](mangiucugna/json_repair@v0.61.7...v0.62.0) Updates `langsmith` from 0.10.15 to 0.10.17 - [Release notes](https://github.com/langchain-ai/langsmith-sdk/releases) - [Commits](langchain-ai/langsmith-sdk@v0.10.15...v0.10.17) Updates `pypdf` from 6.14.2 to 6.15.0 - [Release notes](https://github.com/py-pdf/pypdf/releases) - [Changelog](https://github.com/py-pdf/pypdf/blob/main/CHANGELOG.md) - [Commits](py-pdf/pypdf@6.14.2...6.15.0) Updates `nltk` from 3.10.1 to 3.10.2 - [Release notes](https://github.com/nltk/nltk/releases) - [Changelog](https://github.com/nltk/nltk/blob/develop/ChangeLog) - [Commits](nltk/nltk@v3.10.1...v3.10.2) --- updated-dependencies: - dependency-name: python-liquid dependency-version: 2.3.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: litellm dependency-version: 1.96.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: ruff dependency-version: 0.16.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: anthropic dependency-version: 0.121.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: streamlit dependency-version: 1.61.1 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: json-repair dependency-version: 0.62.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: langsmith dependency-version: 0.10.17 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: pypdf dependency-version: 6.15.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor-and-patch - dependency-name: nltk dependency-version: 3.10.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…updates (#7094) Bumps the github-actions group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `7.0.0` | `7.0.1` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4.3.0` | `8.0.1` | | [github/codeql-action/init](https://github.com/github/codeql-action) | `4.37.6` | `4.37.7` | | [github/codeql-action/analyze](https://github.com/github/codeql-action) | `4.37.6` | `4.37.7` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `9.0.0` | `10.0.1` | Updates `actions/checkout` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v7...3d3c42e) Updates `actions/upload-artifact` from 4.6.2 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4.6.2...043fb46) Updates `actions/download-artifact` from 4.3.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v4.3.0...3e5f45b) Updates `github/codeql-action/init` from 4.37.6 to 4.37.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@5595cca...ff2f1c6) Updates `github/codeql-action/analyze` from 4.37.6 to 4.37.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@5595cca...ff2f1c6) Updates `astral-sh/setup-uv` from 9.0.0 to 10.0.1 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@c771a70...20cfd1b) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions - dependency-name: github/codeql-action/init dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: github/codeql-action/analyze dependency-version: 4.37.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions - dependency-name: astral-sh/setup-uv dependency-version: 10.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… a comparison (#7095) `findDeadMessages` opened two type-argument lists — `$queryRaw<` and `Array<` — and closed only one before `(`. That is not a syntax error: TypeScript falls back to parsing the whole expression as a `<` comparison, so the emitted JavaScript is `$queryRaw < Array(sql)`. The query never reached Postgres, `rows` was a boolean, and every load of /ops/event-sourcing/dead-letters died on `rows.map is not a function` in 3ms, surfacing as a spinner (react-query retrying) and then a generic "unknown error" header. `pnpm typecheck` could not catch it, because a comparison typechecks. The existing coverage is integration-only, so the guard added here is a unit test that calls both dead-letter reads against a stubbed Prisma and asserts on what came back — it fails with the production `TypeError` on the unfixed code and runs on every change. Swept every other `$queryRaw`/`$executeRaw` site in the app by comparing the emitted JavaScript; this was the only one affected. The SQL itself was validated against a real Postgres schema — it had never executed before.
…running the full suite (#7101) release-please creates-or-updates a release PR per package on every merge to main. A release PR's diff is a changelog plus version fields in package.json, platform/app/package.json and three Chart.yaml files. Nine workflows path-filter on `package.json` — correctly, since a dependency bump touches exactly those files — and a path filter cannot tell a version bump from a dependency bump. So a nine-file metadata change pulled the whole suite: ten test shards, four kind-cluster e2e legs, both CodeQL systems. Measured at 88 check runs per push on PR #6930, against ~21 merges to main a day, across eight of these branches at once. One release branch alone logged 300 workflow runs in four hours, which was the sample cap rather than the total. It was the largest single consumer of the org's Actions capacity, for pull requests that change no code and do not merge until a release. Twelve workflows already route their heavy jobs through the `heavy` output of .github/actions/detect-changes, which is false on a draft, so opening these as drafts buys the reduction with no new mechanism — no label convention, no new trigger, and no code path that is not already load-bearing. The full run still happens before anything ships. Marking a release PR ready fires `ready_for_review`, which those workflows list in their `pull_request.types` precisely so the heavy jobs get their run at that point, and a draft cannot be merged — so "ready" is the same click as "I am releasing this". A green draft is deliberately NOT evidence of a full pass; go-ci skips build and the race tests on drafts, which is the whole reason the ready flip exists. The eight open release PRs were marked draft by hand. Setting it in the config matters on the next cycle: release-please updates an existing release PR, but once one merges it opens a fresh PR for the following version, and that one would be non-draft again — so without this the manual step repeats every release. The rationale lives in the release-please-sdks.yml header because release-please-config.json is strict JSON and cannot carry a comment. Not covered by this change: langwatch-chart has no gate job and does not read `heavy`, so its e2e legs — the most expensive in the repo at 697s and 348s by its own measurement — still run on a draft release PR. That gate belongs in the change that is already restructuring that workflow.
`go-ci / test` is red on main and takes every open pull request with it, because GitHub tests the merge ref: run 31998757865 at 60fa445 fails TestLeanCheckoutHoldsInTheLiveRepo with langwatch-app-ci.yml job "shard-durations" does not exclude /docs/media/ ... /docs/images/ ... /assets/ This is a false positive, not a leanness problem. The job checks out one file: sparse-checkout: platform/app/vitest.durations.json sparse-checkout-cone-mode: false which cannot pull the media, because it pulls a single JSON file. But `leanCheckoutStep` exempted exactly one hardcoded value — `gateOnlyPattern`, `.github` — and required the three `!` exclusions of everything else. So an allow-list checkout was judged by a rule written for exclusion-list checkouts, and satisfying it would mean adding three negations next to a single named file: lines asserting the absence of directories the step never asked for, and which under a non-cone allow-list do nothing at all. The fix generalises the existing special case rather than adding a second one. A sparse-checkout that names no whole-tree entry (`/*`, `*`, `/**`, `**`) and no exclusion is an allow-list, and an allow-list is lean by construction: anything it does not name is never checked out. `.github` is simply an instance of that. A denylist — one that starts from the whole tree — still has to name the exclusions, and TestLeanCheckoutStillRequiresExclusionsOnAWholeTreeCheckout covers that so the exemption cannot swallow the case it exists to catch. That test asserts all three missing exclusions rather than two of the three it requires a count of. The alternative was adding the three lines to shard-durations. It unblocks main in one commit too, but it bakes a redundant incantation into the workflow and leaves the next allow-list checkout to trip the same wire. Deliberately its own change against main: it is unrelated to anything else in flight and everyone is blocked on it. The same fix also exists in #7096, which is a larger CI-restructuring PR still under review — merging this first is what unblocks the queue, and #7096 carries the identical content so it will reconcile without conflict.
* refactor(ops): answer each operator question on one surface The ops landing page grew by accretion, so the same question was answered in several places and the reader had to know which mechanism to suspect before they could look. "What is switched off?" was split three ways — parked tenants on the dashboard, switched-off schedules on the schedules page, paused subscribers on the subscribers page — so nobody ever saw the whole answer, and the common case (someone paused something mid-incident and never resumed it) was what the pages reported worst. They merge into one "Switched off" panel that states the shared fact once and keeps a section per mechanism, because the remedy differs: parking clears itself when capacity frees, a schedule and a subscriber each need a human. Nothing switched off renders nothing at all. Two cards move to the surface whose question they answer. Upcoming timed work is a read of the calendar, so it sits on the schedules page above the schedules it previews; replay history sits beside the button that starts a replay. On the landing page each was a table of things that are fine. QueuesContent is deleted. /ops/queues has been a redirect since the event-sourcing section landed, so the component it used to render — five cards the dashboard already mounts, component for component — has been unreachable code since then. The command-bar entries still pointing at it, and at the retired /ops/projections, now name the pages that exist, and the dead-letters page gains the entry it never had. The layout doc gains the rule this follows, and the density spec gains six scenarios for it, all bound. One bug found while building the panel: Stack interleaves `separator` by child count, not by what those children render, so handing it three sections and letting the empty ones return null ends the card in stray rules. Only sections with content reach the Stack, and a test pins it. * Address review comments - Ask Postgres for the switched-off schedules instead of filtering a bounded page of listScheduledJobs. That read orders `active DESC`, and Postgres sorts true above false, so the inactive rows were exactly the ones its LIMIT dropped: on any fleet with more schedules than the page held, the panel reported zero paused schedules and looked like it had checked. listPausedSchedules filters in SQL and returns the fleet total, so the section can say "showing 50 of 137". - The unreachable bounded-page notice goes with it; the count is now a real total rather than a page length, so the card renders whenever something is switched off. - Split PausedCard: the query and its derived state move to a usePausedSchedules hook, and the section list to a private helper. * Fix two integration failures this branch inherited from main Neither is caused by this branch, but both are red on the required integration lane, and one of them is fallout from #7095. Dead-letter tests saw another block's rows. The fleet-count block seeds `dead-1` and `dead-2` under plain `ns` to assert `deadMessages: 2`, and the dead read is fleet-wide, so the dead-letter assertions — which narrowed on `ns` — counted them too. That stayed invisible until #7095 made the read actually reach Postgres: before that it threw before it could return a row, so these tests had never once run green. The block now seeds and narrows on its own `nsDead`. Tracked-event validation answered 500 where it means 400. `predefinedEvents.schema.ts` is authored against `zod/v4` while the base schema is v3, and `fromZodError` from zod-validation-error@3 understands only the v3 error: handed a v4 one it reads `.errors`, which does not exist there, and throws `TypeError: Cannot read properties of undefined (reading 'length')`. That throw escaped the very catch block whose job was to turn a validation failure into a 400, so a malformed `thumbs_up_down` got a 500 naming no field. Both tracked-event routes carried the same two lines, so both were affected — this is a live bug on the public API, not just a red test. `zodErrorMessage` formats an error from either entrypoint and never throws (`fromError`, not `fromZodError`, for the same reason). It flattens union branches, because `z.prettifyError` renders a union as "Invalid input" and drops the branch issues — the caller would have got a 400 that still named no field. * Drop the ZodError type import the formatter change made unused * Remove a scratch script committed by mistake * Bind the paused-panel tests to the scenarios they actually exercise The "still reports the switched-off ones the page would have dropped" test claimed `The paused panel is absent when nothing is switched off` while asserting the panel is present with a schedule row — a binding that vouched for a scenario it contradicts. Two neighbours were loose in the same direction, claiming `Everything switched off is reported together` while asserting the opposite: that a mechanism with nothing to say draws neither a heading nor a rule. check-feature-parity only asks whether some test names a scenario, not whether that test exercises it, so a mis-pointed annotation reads green while leaving the scenario unenforced. Two scenarios added for the behaviour that was being asserted, and the surrounding `describe` now states what the test sets up rather than the server-side situation the panel is immune to. Reported by CodeRabbit on #7099.
* feat(authz): stage B in place - the system backfills itself
ADR-092 stage B as one PR, per the one-PR-per-stage restructure. No
scripts: a leased runner (@langwatch/system-migrations, generic over
tenants) walks every cohort organization through pending -> migrated ->
finalized at worker boot, parked-and-retried on error. The M1 rider
(TeamUserBackfillMigration in @langwatch/authz-server) writes each legacy
TeamUser row's equivalent TEAM-scoped binding idempotently, audits the
batch as source: backfill-b, bumps the org epoch once, and finalizes only
on a clean collect-once decide-twice parity sweep - organizations leaning
on the legacy org-level union quirk are HELD with the disagreement in
their report, behaviour unchanged, re-verified every pass.
Finalized organizations stop consulting the legacy fallback through one
gate (rbac.ts's three fallback branches and the engine collector), so the
shadow keeps agreeing with the legacy path it wraps. Self-hosted migrates
silently with no configuration; cloud is paced by SYSTEM_MIGRATIONS_COHORT
and the new Ops -> Migrations page (state rollup, held/parked reports,
run-a-pass-now under ops:manage).
Spec: specs/rbac/in-place-authz-migration.feature, all scenarios bound.
Delivery plan amended: one PR per stage, in-place migration doctrine, M1
runbook row updated, stage-B tails ride stage E.
* docs(authz): delivery plan reflects one-PR-per-stage and the in-place doctrine
* chore(workspace): enumerate @langwatch/system-migrations everywhere members are listed
Root files[] (the ADR-076 invariants gate), both manifest COPY lists
(langyagent + mcp images), and the pack completeness guard.
* style(authz): wrap the gate's negative-cache condition to width
* style(authz): format the gate and its test the way the formatter prints them
* fix(authz): review round - the gate's positive cache expires so the state-row rollback lands fleet-wide; operator-kicked passes log their own death
* fix(authz): review round two - the rollback actually sticks, and a died pass finishes its work
The CI failure first: SystemMigrationTenantState landed without a tenancy
regime, so dbMultiTenancyProtection's partition test refused to classify it.
It is a SCOPED_MODEL now - one tenant's row, or one migration's rows for the
platform-scope ops rollup, and a query naming neither still throws.
Then the two real defects review found.
A migration that committed its bindings and died before bumping the epoch
could never bump it: the retry found nothing missing and returned early, so
every cache kept serving pre-backfill decisions forever. The runner now hands
migrateTenant the tenant's previous record, and a parked predecessor is the
signal to redo the publish step.
And the documented rollback did not roll anything back. Moving an org off
`finalized` only bought minutes - the next pass re-ran a migration whose proof
still passed and re-finalized it. `rolled_back` is a real terminal status now:
the only one the runner refuses to act on, so writing it returns the org to
its legacy path fleet-wide and pins it there.
Smaller ones, all found by review:
- the lease is held by a heartbeat for the whole pass, not renewed between
tenants - one big org's parity sweep can outlive a 60s term on its own
- the parity sweep honours the abort signal, and throws rather than returning
a short diff list that would read as a clean proof and finalize the org
- a null report writes Prisma.DbNull instead of omitting the column, so
finalizing clears the parked error it replaces
- the gate's two caches became one, so expired entries are dropped rather
than accumulating for the life of the pod
- ops.ts uses top-level imports (startWorkers.ts keeps its lazy ones - a
static import there hoists above setEnvironment and breaks env loading)
- one failed 30s poll no longer replaces the loaded migrations table with an
error panel
* refactor(authz): the app layer owns the migration pass, and binding identity matches the index
The migrations were a boot stage in startWorkers with its own lazy import.
They are a worker-only background loop, which the app layer already has a
shape for: presets starts them behind roleRunsWorkers() exactly like the
ADR-044 scheduler, and the App's graceful closeables stop them. startWorkers
loses the stage and the import with it.
One trap that shape sets: runtime.ts read Redis back off tryGetApp(), which is
still null while presets is composing the App - a null handle makes the lease
unacquirable, so every boot pass would have stood down silently. The handle is
passed in now; the ops "run a pass" action still reads it off the App, where
it exists by then.
Separately, bindingKey claimed the partial unique indexes decide identity and
then disagreed with them. Custom-role bindings are unique on (userId,
customRoleId, scopeType, scopeId) - `role` is not in that index. Including it
made an existing custom binding look missing whenever its role differed, and
createMany({ skipDuplicates }) then dropped the insert without a word, leaving
a created count that overstated what landed. The key is now the two keys the
database actually uses.
* style(authz): sort the system-migrations import where biome puts it
* chore(deps): put system-migrations on TypeScript 7 with the rest of the workspace
#7081 moved every package to ^7.0.2 and dropped typescript@5.9.3 from the
lockfile. This branch's new package predated that and still asked for ^5.9.2,
so the merge with main produced a lockfile with no entry for the version its
own importer named, and every --frozen-lockfile install failed.
* fix(ops): bulk writes to migration state must name a tenant, and the route goes through a service
Two review findings, both real.
The tenancy guard ran one validator for reads and writes alike, and mine
accepted a bare `migrationName`. That is right for the ops rollup, which lists
one migration across tenants, and badly wrong for `deleteMany` /`updateMany`:
`deleteMany({ where: { migrationName } })` would drop every tenant's row, and
the rows it drops include the `finalized` latches - silently returning every
switched-over organization to its legacy path. `validateWhere` now takes the
action, and a bulk write has to name a tenant. Seven cases cover it.
The ops route also imported the state repository and assembled the overview
itself, against the standing rule that routes go through a service layer.
`SystemMigrationsService` owns the read model and the fire-and-forget pass;
the route is two thin calls and the repository is no longer exported.
* test(tenancy): give the migration-state guard scenarios a home in the spec
The two @Scenario annotations I put on the new guard tests named scenarios
that did not exist, which feature-parity reports as unknown-annotation
failures rather than binding anything. They belong in the scoped-models
tenancy spec beside the other guard rules, where the read/write asymmetry is
worth stating: a migration-wide predicate is a bounded view of many tenants
and an unbounded edit of them.
* fix(authz): a failing park must not take the fleet down with it
Review round. Two of these are the difference between a pass that degrades
and one that stops:
The parking write sits in the catch block, and it is itself a write. The
failure most likely to park a tenant — the state store being unreachable —
is therefore the one that throws a second time, out of `runMigrationForTenant`
and all the way through `runPass`, so the remaining tenants are never tried
and the summary is lost. That is the opposite of what the comment above it
promises. The park is now best-effort: an unrecorded park costs nothing the
next pass cannot rebuild, because the tenant is still pending.
The Redis lease swallowed its errors whole. Every falsy `acquire` is logged
by the runner as "another process holds the lease", so an unreachable Redis
read as ordinary contention and the migration would never run, on any boot,
without a word anywhere. Both catch blocks now say what actually happened
and keep their fail-safe return.
The rest is convention: `shouldRepublishEpoch` takes the boolean prefix the
path rules ask for and says what the flag controls, the tenancy-guard suites
move to when-condition describes and action-named tests, and the ASCII
diagrams get the language tag markdownlint wants.
The new package also gains the typecheck it never had — `pnpm typecheck` at
the root checks @langwatch/web alone, so nothing in CI was compiling
system-migrations' own TypeScript.
Drop the namespace the dead-letter block stopped using Giving that block its own `nsDead`/`nsDeadB` left `nsB` declared and unreferenced. `noUnusedVariables` is a warning rather than an error, so a whole-tree `pnpm lint` still exits 0 — CI gates on the delta instead (`0 -> 1`), which is the check that caught it.
…o cache that never saved (#7096) * perf(ci): stop leasing runners to decide there is nothing to do CI here is allocation-bound, not compute-bound. Measured across a 380-job sample: peak 76 concurrent jobs is the org-wide ceiling, 477 runs were queued against 23 running, and 220 of 239 successful jobs finished in under 90 seconds. Queue delay is bimodal — p50 0s, p90 21 minutes — which is the signature of a hard cap rather than slow hardware. PR #6902 alone produced 82 check runs, so one pull request can exceed the whole organisation's capacity. So the saving is measured in runner slots, not minutes. The always-run + green-aggregator pattern is NOT what is being removed. A workflow filtered out by `on.paths` never reports a status, and a required check that never reports blocks the pull request forever, so every workflow whose `-complete` check is in the ruleset keeps running unconditionally and keeps reporting green on an untouched PR. Only workflows with no required check are filtered, and a new guard enforces that. langwatch-chart: 17 jobs to 3. A twelve-leg `helm template` matrix plus five single-script jobs each spent ~17s installing helm and building chart dependencies to do 1-5s of work; every leg measured under 30s end to end. They now share one runner. Diagnostic parity with `fail-fast: false` is preserved deliberately: the render step keeps going after a failure and names every preset that broke, and the assertion steps carry `if: !cancelled()` so one failure does not hide the next four. ssrf-conformance and agent-plugin-ci: gate job and unrequired aggregator removed in favour of `on.paths`, 3 jobs to 1 each, and 0 when irrelevant. agent-plugin-ci's draft skip moves onto the job itself rather than costing a whole gate job to compute. go-services keeps its gate, because it runs three jobs off three different filters and `on.paths` is workflow-wide. What it gains is the case where none of the three match. migration-order gains the two directories tools/migrationorder actually scans. Only `pull_request` is filtered anywhere. Pushes to main still run everything, per specs/ci/path-filters.feature — the branch we deploy from keeps full coverage. New guard, .github/scripts/guard-path-filters.ts, wired into workflow-security-guard alongside the existing pull_request_target guard and run from the base checkout so a PR cannot disable it in the same commit that breaks a filter. It enforces two rules that otherwise fail silently: R1 `on.paths` must be a superset of every path the workflow's own gate filters on. Drop `charts/gateway/**` from go-services and the helm job stops existing, with the PR green. Verified against the real file: the guard fails naming that exact path. R2 A workflow must not have both `on.paths` and a `-complete` aggregator, since the filter is what stops the aggregator reporting. specs/ci/path-filters.feature described "-unmodified" stub pairs that were replaced by the aggregator pattern in #3223 and no longer exist. Rewritten to describe what is actually there, with the three new scenarios tagged and bound (6/6 bound). Deliberately unchanged: code-scanners stays always-run. It carries the gitleaks/trufflehog secrets scan, and secrets can land in any file type, so path-filtering it would trade a real guarantee for one slot — which the spec already says ("gitleaks, trufflehog and semgrep still run"). Also unchanged, because both need a settings change landing at the same instant and cannot go in a code PR: - CodeQL runs twice. GitHub's default code-scanning setup is enabled at org level and runs go/javascript-typescript/python/ruby on every PR (~816 runner-seconds, 4 slots) on top of this repo's own tuned codeql.yml. There is no Ruby in this repository. - Legacy branch protection on main requires bare `build` and `typecheck`. `build` is a job name in six workflows and `typecheck` in three, so which check satisfies the requirement is undefined. Both are already covered by `langwatch-app-complete`, which needs them. This is why go-ci and sdk-python-ci are NOT path-filtered here: both expose a job named `build`. * fix(ci): make the path-filter guard fail closed, and accept a lean allowlist checkout Addresses the CodeRabbit review, plus the pre-existing ciguard failure on main. The guard failed open on every shape its hand-written parser could not read. `pullRequestPaths` returned null for a quoted `"on":` key, a flow-style `paths: ["pkg/**"]`, `paths-ignore:`, or four-space indentation, and `inspect` read null as "not filtered" and passed the file. `gateFilterPaths` matched only `filters: |`, so `filters: >-` and dorny's filters-file form contributed no declared paths and R1 could not fire. That is the guard reproducing, inside itself, the exact silent-pass it was written to prevent. Now every unreadable shape is reported as R3. A guard that cannot read a file says so instead of going green. The parser also handles the shapes it was missing: quoted trigger keys, `pull_request_target`, flow sequences, arbitrary indentation, and folded block scalars. `covers` ignored negation, which defeated R1 with the pattern R1 exists to catch: with `on.paths` = [pkg/**, !pkg/ssrf/**] and a gate filter naming pkg/ssrf/address.go, the old check matched the first entry, passed, and GitHub then excluded the file so the workflow never started. It now evaluates the list in order and lets a later negation remove coverage an earlier entry granted. Prefix matching is replaced by real glob matching, so `**/*.go` and `platform/*/prisma/**` no longer produce spurious R1s, and the duplicated trailing-`**` branch is gone. `listEntries` promised "comments dropped" but only dropped whole-line comments: `- 'pkg/**' # why` left a mangled value and produced a spurious R1. It now strips a trailing comment whether or not the entry is quoted. Both filter blocks in this repo already carry per-line comments, so this was reachable. The chart render step's comment claimed "No `set -e`" while never clearing it. Actions runs run: steps with -e already on and `set -uo pipefail` does not disable it, so the promise was not implemented. The `if` around the helm call is exempt from -e on its own, which is why the loop worked, but that is a property of `if` rather than of the step — the next unguarded command would have silently reintroduced fail-fast. `set +e` now does what the comment said. Verified by running the extracted step under `bash -e` against a stub helm that fails two presets: both are named and the step exits 1. Separately, TestLeanCheckoutHoldsInTheLiveRepo was already failing on main -- neither .github/workflows/langwatch-app-ci.yml nor tools/ciguard/ is touched by the rest of this branch. The rule exempted exactly one pattern, `.github`, so the shard-durations job's `sparse-checkout: platform/app/vitest.durations.json` fell through to the media check and was asked to exclude three directories it had never asked for. The exemption is now the general property the special case was an instance of: a sparse-checkout that is a pure allowlist, naming no whole-tree entry and no exclusion, cannot pull the media, because anything it does not name is never checked out. A denylist -- one that starts from `/*` -- still has to name the exclusions, and a test covers that so the exemption cannot swallow the case it exists to catch. 22 guard tests pass, ciguard passes, and both specs are fully bound (path-filters 8/8, lean-checkout 8/8). * perf(ci): release the chart e2e slots on a draft The two e2e legs are the most expensive jobs in the repository — `e2e: infra` runs to 697s and `e2e: overlays` to 348s, both building images and standing up a kind cluster — and `behavioral` installs pnpm, goose and a datastore suite behind them. Twelve workflows already skip their heavy jobs on a draft through the `heavy` output of detect-changes. This one has no `changes` gate to read that from, so the check is inline, and it was the last heavy thing still running on a draft release-please PR — which touches `charts/**` and `package.json` on every merge to main and so fires this workflow every time. `ready_for_review` joins `pull_request.types` in the same change: it is not a default activity type, so without it taking a PR out of draft fires no event at all and these jobs would never get their non-draft run. * fix(ci): close three more fail-open holes in the path-filter guard All three are the same class the guard exists to prevent, one rule over. `aggregatorJobs` required a job key at exactly two spaces. Two-space is a convention, not a rule, so a four-space workflow had no detectable aggregators and R2 could never fire — the aggregator-plus-filter contradiction passed silently. The indent is now read from the `jobs:` block itself, with a test that a nested key is still not mistaken for a job. `inspect` returned as soon as the trigger was unparsed, which skipped R2. But an unparsed filter is still a FILTER: `pullRequestFilter` only reaches that state having found a paths-like key under the pull-request trigger. R2 now runs for those files too; only R1 is skipped, because there are no entries to compare. `inlineValue` treated a trailing comment as a value, so `paths: # only the Go tree` above a normal block list read as the inline value `# only the Go tree`, was not decomposable, and reported R3 against a perfectly legal shape. A false positive is not a safe failure — it is how a guard teaches people to ignore it. The comment is stripped before the key rather than after, since afterwards there is no leading whitespace left for the pattern to anchor on. A `#` inside a quoted entry still survives. Also merged the two byte-identical branches in `gateFilters` and renamed `sawBlock` to `hasInlineBlock`, which is what it actually means. The ciguard whole-tree test now asserts all three missing exclusions rather than two of the three it requires a count of. 27 guard tests pass, ciguard passes, and the guard still catches the live regression it was written for. * perf(ci): name the composite actions the app suite depends on `.github/actions/**` in the app CI gate matched every composite action in the repository, including ones this workflow has never used — and that one flag gates nine jobs: typecheck, all ten test shards, build and e2e. Observed on #7103, which adds `.github/actions/go-build-cache` for go-ci and touches two Go CI files and nothing else. It ran the entire application test suite: four unit shards, four integration shards, two component shards, typecheck, build, e2e. Replaced with the three actions this workflow actually uses. The comment above them states the trade, because it is a real one: a list can go stale where a glob cannot, so adding a `uses: ./.github/actions/<name>` step here without adding it to the filter means an edit to that action silently stops re-running the suite it feeds. `workflow-security-guard` keeps its `.github/actions/**` — it scans every action for pull_request_target safety, so a new one genuinely does need scanning there. * perf(go-ci): give the build cache a rotating key, and put the cheap checks on one runner Eight jobs become five, and the slowest one stops recompiling from a cache that could never learn. THE CACHE. `actions/setup-go` caches ~/go/pkg/mod and ~/.cache/go-build together under one key derived from go.sum. That is right for the module cache — a dependency set is immutable, so an immutable key is correct — and wrong for the build cache, which changes with every commit. Every go-ci run says so: Cache hit for: setup-go-Linux-x64-ubuntu24-go-1.26.6-<hash of go.sum> Cache restored successfully ... Cache hit occurred on the primary key ..., not saving cache. `actions/cache` never saves on a primary-key hit, so while go.sum is unchanged — most of the time — the build cache is frozen at whenever that key was first written. Each run recompiles whatever drifted since and discards the result. Under `-race`, which roughly doubles compile time, that was 164s of the `test` job's 188s. .github/actions/go-build-cache keys on the commit, so it always misses and therefore always saves, with restore-keys falling back to the newest cache for the same go.sum. Race-instrumented objects get their own suffix, since restoring plain objects into a `-race` build rebuilds everything anyway. setup-go's cache stays on for the module half, which is the half it gets right. THE VET JOB IS MOSTLY REDUNDANT. govet is one of golangci-lint's always-on defaults (.golangci.yml: "errcheck, govet, ineffassign, staticcheck, unused"), so vetting the packages `lint` already lints ran the same analyser twice for 103s and a runner lease. Mostly, not entirely: `./tools/thuishaven/...` is deliberately outside the lint scope — 231 pre-existing findings, see that step's comment — and the vet job did cover it. Deleting the job wholesale would have silently ended govet there, so that one package survives as a step. `./tools/ciguard/...` goes the other way, linted but never vetted, and needs nothing. THE GROUPING. lint (118s), the generated-code check (39s) and the CI guards (35s) each leased a runner and each paid ~25s of checkout and toolchain setup to do it. Under a saturated queue the lease, not the work, is the cost — a 35s job can wait twenty minutes for a slot it holds for half a minute. They now share one runner as `lint · gen · guards`. Serial that is ~192s against `test`'s 185s, so it does not become the critical path — but only because vet is gone. With it the total was 295s and this would have cost ~110s of wall clock to save three slots. That arithmetic is in the job's comment, because it is the thing to recheck before adding a fourth item. Two behaviours are preserved deliberately. `ci-guards` ran on `go OR ciguards` because it reads workflow files rather than Go source, so the job's `if:` is the union and the steps keep the distinction individually — a workflow-only change still does not pay for golangci-lint. And every step carries `!cancelled()`, so one failure does not hide the next four: a run reports everything broken, the way four separate jobs did, and the job still fails because a failed step fails its job. Verified locally: `go vet ./tools/thuishaven/...` clean, `go test ./tools/ciguard/...` ok, `go run ./cmd/ciguard` passes both guards, and go-ci is not in LeanCheckoutWorkflows so the fetch-depth: 0 checkout does not trip that rule. * perf(app-ci): put the five cheap checks on one runner ast-grep (26s), feature parity (49s), OpenAPI completeness (85s), the CI script tests (16s) and the CI self-test (41s) were five jobs, and three of them independently ran `setup-pnpm-node` with the same `@langwatch/web...` filter. Under a saturated queue the lease, not the work, is the cost — and the duplicated install was most of the work. Serial that is ~217s of steps against test-integration's ~700s, so it never becomes the critical path, and collapsing three installs into one puts the real figure below that. Two jobs are deliberately NOT folded in. `lint` carries a job-scoped `pull-requests: write` so reviewdog can annotate the diff, and its comment is explicit that scoping it to that job is the point: nothing else gains the write. Merging it would hand that scope to five more steps. `typecheck` and `build` are required by BARE NAME in the legacy branch protection on main. Folding either into another job would delete the check that protection waits for, and every pull request would block on a context that can never report. That is the same ambiguity already noted in this branch — `build` is a job name in six workflows and `typecheck` in three — and it is why those two stay standalone until the requirement is disambiguated. `feature-parity` had its own gate, tracking specs and test roots rather than app source, so the job's `if:` is the union and the steps keep the distinction individually. Every step carries `!cancelled()`, so one failure does not hide the next four. A failure in one check does not stop the others — every step carries `!cancelled()` — and a Summary step at the end names all of them together, so five checks in one job does not mean five places to look. It re-fails on purpose: the job is already red, and this only adds a readable verdict at the bottom of the log. go-ci's `checks` job gets the same treatment. The summary was tested against three cases: two failures are both named and it exits 1; all-green exits 0; and gated-off steps come through as empty strings without tripping `set -u`. Verified: every command from the five jobs appears exactly once in the merged job, `typecheck` and `build` are still standalone, the aggregator's `needs` and `allowed-skips` are updated, and the workflow's job graph resolves. * revert(ci): one check per job again, in go-ci and the app suite Batching the cheap checks onto shared runners saved slots and cost diagnosability, and the trade is not worth it. A red `checks` job says only that something in it broke; five red job names on the pull request say which, without opening anything. With the queue back to single digits the slots were never the scarce thing. So `ast-grep`, `feature-parity`, `openapi-completeness`, `ci-scripts-test` and `ci-self-test` are five jobs again, and go-ci's `lint`, `generated` and `ci-guards` are three. The `Summary` steps that named every failure go with them — they existed to buy back what batching took, and nothing has been taken. Two things from that work are deliberately kept, because neither is batching. The `go-build-cache` action stays. `setup-go` keys the module cache and the build cache together on go.sum, and `actions/cache` never saves on a primary-key hit, so every run logged "not saving cache" and recompiled the drift under `-race`: 164s of the test job's 188s. The new key rotates with the commit, so it always saves. The `vet` job stays gone. govet is one of golangci-lint's always-on defaults and `lint` covers the same package list, so it was the same analyser twice for 103s. That is redundancy, not consolidation — removing it costs no signal. One package is the exception and is now unvetted: `./tools/thuishaven/...`, which `lint` deliberately excludes over 231 pre-existing findings. It is a local dev CLI, so the exposure is a printf or lock-copy mistake reaching developers rather than production. The note where the job used to be spells out the one-step job that restores it, if that trade reads differently later. The app CI gate also keeps naming the three composite actions it uses instead of globbing `.github/actions/**`, which had been running the entire app suite on changes to actions this workflow has never used. * revert(go-ci): keep the vet job Restored. The case for deleting it was that govet is one of golangci-lint's always-on defaults and `lint` runs golangci-lint over every package `vet` covered except tools/thuishaven — so it looked like the same analyser twice for 103s. That case was more confident than the evidence supports. golangci-lint runs govet with its OWN default analyser set, and .golangci.yml has no `govet` block pinning that to what `go vet` runs, so the overlap is likely but not guaranteed. A cheap job is the wrong thing to bet coverage on, and this also closes the tools/thuishaven gap that deleting it opened. The reasoning is now written into the job so the next person to notice the overlap finds the answer rather than repeating the change. `go vet` type-checks everything it inspects, so it picks up the same go-build-cache as `build` — plain rather than `race`, since it compiles the same objects. * fix(ci): keep a # that YAML treats as scalar content `inlineValue` stripped `/\s+#.*$/` unconditionally, so `paths: ["pkg # b/**"]` was truncated to `paths: ["pkg`, no longer parsed as a flow sequence, and reported R3 against a perfectly legal filter. A `#` opens a comment only outside a quoted scalar. `stripComment` walks the line tracking quote state and cuts at the first `#` that is both outside quotes and preceded by whitespace, so `["pkg # b/**"]` and `a#b` both survive while `paths: # only Go` and `paths: ["a"] # why` still lose their comments. This also makes the comment above the function true. It claimed a `#` inside a quoted entry survived, which held for `"a#b"` — no preceding space — but not for `"pkg # b"`, which is the case the rule exists for. `unquote` needed no change: its `^(['"])(.*?)\1\s*(?:#.*)?$` expands to the final quote, so a quoted list entry already kept its hash. The bug was confined to `inlineValue`, which sees the whole line including the key. 31 tests, four of them new and covering both directions.
…7106) Moving `thresholdConfig.schema.ts` to `zod/v4` turned every rejection it produces into an unnamed 500. The gate that catches it, and the tRPC formatter behind that, both ask `err instanceof ZodError` against the v3 class they import, and a v4 `ZodError` is a different class — so the rejection stopped being a 422 `validation_error` the admin could act on and became an INTERNAL_SERVER_ERROR carrying a raw ZodError as its cause. The mismatch is invisible to typecheck: the schema and the gate that catches it are different files, and nothing at that seam disagrees. Only the integration suite saw it, and only after the schema moved. `instanceof` is the wrong question while the repo runs two zod majors, so the boundaries ask about shape instead. `isZodLikeError` lives beside the `ZodLikeError` interface that already models zod structurally for exactly this reason — the package deliberately imports no zod, because it is consumed by trees on either major. Applied at the three shared gates: the anomaly-rules router (whose two config schemas are now on different majors), the tRPC error formatter, and the Hono error handler.
…d settle the #6555 nits (#6701) * test(event-sourcing): pin that a bisected descent emits in the queue's order Follow-up to #6555, which asked for a sanity check that bisection also holds FIFO. It does, but nothing asserted it: the existing check proves each sub-batch is internally ordered and contiguous, which a descent that ran the right half first would satisfy while folding later events before earlier ones. Sends the batch id-shuffled with one shared score, so every payload becomes due together and coalesces into a single root, and `sendBatch`'s positional tiebreak makes the queue's arrival order differ from the id order — a bisector keyed on anything but the queue's sequence is caught. Asserts the global emission order across sub-batches, plus that a batch larger than the handler's limit was actually split, so the test cannot quietly go vacuous. Verified live by reversing the descent to right-half-first, which fails it: expected [6,3,4,1,7,0,5,2] to deeply equal [5,2,7,0,4,1,6,3] Two earlier drafts were vacuous and are worth naming. Spacing the scores to defeat the positional tiebreak made the jobs due a second apart, so they never coalesced, every job took the single-payload path and bisection never ran — the reversal mutation passed against it. FIFO ordering is by the queue's sequence, which is score first and send position as the tiebreak, not by anything in the payload. Test-only; no source change. * docs(event-sourcing): specify batch bisection, and bind its scenarios to the tests Bisection shipped with no feature file. Nine behaviours were covered by tests and by nothing a reader would find first — the split itself, the ordering guarantee, the continuation flag, the split budget and its kill switch, and the commit rule that records recognised rather than freshly-folded ids. The sibling queue guard has its own spec; this gives bisection the same. Also the only written home for LANGWATCH_GQ_BISECTION_SPLIT_BUDGET, which was documented nowhere despite being the lever for turning bisection off without a deploy. Every scenario is tagged and bound to the test that proves it, so the file enforces rather than describes: check-feature-parity reports 9/9. Written without incident dates or measured volumes, unlike some of its neighbours. One binding silently failed at first and is worth naming: a single-quoted @Scenario whose title contains an apostrophe ("the queue's order") terminates early and matches nothing. It reads fine and the suite stays green — only the parity check catches it. * refactor(event-sourcing): narrow the coalesced-batch wrapper instead of asserting through it Closes #6699 — both review nits from #6555. The batch wrapper carried three non-null assertions (`processBatch!`, `r!.clean`, `survivors[index]!`). They were correct but load-bearing: the homogeneity guard proves every entry resolved, yet TypeScript cannot narrow an array through a boolean, so the proof lived in a variable and a future edit to that guard could not fail the build. Unroutable payloads are now rejected up front. `rejectUnroutableJob` returns `never`, so that genuinely narrows the list for the compiler and the three assertions delete themselves — the guard now proves its claim in the type rather than around it. Behaviour is unchanged: a null entry could only ever reach the heterogeneous branch, which rejected it there anyway. The lint gate records one violation removed and none added. The second nit was a question — whether the store context has `.attempt` — and the answer is that it deliberately does not: `JobDelivery` describes one delivery so a bare `attempt` reads fine, while the store context is a grab-bag also carrying `aggregateId`, `tenantId`, `key` and `retentionPolicy` where it would not say attempt of what. Keeping the prefix and writing that down once at the type, rather than renaming, since two readers have now had to re-derive it. * docs(event-sourcing): state the bisection scenarios inside the budget that bounds them An adversarial pass over the scenarios written last commit. Two of nine were false in a state their own Given permitted, and three more carried a clause nothing asserted — the failure mode being that a wrong scenario does not fail, it becomes the definition of passing. The budget check runs on every sub-batch, before the descent can reach a single payload. Spend it mid-descent and the remaining chunk is abandoned un-split, so a payload sequenced BEFORE the offender can go uncommitted and the offender is never isolated, never named. "Payloads ahead of an unprocessable one still commit" and "an oversized batch converges by halving" both claimed to hold universally; they hold within the budget, and now say so. The interaction is documented in the feature's preamble, including that a budget below log2(coalesceMaxBatch) makes isolation unreachable for a full batch. Dropped three clauses that asserted nothing: - "named as the cause" — the code logs the offending staged-job id and stamps the span, but no test observes either; - "their commits extend the applied-event-id set rather than replacing it" — the bound test observes the delivery flag only, and the extend-vs-replace behaviour belongs to the scenarios that actually check it; - "the failure is surfaced immediately" — unobservable; now says what the test checks, that no further handler call follows. And "no payload is committed more than once" became "no payload ahead of it is applied more often than its peers", which is what the assertion says: counts rise together under legitimate redelivery, so equality is the invariant, not one. Also removed "a descent that took the second half first would be rejected" — a statement about the test's sensitivity, not about the system. The bindings themselves held: every scenario was mutation-checked by breaking the behaviour it names and confirming the bound test goes red — interleaved halves, reversed descent, budget forced to zero and to a million, the continuation flag dropped, and fresh ids passed where delivered ids belong. Still 9/9 bound. * refactor(event-sourcing): derive the batch entry once, and fail over-delivery as a diff Two review nits on this PR, both real. `first.entry` inside the homogeneity closure was safe only because `!batchHandler ||` short-circuits ahead of it — the reader has to reconstruct that argument to see it. Deriving `firstEntry` from `routed[0]?.entry` once makes the guard locally provable, which is the whole point of removing the assertions in the first place. The FIFO test waited on an exact length, so an over-delivery could never satisfy the wait and would have surfaced as an opaque 30s timeout rather than the array diff sitting right below it. Waiting for at-least-8 keeps the exact-order assertion as the thing that reports the failure. * docs(event-sourcing): state the real cost of isolating an offender The budget note claimed a budget below log2(coalesceMaxBatch) put isolation out of reach for a full batch. It breaks at exactly one value: splitting takes the floor of each half, so a 500-payload batch reaches a singleton down its left spine in 8 splits, and log2(500) is 8.97 — a budget of 8 isolates while the sentence says it cannot. State the cost as what it is: one split per level of the path down to the offender, floor(log2(coalesceMaxBatch)) or one more depending on which side of each halving it falls, plus a full descent for every additional offender in the same batch. * test(event-sourcing): stage the FIFO batch before the consumer exists Main landed `stageThenConsume` and the past-dated scores for the four pre-existing coalescing tests while this branch was open, fixing the same flake independently and with a better-informed comment: it names the coalescing drain (`ZRANGEBYSCORE jobs -inf now`) as what makes a consumer waking mid-spread take a prefix of the group, and notes that `sendBatch` is one atomic Lua call so there is no half-staged group either. That version is kept wholesale. Only the FIFO test added on this branch was left on the old pattern, since it does not exist on main. It moves to the same helper. Its scores stay tied where the others are ordered — that is the point of the test. A shared score hands ordering entirely to `sendBatch`'s positional tiebreak, which is what lets the send order differ from the id order and catches a bisector keyed on the payload rather than on the queue's sequence.
* fix(lint): let Biome run from the repo root
`biome check` from the repo root has never worked. platform/app/biome.jsonc
is the only Biome config in the tree and it is not at the root, so Biome 2
makes the invocation directory the workspace root -- its built-in defaults,
there being no config file there -- and then rejects the config below it:
× Found a nested root configuration, but there's already a root
configuration.
It exits before checking a file. Every scripted run (`pnpm lint`, `lint:fix`,
`lint:plugins`, `format`, the CI delta gate) invokes Biome from platform/app,
so none of them ever hit it; editors, agents and anything run from the root
hit it every time.
Add a root config that is a marker rather than a ruleset, and mark the app
config nested. The rules stay where they are.
The root's scope is platform/app and nothing else, so a run started from the
root checks exactly what `pnpm lint` checks. Left unscoped, the rest of the
repo -- packages/, sdks/, services/, mcp/, tests/ -- comes in under the
DEFAULT ruleset, none of it having a config: 58 diagnostics in packages/api/src
alone, mostly formatting. Noise on `check`, and damage on `check --write`,
which would reformat packages/ from tabs to the default spaces in a tree no
CI job gates.
The delta gate has to copy the root config into its base tree alongside the
app config. A nested config whose root is missing does not fail -- Biome
silently falls back to its defaults, turning every disabled rule back on and
reformatting to the default style. The base tree is checked out from the merge
base, which predates the root config, so without this the gate lints the base
under defaults and reports every real diagnostic in the head as new: 179 on
the subset used to verify it, and CI red on a PR that changed nothing.
Verified: over the full `pnpm lint` path set (./src ./ee ./scripts ./e2e
./prisma ./vite), all 3487 diagnostics are byte-identical before and after,
same paths, rules, lines and messages. The delta gate runs end-to-end clean
(base 179, head 179, no new violations), and fails with 179 phantom
violations if the base-tree copy is removed.
* fix(lint): name the missing-root failure, and correct a comment it staled
Two follow-ups from auditing the previous commit.
The gate's `cp` of the root config runs under `set -euo pipefail`, so a
missing /biome.jsonc aborted it with a bare "No such file or directory".
That is the one failure this whole change exists to make loud -- without
the root config Biome lints under its defaults in silence -- so it gets a
named error saying what went wrong and what it costs.
The `BIOME_PATHS` comment in langwatch-app-ci.yml said Biome's project root
is the app directory, "where biome.jsonc lives". As of the previous commit
the project root is the repo root and there are two configs, so the reason
packages/ is out of scope is now the root config's includes rather than the
directory the run starts in. Restoring that coverage is a different job than
it was, too: a nested config plus an includes entry, not a second root.
#7112) `test-component` fails intermittently with what reads as a hang, and is not one. vitest.component.config.ts spreads `...unitTest`, so it inherits `globalSetup: ["./src/test-unit-global-setup.ts"]` and with it a DEFAULT_HARD_FLOOR_MS of 4 minutes. That number is sized for a unit shard, and the file says as much: "A healthy unit shard finishes in ~3 min, so a 4-min floor only fires on a wedge." A component shard does not finish in ~3 min. Measured over recent runs the median is 297s — 4.95 minutes — so the floor fires on a HEALTHY shard, part way through its files, whenever it lands on the slow side of its own average. The floor already says this in the log, and the message is worth quoting because it is what stops the next person debugging a phantom infinite loop: [unit globalSetup] the shard still had files to start, so the floor cut a run that was working rather than one that was wedged. Read that as a shard too slow for the floor, not as a hang. Seen on main as well as on pull requests — one of the last three main runs lost shard 2 this way — so it is intermittent rather than newly introduced, and it takes `langwatch-app-complete` with it every time because that is a required check. Fixed with the knob the setup file already exposes for exactly this, LANGWATCH_UNIT_HARD_FLOOR_MS, set on the component job only: 12 minutes. The floor keeps doing its job — a finalize wedge is still cut well before the 25-minute job cap, which is the outcome it exists to avoid — while a healthy 5-minute shard is left alone. The unit lane keeps its 4-minute default, since its own shards really do finish in ~3. Its own change because it is unrelated to anything else in flight and every PR is exposed to it.
chore: sync model registry Updated llmModels.json with 441 models. Co-authored-by: langwatch-app[bot] <langwatch-app[bot]@users.noreply.github.com>
#7113) The Ops -> Migrations page shipped in #7079 with its page module and its sidebar entry, but no entry in routes.tsx. Routing here is a hand-maintained table, not filesystem-based, so the link rendered, was enabled, was correctly labelled — and landed on the 404. The page module typechecked, the href is a plain string, and nothing in CI rendered the menu and followed it. Registering the route is the fix. The guard is the point of the rest. The guard resolves each sidebar href the way React Router would, via matchRoutes, rather than asking whether it matches some pattern in the file. That distinction is load-bearing twice over: the table ends in a `*` route that renders the 404, so every path on earth matches something, and `/ops` also matches `/:project`, which is a different page. A first cut using matchPath passed with the route deleted; the ranked version fails with `expected '*' to be '/ops/migrations'`, which is the assertion worth having. A sweep of the rest of src/pages found no second instance — every other unrouted page file resolves through a directory-index import. Spec: specs/ops/ops-navigation-reachability.feature, 2/2 scenarios bound.
…odule graph (#7107) * perf(test): fix the integration teardown that blocked a shared module graph, and record what still blocks it Import is the largest line item in the integration lane — ~216s against ~203s of actual test execution on a CI shard, 42% of the lane's runner time, ~1.6s per file rebuilding the same Prisma client and the same server graph. Sharing the module registry reclaims most of it, and the config has carried a note for a while saying the way in is to give the app container an explicit reset between files. That note is half right, and this change proves which half. FIXED: the teardown. `setup.ts` is a setup FILE, so its `afterAll` runs once per test FILE, and it was disconnecting Prisma and quitting the app-layer Redis — the two singletons a shared graph exists to keep. With a fresh registry that is correct and load-bearing (the sockets would otherwise pin the worker open past the last test and the CI step would hit its job cap). With a shared registry it is exactly wrong: the singletons outlive the file that closed them and the next file resolves a disconnected client. That is the "first file's teardown takes the next file's client with it" failure. Now the per-file teardown resets only the App container, whose lifecycle genuinely is per file, and the process-wide clients are closed once at worker exit with their sockets unrefed first. With that in place the sharing works, measured against native local services: src/app/api (49 files) 138.8s -> 43.9s import 72.1s -> 11.9s (-84%) ee/governance (23 files) 44.1s -> 17.3s import 23.1s -> 9.9s (-57%) STILL BLOCKED, and not by anything a teardown reaches: `vi.mock`. Eight files in the app/api slice fail with the graph shared and every one of them PASSES ALONE. They are contaminated, not broken. Vitest hoists a module mock per test file and applies it while building that file's registry, so when the registry is shared and an earlier file already instantiated the real module, the mock never takes and the test calls the real collaborator — hence `ECONNREFUSED ::1:5560` and `expected 500 to be 200` rather than anything about containers or clients. 123 of the 414 integration files call `vi.mock`. That is not a set to rewrite, so a single `isolate` for the whole lane cannot be the answer. The shape that would work is a partition, exactly like the component/datastore split in integrationLanes.ts: files that mock keep a fresh registry, files that do not share one — ~291 files taking the speedup above. That needs a second vitest project and its own CI lane, and belongs in its own change. So the flag ships OFF and the lane behaves exactly as before. Verified by capturing the failing-file SET rather than a count: with the flag off it is byte-identical to the pre-change baseline over src/app/api. (One file, workflows-api, appeared in one run and not the next in both configurations — pre-existing flake, not this change.) Left wired up rather than deleted so the next attempt starts from the measurement instead of the assumption the old note recorded. * perf(test): let the 331 integration files that mock nothing share a module graph Import was the largest single line item in the integration lane: ~216s against ~203s of actual test execution on a CI shard, 42% of the lane's runner time, roughly 1.6s per file rebuilding the same Prisma client and the same server graph. This reclaims most of it for the files that can take it. Two things stood in the way, and only one was the one the config note named. The teardown, now fixed. `setup.ts` is a setup FILE, so its `afterAll` runs once per test FILE, and it disconnected Prisma and quit the app-layer Redis — the two singletons a shared graph exists to keep. With a fresh registry that is correct and load-bearing, since those sockets would otherwise pin the worker open past the last test. With a shared one it hands the next file a dead client: the "first file's teardown takes the next file's client with it" failure. The per-file teardown now resets only the App container, whose lifecycle genuinely is per file, and the process-wide clients close once at worker exit with their sockets unrefed first. `vi.mock`, which no teardown reaches, and which is why this is a partition rather than a flag. Vitest hoists a module mock per test file and applies it while building THAT FILE's registry; share the registry and a module an earlier file already instantiated stays unmocked, so the test calls the real collaborator. The symptom is `ECONNREFUSED ::1:5560` and `expected 500 to be 200` — nothing about containers or clients, which is exactly what sent the previous attempt looking in the wrong place. So the datastore lane splits again, on the same terms as the component split it nests inside: 146 files that call `vi.mock` keep a fresh registry, 331 that do not share one. `include` and `isolate` come out of a single function so they cannot disagree — deriving them separately is how a lane ends up running the mocking files with a shared graph. The default with no lane selected is every file with a fresh registry, so a plain `pnpm test:integration <path>` on a laptop is unchanged and nobody has to know the split exists. CI keeps the same four runners, now two shards per lane. Measured on src/app/api (49 files) against native local services: baseline, one lane 49 files 138.8s shared lane 38 files 34.1s mocking lane 11 files 32.9s which is 4.1x on wall clock for that slice and 2.1x less total runner time. Verified by comparing failing-file SETS rather than counts, on two independent slices. src/app/api and ee/ each produce a union across the two lanes that is identical to the pre-change baseline: zero new failures, nothing dropped. (Both slices carry pre-existing failures — anomalyRule.thresholdConfig among them, which is the zod dual-major boundary issue and fails on main too.) The partition itself has 11 unit tests, including that the two lanes are a total and disjoint cover, and that an unreadable file goes to the isolated lane because unreadable is not evidence that sharing is safe. * fix(ci): make the integration duration artifact name lane-specific Both lanes number their shards from 1, so `shared 1/2` and `mocking 1/2` would both have uploaded `shard-durations-integration-1`, and upload-artifact rejects a duplicate name within one run — the durations job would have failed on every dispatched run. The lane is now in the name. The merge job globs `shard-durations-*`, so it picks both up unchanged. Also two house-style fixes in the partition's tests: `writeFile` takes an object rather than two positional parameters, and the last nested `describe` states a condition rather than a topic. * style(test): run biome over the files this branch touched The caller rewrite in the partition tests was done with a script, which is reliably how line width ends up wrong here. `lint` caught a format error in integrationModuleGraph.unit.test.ts; setup.ts had one too. Both are formatter-only. 11 tests still pass.
…links to its traces (#7067) * feat(ai-gateway): virtual keys can auto-expire, and every usage view links to its traces A virtual key can now carry an expiration date. The gateway refuses it after that moment with its own error code, virtual_key_expired, and the key stays ACTIVE so extending the date puts it straight back in service with the same secret. Every place that shows a key's usage now links to the traces behind it: the Usage page, the key detail header, and the usage block. The links carry the period the reader is looking at, and the model when one is picked. The key detail page also tells the truth about scope and routing: the routing policy is named and links to itself, and the provider panel lists what the key may use rather than what its scope reaches. * fix(ai-gateway): an expired key stays editable, and the 403 code decides the rejection CI and review follow-ups on the virtual-key expiry change. CI: - gitleaks flagged the sequential-hex HMAC fixture in the new vk-expiry route test. A `gitleaks:allow` comment only counts on the line of the finding, so the file joins its two sibling route suites in the .gitleaks.toml allowlist instead. - The Biome gate counted four new violations. The expiry guard in both drawers moves into `expiryIncompleteReason`, `baseVk` reads its status from a map instead of nested ternaries, and `resolveEligible` takes named parameters, which is the repo rule anyway. Review: - The edit drawer resent the stored date on every save. On a key that had already expired the server refused it, so an expired key could not be renamed or extended: exactly what a date rather than a status is for. An untouched block now leaves `expiresAt` out. - The gateway read the 403 reason with a substring scan over the whole body, so a message naming another code decided the answer. It reads the decoded `error.code`. - A custom date that names a day that does not exist (2026-02-31) was rolled forward into the next month. It is refused. - Docs: `expires_at` joins the documented list shape, the offline window states the auth-cache hard grace rather than only the JWT TTL, the removed bootstrap claim was never implemented, and the detail page sentence separates active from expired. - The virtual-key timestamps carry `format: date-time` in OpenAPI. - Two test blocks stated conditions their cases contradicted, and the precedence case asserted tab membership rather than the badge. * fix(ai-gateway): an expired key stops at its date, even during a control-plane outage The expiry check only ran at resolve-key. The gateway's auth cache could not enforce a date it did not carry, so a key that expired at 12:00 kept calling providers until about 18:15 if the control plane was unreachable across that instant: the token still had up to 15 minutes on it, and the stale-while-error window added 6 more hours on top. The token now ends at the key's expiration date when that comes first, and carries the date as a vk_expires_at claim. The gateway caps both auth-cache deadlines at the same instant and refuses a request past it with virtual_key_expired, with no control-plane round trip. Revoked and disabled keys keep the full grace window by design: a revocation happens after the token was minted and cannot be predicted. The operator opt-out for both is a negative LW_GATEWAY_AUTH_CACHE_HARD_GRACE_SECONDS. A key with no expiration date keeps exactly the behaviour it had. * chore(ci): allowlist the gateway JWT suite's signing fixture * chore(ci): scope the gateway fixture allowlist by value, and name the field gitleaks reads The allowlist matched by path alone, so any generic-api-key-shaped string added to one of those test files was allowed with it. It now matches path AND the anchored fixture value, verified both ways: the fixture stays allowed, and a foreign key planted in the same file is still reported. The AND field is `condition`. `matchCondition` is not a name gitleaks knows and an unknown key is dropped in silence, which left the CREDENTIALS_SECRET entry as a bare path allowlist over the whole workflow file. Measured on gitleaks 8.30.1: `condition` narrows, `matchCondition` does not. The two sibling route suites that sign with the same fixture are listed too, so a change to either does not meet the same wall. Test helpers take named parameters, per the repo convention for more than one argument. * chore(ci): anchor the workflow fixture value in the gitleaks allowlist * fix(ai-gateway): a changed expiration date reaches the gateway on the config channel The auth cache took the key's terminal expiry only from the token claim in the bundle it held. The config-TTL refresh copied that bundle and swapped only Config and Credentials, so it wrote the old date back. An admin who shortened expiresAt while the change feed was unavailable was then followed only when the token expired, up to 15 minutes later, and the grace paths kept keying off the old date. The date now rides the config channel as well. The config payload carries expires_at as unix seconds, null for a key that never expires, and the gateway's config revalidation writes it onto the bundle before storeL1 recomputes both deadlines. A shortened date therefore lands within one ConfigTTL, 60 seconds by default, and an extended one lands too, which removes the single 403 a request used to get at a boundary the control plane had already moved. The token claim stays the mint-time floor, so a control plane the gateway cannot reach at all still stops the key at the last date it was told. A response with no expires_at field comes from an older control plane and leaves the cached date alone. Reading absent as "no expiry" would lift the cap off a key whose own token says it expires. * chore(prisma): renumber the virtual-key expiry migration above main's newest Migrations run in the order their names sort, so a migration numbered below one that already merged is skipped outright on any database that has run the newer one. It then passes locally and never runs in production. main gained 20260817000000_system_migration_tenant_state while this branch was open, so 20260816130000_virtual_key_expires_at moves to 20260817000001. The SQL is unchanged. The migration has never run outside local databases, so the directory name is still free to move. * fix(errors): a validation failure from either zod build reaches the customer as one zod 3.25 ships two builds in one package, the classic export and the `zod/v4` subpath, and each has its own `ZodError` class. The app imports the classic one and `@langwatch/langy` compiles against v4, so a schema built by one build raises an error the other build's `instanceof` rejects. Four boundaries tested class identity: the tRPC error formatter, the Hono error handler, its request logger and the anomaly-rule router. A validation failure the customer could fix then fell through all of them and was reported as an unnamed 500, logged at error against the 5xx budget. Deduping the package cannot fix it, because `zod` and `zod/v4` are one package and two class identities. So the boundaries now recognise the error by shape: `isZodLikeError` in `@langwatch/handled-error`, which is where the version-agnostic `ZodLikeError` type already lives. Both builds set `name` to "ZodError" and expose `issues` and `flatten()`, which is all these callers read. This is what turned `anomalyRule.thresholdConfig.integration.test.ts` red on main: the shard it lands in decides which build wins, so the file passes alone and fails beside its neighbours. A path segment is stringified before it is joined, because a zod 4 path may hold a symbol and `Array.join` would throw on one inside the handler whose job is to stop a throw reaching the client. * fix(ai-gateway): refuse a virtual key at its expiration instant The gateway cache and the control plane disagreed on the boundary. The control plane refuses a key when its date is at or below the current time, while Bundle.KeyExpired asked `now.After(date)`, so the exact expiration instant served from cache and was rejected by the control plane. It now asks `!now.Before(date)`, and a domain test pins the three instants around the boundary. Also from review: - The vk-expiry route suite signs a JWT in the cases that expect a 200, but only set LW_GATEWAY_INTERNAL_SECRET, so it passed only where LW_GATEWAY_JWT_SECRET happened to be in the environment. It now saves, sets and restores both, the way the vk-lifecycle suite does. - The expiration drawer recomputes the earliest selectable date every render. Memoised once, a drawer left open across the UTC-day boundary kept offering a date the server refuses. - The disabled-key scenario now states that the key has a reachable trace destination, which is what its fixture gives it. Without that the spec required "View traces" for a key the other scenarios say must not offer it. - BDD context blocks around the loose cases in the expiry, traces-href and route suites, and a hyphen in the docs. The component test lane runs on three shards rather than two. The hard floor in test-unit-global-setup fires at 4 min and is calibrated for a lane whose healthy shard takes ~3 min; measured on 540 files over two shards, shard 1 spent 3m34s in the test step, so 26 seconds separated a green lane from one the floor cut mid-run. Six more files crossed it, and the floor reports that as a shard with no failed test in it, which reads as a wedge rather than as a lane that outgrew its split. * refactor(ai-gateway): type the virtual-key detail page's provider props Both `as any` casts on `orgProvidersQuery.data?.providers` were unnecessary: the query already returns the shape the preview and summary components declare, so the casts only switched checking off. Removed. The third cast, on the whole key passed to the edit drawer, was hiding a real mismatch. The drawer's status union left out "disabled" while VirtualKeyCamelDto has all three values, so the drawer's type said a state the page can pass could not arrive. The union now matches the DTO. What still needs a cast is one field: the DTO types `config` as `unknown`. The cast is now to the drawer's own exported type rather than to `any`, so every other field on the object is checked again, and the comment names what would remove it. Also corrects the reason on the expiration floor. The picker's minimum sits a whole day ahead, which absorbs a single UTC-day boundary: the stale value then names today, and today resolves to 23:59:59.999Z, which the server accepts. A refusal needs the drawer open for more than a day. The comment claimed the shorter case, which the code does not do.
…ssing from file size (#7115) * perf(test): commit measured shard weights, so the sequencer stops guessing from file size vitest.durations.json has never existed, so WeightBalancedSequencer has been falling back to byte size for every file in every lane (shardWeights.ts). Byte size is a proxy for how long a file takes, and a poor one — a short file that replays migrations costs more than a long one asserting on strings. Generated by the `shard-durations` job, which exists for exactly this and had never been run. Measured on the branch that introduced the shared/mocking split, which matters: with a shared registry a file's duration excludes most of its import cost, so weights taken before the split would describe a world that no longer exists and would mis-balance the shared lane while looking authoritative. 1,503 entries. Coverage is 314 of 331 shared files and 142 of 146 mocking ones. WHAT IT ACTUALLY BUYS, which is less than I expected. Predicted spread between the two shards of a lane, scoring each split by its real measured cost: shared lane by byte size 226s | 252s spread 26s by duration 239s | 239s spread 0s mocking lane by byte size 93s | 108s spread 16s by duration 101s | 101s spread 0s So ~26s and ~16s, not the ~1 minute the observed CI spread suggested. The 8m54s-vs-6m43s gap seen on a recent run is mostly per-shard fixed cost and runner variance, not weight imbalance: the whole shared lane is 478s of measured file time against shards that run 7-9 minutes, so most of a shard is container startup, migrations, install and first import — identical in both, and untouched by how the files are divided. The lasting value is that shard-count decisions become measured rather than guessed. Splitting a lane further, or merging two, can now be argued from the manifest instead of from file sizes. Unit coverage is partial: the dispatched run was superseded and cancelled once its branch merged, so only shard 4 of 4 reported. The mixed case is handled — shardWeights scales byte-derived weights onto the same footing as measured ones — and the next dispatch on a quiet branch will fill the rest. * chore: mark the shard duration manifest as generated vitest.durations.json is 1,505 lines of machine-written timings, regenerated wholesale by the `shard-durations` job. Collapsing it in review is the point: it is never read by a human and it would otherwise bury the change it arrives with. `merge=ours` for the same reason the lock files have it. A conflict here is two sets of measurements, neither more correct than the other, and resolving it line by line is meaningless — the answer is always to re-measure, not to merge. The manifest also has to be byte-identical for every shard that reads it, which a hand-resolved merge cannot promise.
* feat(event-sourcing): transient process evolutions commit intents without a transaction An evolution that keeps the initial state and arms no wake has nothing to lose between two writes, so it no longer pays for a transaction, an advisory lock, a revision compare-and-swap, an instance row or an inbox row. Its intents go down as one idempotent multi-row insert, and the outbox's existing (processName, projectId, messageKey) uniqueness is the consumption record the inbox marker used to be. Transience is a property of the evolution rather than of the process, so one process manager can be transient for the keys that hold nothing and durable for the keys that hold a buffer or a deadline. No process declares it yet, so this is behaviour-preserving. * feat(gateway): stop minting a durable process row per LLM call The gateway spend pipeline's aggregate is one per gateway REQUEST, so the framework keyed a process instance per request, and three process managers were mounted on it. Each gateway request therefore wrote three permanent rows into `ProcessManagerInstance` — the one table the retention sweep deliberately does not touch, on the documented premise that it is "bounded by entity population rather than by traffic". A row per LLM call made that false, and the spec that granted the exemption named this exact exit condition: it holds "until something makes it grow". Rather than sweep harder, stop writing the rows. - The outcome commands now carry the attribution the admission carried. `outcomeFor` fills it from the same `call.Bundle` the admission reads, so the two records cannot disagree and end-user resolution runs once. That is the only reason the debits and delivery processes kept state at all. - `gatewayDebits` and `webhookDelivery` are declared transient: an outcome that states its own attribution mints its intent and keeps the initial state, so nothing durable is written. Their per-request instances go away; webhookDelivery's per-endpoint streams keep their buffer and their wake. - `spendSettlement` becomes ONE scheduled sweeper for the install. The fold already leaves a request at `admitted` until an outcome arrives, so "which requests are still open" is a query rather than a timer per request. Per LLM call, on the process-manager path: 6 transactions and ~35 statements become 0 and 2, three permanent rows and six 7-day inbox rows become none, and the outbox rows that represent real work are unchanged. Rolling upgrade needs no ordering. The admission declares whether its emitter repeats attribution on the outcome; admission and outcome always come from the same pod and build, so an older build keeps the durable join and a newer one skips it. The flag is removable once the fleet has cycled. * fix(gateway): satisfy typecheck, the biome delta, and module mocking Three CI failures, all in the new code: - The settlement sweeper's wake handler was written inline, so the builder inferred its intents from the handler itself and typed `ctx.intents.sweep` as possibly-undefined. Declared out of line with an explicit intents type, the way every other scheduled process does it. - `spendSettlement.process.ts` imported a CONSTANT from the ClickHouse repository, which pulled the ClickHouse client into the module graph of everything that reaches the pipeline registry — turning a `vi.mock` factory in the gateway-spend REST suite into a hoisting failure. The constant is settlement policy rather than a repository detail, so it moves to the process and the repository import becomes type-only. - Two new Biome violations: `webhookDeliveryPM` grew past the 60-line limit, and `enrichAttributedCommands` past the complexity limit. The three outcome handlers were identical apart from their payload builder, so they now share one; the ingest seam's gap reporting moves to its own function. * fix(gateway): update test fixtures for the attribution-carrying outcomes The typecheck:tests step is separate from typecheck, and these ten errors only surface there. The attribution fields default on the wire, but zod's output type makes a defaulted field REQUIRED, so every fixture that builds a confirm, fail or settle command has to state them. Two files spell them out; the fold's fixtures share one UNATTRIBUTED_OUTCOME constant, which also names what the case is exercising — an emitter that does not repeat attribution. The transient probe declared its own event types, which the pipeline's event union does not know. It now branches on the payload of the one registered test event type instead. The ProcessStore stub gained appendIntents, and the ingest test now asserts the enrichment fields are ABSENT rather than empty, which is what skipping enrichment leaves. * fix(gateway): keep the model on settled envelopes, and settle per instance Two findings from the review, both real. A settled webhook envelope lost its model. On the durable path the delivery process read `state.attribution`, which came from the admission and carried the requested model. Now the outcome's own attribution wins — and settleSpend carried no model field, so every settled envelope shipped an empty one. The settle command carries `model` and `model_provider_id` now, the sweeper copies them off the spend record it already reads, and the comment that claimed this was true becomes true. `Promise.all` over the ClickHouse instances is fail-fast, so a single unreachable private instance rejected the whole read, failed the sweep intent, burned its attempts and kept failing every wake while that instance was down — taking the shared instance's open admissions with it. That contradicts the rule the sweep already states for one tenant's failure, so it now applies at the instance level: reachable instances settle, the unreachable one is logged and retried on the next sweep. * fix(event-sourcing): decide the transient path after evolving, not before The review found a real hole in the speculation, and it is my own invariant that was wrong. The fast path evolved against the INITIAL state and took the transient route when that came back state-unchanged and wakeless, on the claim that such an evolution "provably did not read the store". That is not provable. A handler may preserve its state while deriving its INTENTS from that state: the speculative run then looks transient and mints the wrong intents — or none. Both process managers here happen to stash when they lack what they need, so neither exhibits it, but nothing stopped the next one from doing so and it would have failed by silently dropping work. The evolution now runs once, against the real previous state, and the path is chosen from the result plus the absence of an instance. That costs one indexed lookup and keeps every saving that motivated the path: no transaction, no advisory lock, no compare-and-swap, no instance row, no inbox row. A test pins the exact shape, and fails without the change. Also from the review: the two commit paths built identical outbox rows in two places, which would drift the moment a column is added, so they share one mapper; and the debits payload builder takes named parameters. * fix: address the remaining review threads on the gateway PR - The sweeper test copied the private `__schedule_arm` literal instead of the runtime's constant, so the two could drift. The constant is exported and imported. - The sweep's cap doc block promised a report it never emitted: a full page meant the sweep had not finished and an operator saw only a steady settled count. It warns now, and so does a sweep with failures. The cap moved to the process that reports on it, because this file must not import a value from the ClickHouse repository. - The metadata echo passed `json.Valid`, which accepts `[1,2]`, `"x"` and `3` — none of which the ingest schema's "must be a JSON object string" refinement accepts, and none of which respect its 4096-byte bound. A record carrying one was rejected whole at the control plane, so the emitter traded a dropped echo for a dropped billing record. It now checks what the schema actually checks. - Object parameters for the transient test's envelope helper. * fix(gateway): reject null metadata echoes and cap the sweep after the merge - validMetadataEcho accepted JSON `null`: it unmarshals into a map without error and leaves it nil, so the emitter forwarded "null" and ingest rejected the whole spend record on `parsed !== null`. Requires a non-nil object now. - MAX_OPEN_ADMISSIONS_PER_SWEEP bounded each ClickHouse instance's query, so N instances handed the sweeper N times the cap. Re-applied at the merge, oldest-first across instances so the cap sheds the newest rows. * fix(gateway): release a stashed outcome on a self-describing admission The admit handlers dropped `state.pendingOutcome` when the admission declared its outcomes self-describing, on the grounds that a stash and that flag cannot coexist. The two conditions are not the same one: an outcome stashes on its OWN empty organization, not on the build that sent it. Where they disagree the stash was lost — no debit, no envelope, and a permanently stranded instance row, since that branch was the only thing that could clear it. Both handlers now release it against the admission's scopes. Also name DISPATCHED_OUTBOX_RETENTION_MS as load-bearing: a transient evolution writes no inbox marker, so the dispatched row is its only idempotency record, and 24h is shorter than the ~25h redelivery horizon the inbox's 7 days is reasoned against. What closes the gap is a precondition on .transient() rather than the window — intent sinks must be idempotent — now stated on the builder and in the spec. * refactor(gateway): team_id is nullable, and isDeepJsonEqual moves to json.ts team_id was empty-string-for-absent like its neighbours, but it never earns that compromise: the empty string is the wire contract's "not stated" only because the wire fields cross into Go, whose string zero value IS "". team_id never crosses that boundary — the gateway cannot see it and the ingest seam fills it. The rest of the stack already treats it as nullable (the gateway JWT signs `traceProject?.teamId ?? null`; budget resolution takes `teamId ?? null`), so this was the one layer coercing. Legacy "" still parses everywhere, since appended events and durable outbox rows carry it. Also moves isDeepJsonEqual out of processManagerService into json.ts, beside JsonValue and ensureJsonSafe, where a JsonValue helper belongs. Adds the JOIN section clickhouse-queries.md never had: join at write time, then IN, then smaller side right, pre-filter both sides, and match key types exactly (cast the parameter, never the column). See #7097, #7098. * Revert team_id to an empty-string default The nullable form was correct in principle — team_id never crosses the Go boundary, so it never earned the empty-string compromise the wire fields make — but it is not worth its blast radius: a schema change, four consumers, the process state, the ingest seam and two back-compat tests, for a field whose only reader already coerces it with `payload.team_id || null`. Keeps the two unrelated changes from the reverted commit: isDeepJsonEqual moves to json.ts, and clickhouse-queries.md gains its JOIN section. * fix(test): type the harness result before reading process state The capture harness types handler results as `unknown`, so reading `released.state.pendingOutcome` failed typecheck:tests (TS18046). Cast to GatewayDebitsState at the read. * refactor(gateway): split the settlement sweep's command and reporting out The cap-reporting branch pushed runSweep past the cognitive-complexity threshold, which the biome delta gate counts as a new violation. Extracts settleCommandFor and reportSweep; the sweep loop is now the loop. * test(gateway): cover the query that replaced the sweeper's durable rows The three scenarios this PR deleted — a confirmation standing the sweeper down, an outcome racing ahead of its admission, a duplicate wake — described the durable per-request row. They are now one behavior of the open-admission read: a request the query does not select. Nothing tested that. The gap was not academic. The fold writes one ReplacingMergeTree version per lifecycle transition, so a request that resolved still has its superseded `admitted` version on disk. A regression in the Status filter or the IN-tuple dedup passes every test in the repo and settles live requests, shipping `gateway.request.settled` for traffic that is still in flight. Drives ClickHouseOpenAdmissionFinder against real fold rows: an admission past its grace is returned with the attribution the settle command carries forward; a confirmed, failed or already-settled request is not; an admission inside its grace or older than the lookback is not; a twice-written admission is offered once; and a population over the cap yields the cap's worth, oldest first. Merges are held off for the fixture. Both dedup scenarios pass on their own once a background merge has collapsed the versions, which would have left them green for a reason the query had no part in. Each assertion was checked against a mutated query — dedup removed, Status filter widened, grace bound dropped — and each mutation fails at least two of them.
…ty by default (#7069) * feat(coding-agent): capture codex conversations and repository identity by default Every seam that persists the codex exporters now wires the turn harvest: the wrapper persist and the login telemetry refresh call assertCodexTurnHarvest beside the otel block write, the same as instrument. A harvest that cannot be wired prints what is wrong instead of staying silent, and a fresh install prints the backfill hint. The rollout parser reads session_meta (session id, cwd, git branch and repository url), and the harvest posts one langwatch.session_context log record per session, deduped by fingerprint through the shared hook state. The platform folds it with no receiver change, so a plain codex session shows its repository and branch and connects to its pull requests. resolveTarget reads the logs endpoint out of the config's otel block, and the wrapper passes it to the live streamer. Two trace-view defects found while dogfooding the capture: - Newer codex emits a session_task.turn rollup under codex_exec that repeats the usage its handle_responses spans already report per call, so every exec trace counted its tokens twice. The extractor now marks a usage-bearing exec rollup with skip_token_accumulation; the response spans stay the counted record on that wire, for old and new codex. - The terminal replay showed the redacted user_prompt event next to the recovered prompt. The derivation now drops withheld-text prompt stubs when a content span recovered the real prompt, and reads a withheld stub's chars from prompt_length instead of the sentinel's own length. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(coding-agent): review findings on the codex capture path - a redacted prompt event is now matched to its recovered turn by character count instead of a trace-wide flag. Codex reports the real length on `prompt_length` even while it withholds the words, so equal counts identify the same turn, matched one-to-one. A trace whose rollout recovery reached only some turns keeps the stubs of the turns it missed, which were the only record those prompts happened - a state directory that cannot be written no longer takes the conversation with it: the fingerprint write is bookkeeping that saves one re-POST next turn, and the caller awaits it before posting the turn spans - login refresh asserts the codex harvest hook for any existing [otel] block, not only for a block whose endpoint drifted. A device already pointing at this login but predating the hook was left unable to capture content - `readRollouts` takes named parameters - the fetch double's comment says why URLs are recorded beside bodies Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * feat(coding-agent): name sessions by their first prompt when no generated title exists Most agents rarely generate a session title, so the sessions table read as a wall of untitled rows. The dispatcher now stamps a fallback title fact from the first real user prompt, the fold fills an empty title from it, and a generated title still overwrites it. Codex never generates a title and withholds prompt text from telemetry, so the rollout harvest reads the first typed prompt from the transcript and carries it on the session-context record. Machine-injected first prompts (tag-wrapped context, redacted stubs) name nothing. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(coding-agent): tolerate a membership row that outlived its user The schema has no foreign keys, so a TeamUser row can point at a deleted user. personalTeamOwnerNames read member.user.name unguarded, and one orphan row made the whole pull-requests page fail with a 500. The workspace now falls back to its own name, the same as a workspace with no member. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(coding-agent): pair a redacted prompt to its turn by moment, not length alone Two turns can type prompts of the same length. Matching stubs to recovered prompts on the character count alone then claimed whichever stub came first, so a trace that recovered only the later turn deleted the earlier one and showed the recovered turn twice: the same loss the count was added to prevent. A stub and the recovered span for one turn land milliseconds apart while separate turns are far apart, so the pairing now takes the nearest unclaimed stub of equal length, nearest-first and one-to-one. The pairing moves into its own function, which keeps the transcript builder under the complexity budget. The test fetch double takes named parameters and asserts the logs POST happened, so a missing record fails on that line instead of throwing a TypeError from `bodies[-1]`. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(coding-agent): mark the open Sessions and Pull requests destination in the rail The router resolves a pathname from a list of route patterns and falls back to the literal path when it finds none. Neither project route was in that list, so the rail compared /demo/sessions against /[project]/sessions, read false, and marked nothing while the reader stood on the page. The page title in the breadcrumb was missing for the same reason. The personal twins are registered above the project patterns, because ':project' captures 'me' and would otherwise resolve /me/sessions to the project pattern, which would break the personal rail and read a personal page as a project one. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(coding-agent): a prompt stub is only claimed inside its own turn's window Pairing took the nearest same-length stub at any distance. The log read is capped, so a recovered turn can arrive with no prompt event of its own, and that turn then claimed some older stub and deleted the one turn that stub was the only record of. A prompt event fires as its turn starts and the model-call span opens right after, so the two sit milliseconds apart. Two seconds bounds that without reaching the next turn. Past the window nothing is claimed, so being wrong costs a prompt shown twice rather than a turn lost. The rail's rule text describes what a reader sees instead of how the router names a route, and the router mock takes a top-level type import. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * test(coding-agent): pin the harvest to every seam that writes the codex exporters The exporters and the turn harvest are one wiring: the [otel] block on its own reports tokens and captures no conversation. All four seams pair them today, but that is a convention repeated at each call site, and two seams already shipped without the harvest. The check reads the CLI source, so a seam added later fails it rather than quietly capturing nothing. Also drops three imports shell-rc.ts no longer uses. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * test(coding-agent): hold each codex persist seam to a single write The pairing check reads a file at a time, so on its own it proves every file that writes the exporters also wires the harvest, not that every write is wired. Each seam persists in exactly one place today, which makes those the same statement. Holding them to that shape means a second write has to land here first, where the author wires the harvest for it. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(governance): catch zod v4 errors in the anomaly rules router Two zod entry points are live in this package while the migration to v4 finishes, and each carries its own ZodError class. The threshold config schema is built with zod/v4, but this router imported zod v3, so translateConfigValidationError never matched a threshold validation failure. The raw ZodError escaped and the admin got a wall of JSON in place of the validation_error copy. The guard now tests both classes. Both shapes expose issues, so the translation below it needs no change. This repairs seven failing tests in anomalyRule.thresholdConfig.integration.test.ts that main also carries. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD * fix(codex-capture): bound the context posts, and cover the two gaps review found The session-context records go out before the turn spans, because the title they carry is first-write. Posted one at a time, that made a slow logs endpoint a delay on the conversation itself, and the delay grew with the session count: `ingest codex --all` reads every rollout on disk, and each session waited the full 5 s per-post timeout before the next one started. The posts now run a few at a time under one wall-clock budget, so the wait is the budget plus the one post still in flight, whatever the count. A session the batch does not reach keeps no fingerprint and is offered again on the next harvest, the same path a refused POST already takes. Two test gaps closed with it. The streamer's per-tick context loop had no coverage at all, because every streamer test passed a null logs endpoint, so the claim that re-offering a session posts it once was unproven. And the 2 s window test read as passing on its prompt length rather than on the window, so its fixture length is pinned the way its sibling already does. Claude-Session: https://claude.ai/code/session_01XRYUK1mbAWQ18rUFdjminD
…7088) Bumps the react group with 2 updates in the / directory: [react](https://github.com/react/react/tree/HEAD/packages/react) and [react-dom](https://github.com/react/react/tree/HEAD/packages/react-dom). Updates `react` from 19.2.7 to 19.2.8 - [Release notes](https://github.com/react/react/releases) - [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md) - [Commits](https://github.com/react/react/commits/v19.2.8/packages/react) Updates `react-dom` from 19.2.7 to 19.2.8 - [Release notes](https://github.com/react/react/releases) - [Changelog](https://github.com/react/react/blob/main/CHANGELOG.md) - [Commits](https://github.com/react/react/commits/v19.2.8/packages/react-dom) --- updated-dependencies: - dependency-name: react dependency-version: 19.2.8 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: react - dependency-name: react-dom dependency-version: 19.2.8 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: react ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…cter typing (#7114) * test(automations): settle the Slack combobox typing before acting on it The picker suite drops a keystroke on CI. Twice observed, a different character each time: expected [ '#adoc' ] to deeply equal [ '#adhoc' ] expected [ '#ahoc' ] to deeply equal [ '#adhoc' ] The cause is already known and written down in the file: the combobox resyncs its input from a passive effect, so synthetic keystrokes can outrun React's effect flush. The mitigation was `userEvent.setup({ delay: 10 })`, a human cadence. That makes the race unlikely rather than impossible, and a fixed delay only moves the threshold — on a loaded runner it is still lost occasionally. What this changes is not the odds but WHERE the loss lands. Six call sites typed and then went on to tab, click or press Enter, carrying the corrupted value through three more interactions before asserting something else entirely. A dropped character then surfaces as a claim about which item holds a tick, which reads like a real defect in the component and is not one. `typeAndSettle` waits for the box to actually hold the typed text before the test moves on. `waitFor` retries, so it absorbs the race rather than betting on a delay, and any residual failure names the typing instead of the tick. The `{Enter}` case sends the key separately, because with `#adhoc{Enter}` in one call there is no moment at which the box should hold exactly `#adhoc`. The four sites that type and immediately assert the value are deliberately left alone: there the dropped character IS the assertion and should fail. NOT REPRODUCED LOCALLY, and worth being plain about. The unfixed suite passes 10 of 10 rounds on this laptop, and 8 concurrent copies competing for cores still pass, so the race needs conditions I could not create here. The fixed suite also passes 10 of 10, which therefore demonstrates nothing about the fix. The case for this change is the mechanism above and the fact that a test should not carry an unverified value into three further interactions — not evidence that the CI failure is gone. CI is the only place that can confirm that. * test(automations): enter the channel in one event where typing isn't the subject Settling on the typed value after the whole string could only rename the failure, not prevent it: the dropped character is gone from the combobox's own state, so no amount of retrying makes the box hold it. Five of the six sites are not about typing at all — they blur, press Enter or pick from the list, and typing was only how they got a channel into the box. Those paste instead. A single input event has no gap between characters, so there is nothing for a late resync to eat, and that holds without having to know what makes the resync late. The one site that IS about the per-keystroke path — the search filtering on the whole term rather than the last letter — keeps typing, now settling between keystrokes so a stale value is caught while it is still recoverable instead of being carried into the assertion. The two tests that type and immediately assert the value are untouched. There a dropped character is the assertion, and it should fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )