Repository navigation
Commit aa56857
feat: send the flow bundle as one xz document per side (#171)
* feat: send the flow bundle as one xz document per side
Large repos got no flows or merge risk: crate's bundle was 24.6 MB of gzip, over the service's
4.3 MB limit, because every fact went four times. 149,315 of its 149,316 analysis edges are call-graph
edges, nested relations repeat the top level, and the methods index and the graph's nodes are one
table.
- `side_document()` writes each commit as one document, `head.json` and `base.json`, in place of the
two analyses and two call graphs. It holds a row per method or graph node, calls by caller with
their lines, the component tree, and only the analysis edges the calls do not imply. The service's
`bundle.expand_side` reads it back into every field a feature reads.
- `_compress` is xz -9e instead of gzip: its window spans the bundle, so the base side costs little
next to a head that repeats most of it. The briefing request shares it.
- A changed file over 300 KB is sent at neither commit (the same limit as source).
On crate the bundle is 1.42 MB. Needs CodeBoarding/licensing-aws#56 deployed first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: send only the analysis edges that are no call
The service reads every call already, so the analysis edges that repeat one were rebuilt on its side
from leaf membership and then corrected with hidden_edges. Without them crate's guide holds the same
content and its peak memory drops from 1.8 to 1.3 GB, so extra_edges is now just the analysis edges
that are no call, and the derivation and hidden_edges go.
call_graph() no longer builds what side_document dropped: `callable`, and each site's file and
column. It returns each call's lines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: rename closed_features to remote_features and make it readable
The script calls the service's review-guide and architecture endpoints;
nothing about it is closed. scripts/remote_features.py replaces
scripts/closed_features.py, and the action step, problem_history's import
and the tests follow. The public input closed_features_source keeps its
name, since renaming it would break workflows that set it.
The rewrite uses descriptive names and short functions: side_document is
now a _Table of _Row objects plus _calls, _components and
_analysis_edges, and collect() reads changed files through
_changed_contents. The module docstring says what a request sends, why it
is packed as side documents under xz, and each size limit.
The side documents, collect() output and request body are byte-identical
to before on 42 cases, crate's head and base with its real call graph
among them, and call_graph() reads two real pickles the same.
One fix: a parent's members were left out when they equalled its leaves'
members, but the service rebuilds a parent from its direct children. A
parent holding fewer methods than a child read back with the child's
extra methods. It now compares with its children's members. No analysis
in the 42 cases hit it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: read the side documents' analyses through the installed engine
remote_features.py parsed the legacy analysis.json itself: methods_index,
each component's file_methods and every relation's all_edges. It now reads
an analysis only through the pinned release's parse_unified_analysis, the
same engine that unpickles the call graph beside it. The review analyses
both commits with that release, so both are always in the format it reads,
and a format change in Core needs no change here.
The check that a methods_index key is its file and qualified name goes:
the keys are built from Core's method entries.
InstalledCoreTests runs in the core-compatibility job against the pinned
release. tests/fixtures/analysis holds one analysis as Core 0.15.5 and Core
main (9cf50fb9, after #571) wrote it. Each release reads only its own, and
every fixture it reads must give the same side document.
Checked: on CodeBoarding's own analysis (two baseline syncs, 153 files,
22 components) and this repo's, the side document is byte-identical to the
one built from the raw document, and Core main reading the v1 rewrite gives
the same bytes as 0.15.5 reading the legacy original.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: check requests against both of the service's size limits before sending
The service refuses a request on two sizes: what is sent (4.3 MB of xz, the Function URL's limit) and
what it decompresses to (256 MiB of JSON for a feature, 32 MiB for a briefing). The Action checked only
the first, so a briefing of 32-43 MiB of JSON, which xz takes under 4.3 MB, went out whole, was refused,
and went out again on every sync.
_encode now enforces both in one step: it measures the JSON, which is cheap, then compresses once. The
limits sit together and name the service's constants they follow. The briefing leaves findings out
against the JSON limit first, largest first, counted per entry, and only recompresses when the
compressed size is still over, aimed by the ratio it just measured, at most 3 compressions in all. It
used to recompress once per dropped finding.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf: compress each feature request once, one at a time
The guide and architecture each compressed their own body in parallel, nearly the same bytes twice,
and architecture compressed again after dropping source.json. At preset 9 | EXTREME each compressor
takes about 700 MB.
Both features can send the body without source, so it is compressed once and carries why source is
missing, which only architecture reads. Architecture's body with source is compressed only when
there is source; when it does not fit, the shared bytes stand in. A run compresses once without
source and at most twice with it, and never two at a time.
The JSON check needs no ratio guess: the 256 MiB limit is 62 times the 4.3 MB one, so it already
turns away whatever could only fit at better than 62:1, and no tighter cut-off holds for every
request.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: build the review diff from Core's parsed analysis, not the document
load_analysis already parsed both sides with the pinned release's
parse_unified_analysis and rolled relations up with its
project_relations_to_level, but the diff then read members, file hashes and
method hashes from the raw document: file_methods as {file_path, methods}
groups and methods_index. Core #571's "v1" document has neither, so once the
Action pins a release with it, the review diff fails with "'str' object has
no attribute 'get'".
The diff now reads only what Core parsed: each component's file_methods,
the root's file index and Core's projected relations at every level. The
review analyses both commits with the pinned release, so both sides are
always in the format it reads, and a format change in Core needs no change
here.
The core-compatibility job rendered the committed .codeboarding/analysis.json,
which after a pin moves can still be in the format of the release before,
which the new one refuses. It now runs InstalledCoreTests: tests/fixtures/
analysis holds one analysis as Core 0.15.5 and Core main (9cf50fb9) wrote it,
each release reads only its own, and every fixture the pinned one reads must
give the same view.
Checked on CodeBoarding's own analysis (two baseline syncs, 22 components,
depth 1 and 2) and this repo's: output and summary are byte-identical to
main's script on the legacy documents with 0.15.5, and to that again on
their v1 rewrites with Core main, where main's script fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>1 parent 5334187 commit aa56857
7 files changed
Lines changed: 1103 additions & 538 deletions
File tree
- .github/workflows
- scripts
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
45 | 47 | | |
46 | 48 | | |
47 | 49 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
736 | 736 | | |
737 | 737 | | |
738 | 738 | | |
739 | | - | |
740 | | - | |
| 739 | + | |
| 740 | + | |
741 | 741 | | |
742 | 742 | | |
743 | 743 | | |
| |||
777 | 777 | | |
778 | 778 | | |
779 | 779 | | |
780 | | - | |
| 780 | + | |
781 | 781 | | |
782 | 782 | | |
783 | 783 | | |
| |||
0 commit comments