Repository navigation
Expose ModelRepo editor metadata - #46
tony-xmelon wants to merge 15 commits into
Conversation
|
Synced with upstream �2331cc at head 4a05091 and moved back to draft pending the required full hardware suite on this exact commit. Focused catalog/model/hardware-report validation: 567 passed; mypy clean. |
jonathanstokes
left a comment
There was a problem hiding this comment.
Thanks for this. The parser work is clean and I want the change in.
I checked three of your claims rather than trust them. All three hold. The rename of _parse_replaces to _parse_indexes leaves the function body identical, and nothing calls the old name, so replaces parses as before. The six new fields go after the last existing field, so no position moves. The offline suite and mypy are both green on your current head.
I also checked your reading of toggleOn and toggleStep against the catalog. You are right, and the repo currently says something different. I left the details at the line.
Now the part that is on me. After you opened this, I merged a design that changes where a change like yours belongs. None of it was written down when you started, so you could not have followed it. ADR-0020 is the record. One baseline became a registry of device profiles. A Version reply now picks a profile at connect. Anything that differs by firmware or model lives on that profile. contributing.md has the two sections that follow from it: "Before you mark a pull request ready" and "Adding a device profile". Please read both before the next round.
The good news is that the profile system asks almost nothing of this PR. Parameter and parse_model_repo are not methods on QuadCortex, so this adds no operation. That means no VERIFIED entry and no hardware marker. The parser reads whatever the connected unit publishes, so it works for any profile. Leave it where it is. The one exception is linkedSceneMode, and I left a comment on it at the line.
Two points that do not belong at any single line.
critical: This repo keeps a list of catalog attributes we can see but cannot explain. It is the appendix of docs/domain-model.md. Five of your six are on that list today. A field leaves the list when someone writes down the evidence. That list is what stops us reading an attribute as whatever its name suggests. So please edit the appendix in this PR. Its header says the device puts 24 attributes on a <Parameter> and fourteen are parsed. After this change nineteen are parsed, so that sentence needs fixing, and the five rows need to move out with your evidence. Two other files carry the same question and will now contradict your docstrings: docs/protocol.md:3394 and docs/ADR.md:251.
critical: Every number in the description is from CorOS 4.1.0. None of them reproduces on 4.0.1. I get 36 midpoint labels, not 49. I get 1,446 declared positions, not 1,965. I get 132 / 83 / 13 toggles, not 168 / 124 / 16. I get zero linkedSceneMode, not 44. Measuring on 4.1.0 is fine. The problem is that the description does not say which firmware. Under ADR-0020 a 4.1.0 observation goes beside the 4.0.1 record and names itself. Please label them. Where a docstring carries a count, use the 4.0.1 number, so a reader can check it.
nit: A pull request now opens as a draft and carries a Hardware section before it is marked ready. That rule landed after you opened this, so it is not a complaint. This change cannot reach the wire, so I am happy to waive the suite run and will write that in the section myself.
On my side, I will run pytest tests/hardware --hardware on my own 4.0.1 unit before merging. I do that for every PR. If what I see differs from your 4.1.0 audit, I will record my numbers beside yours, not in place of them.
Last thing. This is more work than you signed up for, and most of it is documentation. If it is more than you want, say so. I will split the doc half out and do it myself, and keep your commit for the parser. Or just drop the PR if you would rather not have it. Either is fine, and the parser work stands either way.
Stokes
…tadata # Conflicts: # changelog.md # docs/domain-model.md # pyquadcortex/protocol/catalog.py # tests/test_catalog.py
|
Synced with upstream main at 640a8e9 (including #63–#65). New head: ef95034. Offline suite excluding the three #42-owned Windows portability checks: |
…tadata # Conflicts: # changelog.md # docs/ADR.md # docs/domain-model.md # docs/protocol.md # pyquadcortex/protocol/catalog.py
…tadata # Conflicts: # changelog.md
jonathanstokes
left a comment
There was a problem hiding this comment.
The payload you asked for now exists, and it closes this thread.
PR #80 proposes tests/fixtures/catalog/model_repo_coros_4_0_1.bin (46,695 bytes,
sha256 fdc3086...77ac14) captured from a unit reporting zenos_git_hash 4.0.1,
app_fw_version d14e, device_type QC. It is not merged yet, so take it from the
branch catalog/model-repo-payload-4-0-1. All three generators in scripts/ accept
--payload.
I ran your own parser at 579a244 against that payload. It reproduces every number
this thread asked to be pinned, offline, with no unit attached:
mid_label carriers 36 (35 "C", 1 "A/B" = Micro Processor (ST) "A/B PITCH MIX")
all 36 carry min_string and max_string; all parse to skew 1.0
display_pos carriers 1446
displayPos collisions 5 models
toggle_on / off / step 132 / 83 / 13
toggleStep carriers lacking both toggleOn and toggleOff: 0
self-references 3 (Mono Synth OSC2 WAVE@15, SLOPE@24, FREE RATE@48)
Analog Delay (ST)[20] "FEEDBACK DEPTH" display_pos = None <- the isplayPos casualty
So the evidence gap is closable without a 4.0.1 unit. A ~35 line extractor patterned
after scripts/extract_scale_fixture.py produced a 12.5 KB real-row fixture here in
one run.
Three things to settle before this goes ready.
1. The pinning test does not test the device.
test_real_catalog_editor_metadata_evidence_is_pinned loads editor_metadata.json
and asserts the literals typed into the same pull request. Change a number in both
places and CI stays green. editor_metadata.json is read by that one test and
nothing else; scales.json and option_readings.json are read by generators, by
the documents, and by hardware drift tests. Regenerate the fixture from the payload
with a committed extractor, and the assertions become assertions about the unit.
2. gap_models: 18 does not reproduce.
Against the real 4.0.1 payload I get 21 gap models counting a gap as any missing
position in 0..max, or 17 counting from 1. That holds on the all-parameter basis
and on the visible-only basis. Excluding the 5 collision models gives 20 and 16.
None of the eight combinations gives 18. collision_models: 5 is right, but it is on
the all-parameter basis while catalog.py quotes 1 for the same thing on the visible
basis, and the file does not say which basis it uses. An extractor settles both.
3. The ADR and protocol.md corrections were lost in a merge.
You made them in 45195dc. The 19 Sep merge a8d026c, resolved in favour of upstream,
reverted them. At 579a244, docs/ADR.md:253 and docs/protocol.md:2185 still say
the open question is whether these attributes "describe a footswitch toggle's two
values" - the reading the Splitter evidence disproves - while docs/domain-model.md:1207
in the same commit says the visibility reading holds except for the three self-references.
Two files in one commit, contradicting each other. Please restore the 45195dc text.
While you are in docs/domain-model.md: the section header at 1194 still reads "The
unit puts 24 distinct attributes on its <Parameter> elements. Seventeen are parsed
... and the seven that are not yet explained." There are exactly 24, and this branch
parses three of the seven, so it is 20 and 4 now.
Smaller things, none blocking:
- The changelog entry names
toggleOn/toggleOff/toggleSteprather than the
toggle_on/toggle_off/toggle_stepsa caller types, and carries no firmware tag
where both neighbouring entries do. test_self_referencing_toggle_metadata_is_preserved_not_interpretedbuilds its
parameter at index 0 withtoggleOn="0", where a self-reference is indistinguishable
from ordinary parsing. The real cases are at 15, 24 and 48.toggle_offgets no#:comment, so it renders undocumented next to its two siblings.- Two blank lines are missing before
def test_a_labelled_end_control_carries_the_span_the_unit_draws. - The description still sells five fields.
mid_labelanddisplay_posreached main
during the upstream merges, so this branch now adds three.
I have not run the hardware suite. That is mine to run, not yours to waive, and it
comes after the above.
|
Thank you for supplying the payload and the complete second review. Current head e83a50 addresses every listed point. The fixture is now generated by a committed extractor from #80's real 4.0.1 payload and carries distilled raw rows, not hand-written aggregate assertions. The unreproducible gap count is corrected to 21 on the explicit all-parameter 0..max(displayPos) basis. ADR-0015 and protocol.md have the conditional-visibility reading restored; the attribute overview is corrected to 20 parsed / 4 unexplained; the synthetic self-reference uses index 15; the PR description now says this branch adds three fields; and the smaller inline items are all applied and resolved. Exact-head validation: 2972 passed, 7 skipped; focused catalog/docs 336 passed, 1 skipped; mypy clean across 46 files; build and twine check passed; diff check clean. The full 4.0.1 hardware suite remains yours to run as requested. |
Posted from the wrong account. Reposting the same review as jonathanstokes; please read that one.
jonathanstokes
left a comment
There was a problem hiding this comment.
I reproduced the fixture instead of reading it. scripts/extract_editor_metadata_fixture.py, run against tests/fixtures/catalog/model_repo_coros_4_0_1.bin on main, writes tests/fixtures/catalog/editor_metadata.json byte for byte, same sha256. Every number in the new text falls out of the payload on an independent parse: 3,809 parameters, 36 midpoint carriers at 35 C and one A/B, 1,446 display positions, 5 collision models, 21 gap models, the single isplayPos on Analog Delay (ST) index 20, and 132 / 83 / 13 toggle carriers across 212 parameters. The three Mono Synth self-references are at 15, 24 and 48. The offline suite and mypy are clean on your head and on a local merge with current main. Thank you for making it reproducible, that is exactly what I wanted.
Your reading is better evidenced than the text claims. All 216 toggleOn and toggleOff values are parameter indexes inside their own model, and none is out of range. 214 of the 216 point at a switch, comboBox or rotarySwitch. Every toggleStep value is a valid option index of the parameter that toggleOn names. Beyond the Splitter, Pitch Correction hides ROOT when MODE is Chrom, which is the inverse case toggle_off covers, and Phase Doctor carries both on one parameter to combine two conditions. You could cite either of those if you want a second model behind the claim.
Two things I need changed, then four small ones.
1. The linkedSceneMode record is gone. The description says the field "remains excluded from shared code and recorded as a firmware-labelled observation". The first half is true. The second is not: grep -r linkedSceneMode over fe83a50 returns nothing. It was in docs/domain-model.md at ef95034 and the merge at 579a244 dropped it. That is the whole record of a 4.1 observation nobody else has, and docs/writing.md routes a catalog attribute we cannot explain to that file. Please put it back. Here is the text as it stood:
linkedSceneModeappears 44 times in the contributed CorOS 4.1.0 catalog and
zero times in 4.0.1. Its meaning and relationship, if any, to preset
scene_modeare unknown, so it is recorded here and not exposed as shared API.
2. Please drop the docs/ADR.md hunk. ADR-0015 is Decided (2026-08-26), and that file's own Lifecycle section says a decided record is append-only, with one in-place edit allowed: resolving an open question as Resolved (date): .... The hunk rewrites the question's wording instead, and the original text is deleted. Your evidence already lives in docs/protocol.md and docs/domain-model.md, which is where docs/writing.md sends it. I will append the Resolved line to ADR-0015 myself. There is a suggestion below that restores the original line.
3. _parse_indexes lost its docstring somewhere in the merges. At 45195dc it said the silence is deliberate and why, which is what I asked for, and it went through _as_int. The version on your head is a one-line summary and a hand-rolled int(), sitting next to _parse_replaces, which does the same job the same way. Suggestion below.
4. The gap-model comment names the origin basis but not the model basis. It counts all 533 models, two of which are hidden (PCOM Tape Delay (ST) and Chief DC2W PCOM). The display_pos docstring above it counts over non-hidden models, which is where its 163, 165, 23 and 43 come from. On that basis the gap count is 19, not 21. Both numbers are right, they just answer different questions, and one clause stops the next reader from trying to reconcile them. Suggestion below.
5. docs/STEERING.md section 4 lists the contents of scripts/ by name, and extract_editor_metadata_fixture.py is not in it. Please add it.
6. Nothing holds editor_metadata.json to the payload it comes from. scales.json has tests/hardware/test_scales_on_unit.py behind it, and tests/test_catalog_payload.py explains the failure for the generated snapshots: a saved input nothing reads is a file that rots silently. Your fixture is derived entirely from the payload, so an offline test can regenerate and compare, the way test_the_payload_still_generates_the_committed_snapshot does. That needs the payload, which arrives when you sync with main, so it is the one item here that has to wait for the sync.
On hardware: I am waiving the run for this one. It parses three catalog attributes, changes no operation and writes nothing to the unit, and the evidence is now reproducible offline from the committed payload, so a run would exercise nothing this diff touches. When you push the fixes, please add a line to the description saying the owner waived the hardware run on 2026-09-23. contributing.md wants that recorded there before I can mark it ready.
One last thing, and please do not do it yet. This branch is behind main and the ruleset uses strict status checks, so it needs a sync right before I merge. I will ask for it then, so you only do it once.
|
@jonathanstokes I pushed 200864c addressing the requested items: restored the original append-only ADR-0015 question, restored the 4.1-only linkedSceneMode observation, documented the model basis for the 21/19 gap count (including the two hidden models), improved _parse_indexes using _as_int with the deliberate-invalid-token rationale, and added the fixture generator to the owned-paths inventory. I added the requested hardware waiver dated 2026-09-23 to the PR description. Validation: 2,972 passed, 7 skipped; mypy clean across 46 files; git diff --check clean. I have not synced main, per your request; I will add the payload-regeneration test when you request that single sync. The PR remains draft pending that test and your review. |
|
I followed the cleanup guidance in your inline note against the current upstream main: after sync at d8d40f3, main no longer carries the duplicate _parse_indexes parser, so this PR now reuses its _parse_replaces integer-list parser for the three toggle index lists too. This removes the duplication without reintroducing the helper you asked to leave to main. The other two threads are also reflected on the current head: the ADR-0015 wording matches main, and the editor metadata fixture regenerates from the captured payload. Full suite 2,979 passed / 7 skipped; mypy clean (46 files); diff check clean. Exact-head CI is pending. |
|
All review threads are now resolved against the current head, exact-head CI run 36285068348 is green, and the owner-waived hardware condition does not apply to this fixture-backed metadata change. I marked the PR ready and request re-review. |
Summary
Expose the three remaining shared ModelRepo conditional-editor fields without inventing UI semantics:
toggle_ontoggle_offtoggle_stepsmid_labelanddisplay_posare already on main. The API here is additive with neutral defaults. Documentation records that the conditional-visibility reading is inferred from catalog shape and that three Mono Synth self-references remain unexplained. The CorOS 4.1-onlylinkedSceneModefield remains excluded from shared code and recorded as a firmware-labelled observation.Evidence
The fixture is now generated from the real CorOS 4.0.1 ModelRepo payload supplied in #80 by the committed
scripts/extract_editor_metadata_fixture.py; it is no longer a hand-written aggregate echo. It pins distilled raw rows for:C, oneA/B; all carry min/max labels and parse to skew 1.00..max(displayPos)basisAnalog Delay (ST)isplayPostypotoggleOn/toggleOff/toggleStepcarrierstoggleStepcarriers also carryingtoggleOnortoggleOffThe earlier figure of 18 gap models was not reproducible and is removed.
Review corrections
docs/protocol.mdtoggle_offValidation
Final head:
fe83a5042347264f627f54e5282623b28fa8cdd02972 passed, 7 skipped336 passed, 1 skippedgit diff --check: passedHardware
The complete hardware suite is for the maintainer's CorOS 4.0.1 unit after review, as requested. The device-derived catalog evidence above is reproducible offline from the captured 4.0.1 payload and does not claim a new hardware run.
Checklist
Hardware: The owner waived the hardware run on 2026-09-23; this catalog-metadata change parses fixtures only and sends no device operation.
Offline validation on 200864c: 2,972 passed, 7 skipped; mypy clean across 46 source files; git diff --check clean. The payload-regeneration assertion is now part of the branch after the one-time upstream sync.
Upstream reconciliation — 2026-09-27
Merged upstream
mainatd8d40f37d141215213284b3ba45050ff1efac237in commit466587c. The current main already removed the duplicate_parse_indexeshelper; this PR now reuses main's_parse_replacesinteger-list parser fortoggleOn,toggleOff, andtoggleStepas well, without carrying a second parser.Current head:
3aec9f5b6687e9c98ad05aec802cd716635ecf6dValidation: full offline suite 2,979 passed, 7 skipped; mypy clean across 46 source files;
git diff --checkclean. Exact-head CI run36285068348is green: build + twine, mypy, and Python 3.11/3.12/3.13. The maintainer waived a hardware run for this metadata-only change. All review threads are addressed; re-review is requested.