Skip to content

Add native local backup creation - #47

Merged
jonathanstokes merged 10 commits into
stokes-audio:mainfrom
tony-xmelon:codex/native-local-backup
Sep 14, 2026
Merged

jonathanstokes merged 10 commits into
stokes-audio:mainfrom
tony-xmelon:codex/native-local-backup

Conversation

@tony-xmelon

@tony-xmelon tony-xmelon commented Sep 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • add create_local_backup() for the native LocalBackup{CREATE} chunk stream
  • make Transport.collect(until=) inspect every newly arrived message, eliminating the final-marker polling race
  • distinguish timeout, explicit device refusal, duplicate finals, trailing messages, and malformed payloads with typed errors
  • validate wrapper name, exact 64-hex integrity-id shape, strict Base64, nonempty decoded payload, and a documented chosen 32 MiB safety ceiling
  • mark only QuadCortex41.create_local_backup VERIFIED from contributed evidence
  • document creation as partial local-backup coverage because restore is not implemented

Testing

Final head: fc12095

  • python -m pytest tests/test_transport.py tests/test_client.py tests/test_profiles.py tests/test_docs.py -q: 648 passed, 1 skipped
  • python -m pytest tests/test_session.py tests/test_profiles.py -q: 55 passed
  • python -m mypy: passed
  • full Windows run after the runtime changes: 2712 passed, 3 skipped; it exposed one stale QuadCortex41 expectation, now fixed. The other three failures are the known Windows path/line-ending fixtures fixed in Expose protocol operations and make the test suite portable #42
  • git diff --check: passed

Coverage includes a real Transport / FakeHid batch where a final marker is followed inside the same poll interval, duplicate and trailing-final diagnostics, valid-hex wrong-length rejection, strict Base64 rejection, empty decoded payload, wrapper name validation, and profile guard behavior.

Hardware

Exact final head fc12095, 2026-09-08, standard Quad Cortex on CorOS 4.1.0:

tests: 1 passed, 0 failed, 0 errored, 0 skipped; operations measured: 1 of 1 claimed
passed:                                 ['create_local_backup'] (1)
failed or skipped:                      [] (0)
passed, not VERIFIED:                   [] (0)
VERIFIED and claimed by a test, failed: [] (0)
1 passed in 42.01s

The portable document, nonempty decoded payload, name, creator/type, and 64-hex integrity identifier all validated. Replies carried 12 ordered chunks and one final marker.

A maintainer with the baseline CorOS 4.0.1 profile should run pytest tests/hardware --hardware before merge; the contributed 4.1 record is labelled separately.

Checklist

  • Returned to draft while addressing review
  • Final-head hardware run recorded
  • Offline race/error/profile tests and mypy pass
  • Creation vs restore gap is explicit
  • No unrelated changes bundled in

Upstream sync — 2026-09-12

Merged current upstream main at 640a8e9 (including merged PRs #63–#65) into this branch. Current head: 5cc58ff

Offline suite excluding the three #42-owned Windows portability checks: 2611 passed, 3 skipped; full run: 2738 passed, 3 skipped, 3 known portability failures; race-fix focus: 379 passed; mypy: clean across 46 source files; git diff --check: clean.

The three excluded Windows checks are unrelated path-separator and CRLF-fixture assertions fixed within #42; they remain isolated there rather than being copied into this PR.

@tony-xmelon

Copy link
Copy Markdown
Contributor Author

Updated onto #52 in a317e2a. I also integrated the new hardware-operation inventory by adding a read-only create_local_backup hardware test that validates the complete portable wrapper and decoded payload without saving or changing device state. Focused validation: 369 passed; hardware suite collection 100/100; mypy clean; git diff --check clean.

@tony-xmelon
tony-xmelon marked this pull request as draft September 8, 2026 08:41
@tony-xmelon

Copy link
Copy Markdown
Contributor Author

Synced with upstream �2331cc at head 736c1d3 and moved back to draft pending the required full hardware suite on this exact commit. Focused validation: 410 passed; 100 hardware tests collected; mypy clean.

@tony-xmelon
tony-xmelon marked this pull request as ready for review September 8, 2026 09:36

@jonathanstokes jonathanstokes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, and I want this in. Local backup is a real gap in what this library covers.

You did the hard part. You found that the unit answers one LocalBackup CREATE with a stream of chunks that carry no request id. You handled it with the public transport API, and you did not reach into the collector. Adding until= to collect() is the right shape for that. It also keeps the old behaviour when nobody passes it.

Your hardware section has all four things the policy asks for, including the operations on ... block. You also said plainly that you have no 4.0.1 unit. That is what I need to see.

The profile design was not written down when you wrote this. You could not have followed it. That is on me. Since then I merged ADR-0020, and contributing.md grew three sections: "Working with hardware", "Before you mark a pull request ready", and "Adding a device profile".

Here is the short version. One baseline became a registry of device profiles. QuadCortex is the Quad Cortex on CorOS 4.0.1, and it is the base class. QuadCortex41 is a subclass that connects but has verified nothing yet. Each profile lists what it has measured in VERIFIED. Anything else it inherits refuses.

Two things follow from that for this PR. Neither one means your code is wrong. Both are notes on the lines below, along with the smaller points.

I will run pytest tests/hardware --hardware on my own 4.0.1 unit before this merges. I will write what I see beside your record, with the date and the CorOS version. Then both measurements stand next to each other. If the chunk sizes or the refusal differ on 4.0.1, that is a fact about the device. It goes in the document, not against your PR.

One last thing. If these changes are more than you signed up for, tell me and I will make them myself. The transport change and the client method can land separately. Or just drop the PR if you would rather not have it. Either is fine, and the protocol work here is useful to me either way.

Stokes

Comment thread pyquadcortex/protocol/transport.py Resolved
Comment thread pyquadcortex/protocol/client.py Resolved
Comment thread docs/protocol.md Resolved
Comment thread docs/protocol.md Resolved
Comment thread pyquadcortex/protocol/client.py Resolved
Comment thread pyquadcortex/protocol/client.py Resolved
Comment thread tests/test_client.py Resolved
Comment thread tests/hardware/test_local_backup.py Resolved
Comment thread docs/manual-coverage.md Resolved
Comment thread docs/manual-coverage.md Resolved
@tony-xmelon
tony-xmelon marked this pull request as draft September 8, 2026 18:45
jonathanstokes
jonathanstokes previously approved these changes Sep 9, 2026

@jonathanstokes jonathanstokes left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, this is a thorough response. Every one of the twelve notes is addressed, and the fixes are real rather than cosmetic. I checked that by breaking each of the three gaps I had reported and running the suite: the loosened hash pattern, validate=False, and putting until back to the newest message only now each fail a test. They all passed before. test_collect_checks_every_message_in_a_single_poll_batch is the test I was asking for: it drives the real Transport instead of the stand-in, so it would notice if collect went back to checking only the last message.

Approving. There is one small thing left, on the line, and it is not a blocker.

My hardware run

Quad Cortex, CorOS 4.0.1, app_fw d14e, on your final commit fc12095:

============== operations on QuadCortex (CorOS 4.0.1, MAINTAINER) ==============
tests: 96 passed, 0 failed, 0 errored, 4 skipped; operations measured: 17 of 17 claimed
passed, not VERIFIED:                   [] (0)   <- candidates to add
VERIFIED and claimed by a test, failed: [] (0)   <- regressions by name
================== 96 passed, 4 skipped in 191.41s (0:03:11) ===================

create_local_backup passes on the baseline. So the method belongs on QuadCortex, which is where you put it. Offline 2716 passed, 3 skipped. mypy clean. All five CI jobs pass.

What 4.0.1 does on the wire

Four runs, with a listener attached that just writes down every message as it arrives. This is the "Baseline 4.0.1 pending maintainer verification" line you left in the table.

4.0.1 (measured 2026-09-09) 4.1.0 (your report)
Messages 1 12
Chunk characters 130,178 11 x 150,000 plus 144,890
Final markers 1, and it is last 1
request_id on replies none none
can_apply_backup never appeared false seen once
Wrapper fields 10 10
payload_hash 64 hex 64 hex
Duration 9.1s cold, then about 2.9s not stated

The interesting part: 150,000 looks like a chunk ceiling, not a fixed chunk size. My backup is 130,178 characters and arrives whole in one message. Yours was 1,794,890 and split into 150,000-character pieces with the remainder last. So the docstring's "the unit emits 150,000-character chunks" reads as a device constant when it is really a cap, and a backup under the cap arrives in a single message. Your code handles both, because it joins whatever arrives.

Two smaller facts. Three back-to-back calls on one connection all worked, so "only one backup in flight" is about concurrency and not about calling it twice. And can_apply_backup never appeared in four runs, so the refusal path is still unverified on 4.0.1 - your softened docstring says exactly that, which is right.

I will add the 4.0.1 half of the row after this merges. The header you reverted is being fixed separately in PR #56, so the table can carry a per-profile row without weakening what it promises.

One correction, mine

My note on transport.py said this would "fail once in a while on a real unit." I had not measured that, and I should not have said it. Four runs on 4.0.1 produced no trailing message, and your 4.1.0 run saw exactly one final marker. The code path was real and worth fixing. The field failure I predicted was not something I had seen.

Comment thread pyquadcortex/protocol/transport.py Resolved
@tony-xmelon

Copy link
Copy Markdown
Contributor Author

Synced with upstream main at 640a8e9 (including #63–#65). New head: de4ce12. Offline suite excluding the three #42-owned Windows portability checks: 2611 passed, 3 skipped; full run: 2738 passed, 3 skipped, 3 known portability failures; mypy: clean across 46 source files; git diff --check: clean. The three shared Windows-only baseline checks remain isolated in #42.

jonathanstokes
jonathanstokes previously approved these changes Sep 14, 2026

@jonathanstokes jonathanstokes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving again. 5cc58ff is the change I asked for, and I checked it rather than taking it on trust.

I re-ran the same construction I used before, where a message lands in the list right after the snapshot is taken. With the old line the loop never showed that message to the stopping test, so it sat there until the timeout. With your line it gets picked up on the next pass:

old  checked = len(got)      -> stopping test saw the final message: no
new  checked += len(newest)  -> stopping test saw the final message: yes

My first attempt at that check was wrong, and it is worth saying why in case you try it. I made the message arrive at the moment the code reads the list's length. Your version never reads it, so nothing arrived and both versions looked broken. The message has to land just after the snapshot instead. Once I moved it there the result was clear.

Offline suite 2741 passed, 3 skipped. mypy clean on 46 source files. All five CI jobs pass.

I have not run the unit on this commit

Another session was using the Quad Cortex while I was working, so I could not get to it. I am approving on the offline evidence alone, and the hardware run still has to happen before this merges. That is my job, not yours, and it will not change anything I have said here: this commit changes when the waiting stops, not what the unit is asked for or what it sends back. My earlier run on fc12095 already measured the real thing.

Two small things before it goes in

The branch is behind main now. No conflicts, it just needs main merged in.

The table row still ends "Baseline 4.0.1 pending maintainer verification". I measured 4.0.1 on 2026-09-09 and put the numbers in my last review. The short version is that my unit sends the whole document in one message of 130,178 characters, where yours sends twelve. That reads as 150,000 being a maximum size per message rather than a fixed one, and your code handles both because it joins whatever arrives. Worth folding into the row now, so the file is not committing a claim we then correct a day later. Say the word if you would rather I did that part.

Thanks for sticking with this one. It is in much better shape than where it started.

Stokes

# Advance only by the snapshot we just inspected. The RX
# thread may append between the slice and this update; a
# fresh len(got) here would skip that arrival forever.
checked += len(newest)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This is right, and nothing holds it in place.

I put the old line back and ran the whole suite. All 2741 tests still passed. So if someone later tidies this up and reaches for len(got) again, nothing tells them they broke it.

I am not asking you to fix that here. It is genuinely hard to test: the gap you are protecting against is between two lines that run one after the other, so catching it needs a way to make a message arrive in the middle, and there is no hook for that today. I had to use a fake list that appends to itself when it is read.

Your comment above is a fair substitute, and it says the right thing. I wanted it on the record that the comment is what is guarding this, rather than a test.

Measured on my own unit on 2026-09-09, which is what the row was waiting for.
A 130,178-character backup arrives as ONE push with the final marker set, where
the contributed 4.1.0 measurement saw twelve. Read together, 150,000 is the most
one push carries and not the size every push is, so a document under that
arrives whole. The docstring said it as a flat fact about the device and now
says which unit each number came from.

Also records what I could not see: can_apply_backup never appeared in four runs
on 4.0.1, so treating it as a refusal to CREATE is still inference from the
schema. It sits beside applied_backup, so it may belong to restore.

@jonathanstokes jonathanstokes left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed three things to your branch to finish this off, since you cannot merge here and I can. Shout if you would rather I had left any of it to you.

Merged main in. The branch was 13 commits behind and this repo requires a branch to be up to date before it merges. No conflicts.

Put my own 4.0.1 numbers in the table row. It was still saying "Baseline 4.0.1 pending maintainer verification", and I measured it on 2026-09-09.

Rewrote the docstring's account of chunking, because our two measurements together say something neither says alone. Your unit sent 1,794,890 characters as twelve pushes. Mine sent 130,178 as ONE push. So 150,000 is the most one push carries, not the size every push is, and a backup under that size arrives whole. The docstring stated it as a flat fact about the device and now says which unit each number came from. Your code already handled both, because it joins whatever arrives and stops at the marker.

I also wrote down what I could not see. can_apply_backup never appeared in four runs on 4.0.1, so reading it as a refusal to CREATE is still inference from the schema rather than something measured. It sits next to applied_backup, so it may belong to restore. That is recorded as unknown rather than quietly dropped.

Hardware, on the final commit

9abfb02, Quad Cortex, CorOS 4.0.1, app_fw d14e:

============== operations on QuadCortex (CorOS 4.0.1, MAINTAINER) ==============
tests: 110 passed, 0 failed, 0 errored, 3 skipped; operations measured: 17 of 17 claimed
passed, not VERIFIED:                   [] (0)   <- candidates to add
VERIFIED and claimed by a test, failed: [] (0)   <- regressions by name
================== 110 passed, 3 skipped in 181.05s (0:03:01) ==================

Offline 2779 passed, 4 skipped. mypy clean on 46 source files. All five CI jobs pass.

That clears the last thing standing in the way. Approving, and I will merge it.

Thanks for the patience on this one. It went through a lot of rounds, and the result is better than what either of us would have written alone.

Stokes

@jonathanstokes
jonathanstokes merged commit d4663e2 into stokes-audio:main Sep 14, 2026
5 checks passed
@tony-xmelon
tony-xmelon deleted the codex/native-local-backup branch September 22, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants