Skip to content

Commit 1db9da2

Browse files
pastaghostpastaghost
committed
ci(rc18): run the suite against the release target, non-blocking
The two integration jobs pin firmware 7.16.0, because parts of this suite need firmware that postdates RC18 -- variant_getName() returning "EmulatorBTC" and the Ironwood known-answer vectors. A green run there says nothing about 7.15.0, which is the release this PR is cut against. Add integration-rc18, pinned to fd1012c5 (v7.15.0-rc18 final audit candidate, firmware PR #320), running the same suite. It asserts the emulator really reports 7.15.0 first, so a wrong ref cannot silently turn this into a duplicate of the 7.16 job and retire the gap it measures. continue-on-error for now: the 7.15-vs-7.16 delta has never been measured at current head, so a first red run is information rather than a verdict. The step summary says how to promote it to blocking once green. Tests needing post-RC18 firmware skip themselves here -- the bitcoin-only module via requires_bitcoinOnly(), the Ironwood tests via their explicit 7.16 gate.
1 parent 65f69f7 commit 1db9da2

1 file changed

Lines changed: 184 additions & 3 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 184 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@
77
# └─ lint Python syntax + deterministic protocol contract tests
88
#
99
# Stage 2: TEST (gated by Stage 1)
10-
# ├─ integration full pytest suite against the regular emulator
11-
# └─ integration-btc bitcoin-only product boundary against a
12-
# -DKK_BITCOIN_ONLY=ON emulator
10+
# ├─ integration full pytest suite against the regular emulator (7.16)
11+
# ├─ integration-btc bitcoin-only product boundary against a
12+
# │ -DKK_BITCOIN_ONLY=ON emulator (7.16)
13+
# └─ integration-rc18 the same suite against RC18/7.15.0, NON-BLOCKING --
14+
# reports the release-target result without gating
1315

1416
name: CI
1517

@@ -312,6 +314,185 @@ jobs:
312314
STATUS=$(cat keepkey-firmware/deps/python-keepkey/tests/status 2>/dev/null || echo "1")
313315
[ "$STATUS" = "0" ] || exit 1
314316
317+
# ═══════════════════════════════════════════════════════════
318+
# STAGE 2c: TEST — the RELEASE TARGET (RC18 / 7.15.0)
319+
# ═══════════════════════════════════════════════════════════
320+
321+
integration-rc18:
322+
needs: [lint]
323+
runs-on: ubuntu-latest
324+
timeout-minutes: 15
325+
326+
# NON-BLOCKING BY DESIGN, and that is a statement about evidence, not a
327+
# way to hide failures.
328+
#
329+
# The other two integration jobs pin firmware 7.16.0, because parts of
330+
# this suite need firmware that postdates RC18: variant_getName()
331+
# returning "EmulatorBTC", and the Ironwood known-answer vectors. A green
332+
# run there says nothing about the release this PR targets.
333+
#
334+
# This job closes that gap by actually running the suite against
335+
# RC18/7.15.0. It does not gate the merge yet: the 7.15-vs-7.16 delta has
336+
# never been measured at current head, so a first red run is information,
337+
# not a verdict. Promote to blocking (delete continue-on-error) once it is
338+
# green, and treat a regression from green as a real failure.
339+
#
340+
# Tests that need post-RC18 firmware skip themselves here rather than
341+
# failing: the bitcoin-only module gates on requires_bitcoinOnly(), which
342+
# keys on a variant name RC18 does not report, and the Ironwood tests gate
343+
# on 7.16.0 explicitly.
344+
continue-on-error: true
345+
346+
steps:
347+
- uses: actions/checkout@v4
348+
with:
349+
submodules: recursive
350+
path: python-keepkey
351+
352+
- name: Checkout firmware (RC18)
353+
uses: actions/checkout@v4
354+
with:
355+
repository: BitHighlander/keepkey-firmware
356+
# v7.15.0-rc18 final audit candidate -- the head of
357+
# audit/7.15.0-rc18-final, merged as firmware PR #320. This is the
358+
# release this PR is cut against.
359+
ref: fd1012c5adbacf88e2ca521a95b335a2849d528f
360+
path: keepkey-firmware
361+
362+
- name: Init the submodules the emulator build needs
363+
working-directory: keepkey-firmware
364+
run: |
365+
git submodule update --init --depth 1 deps/crypto/trezor-firmware
366+
git submodule update --init --depth 1 deps/device-protocol
367+
git submodule update --init --depth 1 deps/googletest
368+
git submodule update --init --depth 1 deps/qrenc/QR-Code-generator
369+
git submodule update --init --depth 1 deps/sca-hardening/SecAESSTM32
370+
371+
- name: Overlay this python-keepkey onto the firmware tree
372+
run: |
373+
rm -rf keepkey-firmware/deps/python-keepkey
374+
cp -a python-keepkey keepkey-firmware/deps/python-keepkey
375+
376+
- name: Build the RC18 emulator
377+
timeout-minutes: 20
378+
working-directory: keepkey-firmware
379+
run: |
380+
docker build -t kkemu-rc18-ci -f scripts/emulator/Dockerfile .
381+
382+
- name: Start the emulator
383+
run: |
384+
docker run -d --name kkemu-rc18 \
385+
-p 11044:11044/udp -p 11045:11045/udp -p 5000:5000 kkemu-rc18-ci
386+
sleep 3
387+
docker logs kkemu-rc18 | head -5
388+
389+
- uses: actions/setup-python@v5
390+
with:
391+
python-version: '3.11'
392+
393+
- name: Install dependencies
394+
working-directory: python-keepkey
395+
run: |
396+
pip install --upgrade pip
397+
pip install "protobuf>=3.20,<4"
398+
pip install -e .
399+
pip install pytest semver rlp requests eth-keys pycryptodome
400+
401+
- name: Wait for emulator
402+
run: |
403+
echo "Waiting for emulator bridge on port 5000..."
404+
for i in $(seq 1 30); do
405+
if curl -sf -X POST http://localhost:5000/exchange/main \
406+
-H 'Content-Type: application/json' \
407+
-d '{"data":""}' > /dev/null 2>&1; then
408+
echo "Emulator ready after ${i}s"
409+
break
410+
fi
411+
sleep 1
412+
done
413+
414+
# Assert this really is 7.15.0. A silently-wrong ref would make this job
415+
# a duplicate of `integration` and quietly retire the gap it exists to
416+
# measure.
417+
- name: Assert the emulator is RC18
418+
timeout-minutes: 2
419+
env:
420+
KK_TRANSPORT_MAIN: "127.0.0.1:11044"
421+
KK_TRANSPORT_DEBUG: "127.0.0.1:11045"
422+
KK_UDP_TIMEOUT: "20"
423+
working-directory: keepkey-firmware/deps/python-keepkey/tests
424+
run: |
425+
python - <<'PY'
426+
import sys
427+
sys.path.insert(0, '..')
428+
import config
429+
from keepkeylib.client import KeepKeyDebuglinkClient
430+
c = KeepKeyDebuglinkClient(config.TRANSPORT(*config.TRANSPORT_ARGS,
431+
**config.TRANSPORT_KWARGS))
432+
c.set_debuglink(config.DEBUG_TRANSPORT(*config.DEBUG_TRANSPORT_ARGS,
433+
**config.DEBUG_TRANSPORT_KWARGS))
434+
c.init_device()
435+
f = c.features
436+
got = (f.major_version, f.minor_version, f.patch_version)
437+
print('emulator firmware %d.%d.%d, variant %r' % (got + (f.firmware_variant,)))
438+
if got != (7, 15, 0):
439+
sys.exit('FATAL: expected RC18 (7.15.0), got %d.%d.%d -- the pinned '
440+
'firmware ref is not the release target.' % got)
441+
PY
442+
443+
- name: Run the suite against RC18
444+
timeout-minutes: 10
445+
env:
446+
KK_TRANSPORT_MAIN: "127.0.0.1:11044"
447+
KK_TRANSPORT_DEBUG: "127.0.0.1:11045"
448+
PYTHONPATH: "${{ github.workspace }}/keepkey-firmware/deps/python-keepkey"
449+
KK_UDP_TIMEOUT: "45"
450+
run: |
451+
cd keepkey-firmware/deps/python-keepkey/tests
452+
pytest -v --junitxml=junit-rc18.xml 2>&1 | tee pytest-rc18-output.txt
453+
echo "${PIPESTATUS[0]}" > status-rc18
454+
455+
- name: RC18 summary
456+
if: always()
457+
run: |
458+
XML="keepkey-firmware/deps/python-keepkey/tests/junit-rc18.xml"
459+
echo "## 🔑 python-keepkey — RC18 / 7.15.0 (non-blocking)" >> "$GITHUB_STEP_SUMMARY"
460+
echo "" >> "$GITHUB_STEP_SUMMARY"
461+
echo "Release-target result. Does not gate the merge; see the job comment." >> "$GITHUB_STEP_SUMMARY"
462+
echo "" >> "$GITHUB_STEP_SUMMARY"
463+
if [ ! -f "$XML" ]; then
464+
echo "❌ **No test results** — the suite crashed before completion." >> "$GITHUB_STEP_SUMMARY"
465+
else
466+
TOTAL=$(grep -oP 'tests="\K[0-9]+' "$XML" | head -1)
467+
FAILED=$(grep -oP 'failures="\K[0-9]+' "$XML" | head -1)
468+
ERRORS=$(grep -oP 'errors="\K[0-9]+' "$XML" | head -1)
469+
SKIPPED=$(grep -oP 'skipped="\K[0-9]+' "$XML" | head -1)
470+
TOTAL=${TOTAL:-0}; FAILED=${FAILED:-0}; ERRORS=${ERRORS:-0}; SKIPPED=${SKIPPED:-0}
471+
PASSED=$((TOTAL - FAILED - ERRORS - SKIPPED))
472+
echo "| Metric | Count |" >> "$GITHUB_STEP_SUMMARY"
473+
echo "|--------|-------|" >> "$GITHUB_STEP_SUMMARY"
474+
echo "| Total | $TOTAL |" >> "$GITHUB_STEP_SUMMARY"
475+
echo "| ✅ Passed | $PASSED |" >> "$GITHUB_STEP_SUMMARY"
476+
echo "| ⏭️ Skipped | $SKIPPED |" >> "$GITHUB_STEP_SUMMARY"
477+
echo "| ❌ Failed | $FAILED |" >> "$GITHUB_STEP_SUMMARY"
478+
echo "| 💥 Errors | $ERRORS |" >> "$GITHUB_STEP_SUMMARY"
479+
echo "" >> "$GITHUB_STEP_SUMMARY"
480+
if [ "$FAILED" -eq 0 ] && [ "$ERRORS" -eq 0 ]; then
481+
echo "RC18 is green at current head — this job can be promoted to blocking by deleting \`continue-on-error\`." >> "$GITHUB_STEP_SUMMARY"
482+
else
483+
echo "RC18 differs from 7.16 at current head. Each failure is either a real RC18 regression or a test that needs a version gate." >> "$GITHUB_STEP_SUMMARY"
484+
fi
485+
fi
486+
487+
- name: Annotate RC18 results
488+
uses: mikepenz/action-junit-report@v4
489+
if: always()
490+
with:
491+
report_paths: keepkey-firmware/deps/python-keepkey/tests/junit-rc18.xml
492+
annotate_only: true
493+
require_tests: true
494+
fail_on_failure: false
495+
315496
# ═══════════════════════════════════════════════════════════
316497
# STAGE 2b: TEST — the OTHER shipping product
317498
# ═══════════════════════════════════════════════════════════

0 commit comments

Comments
 (0)