Repository navigation
Club: majestic's own crashes, symbolized against its CI's debuginfo - #415
Merged
Merged
Conversation
majestic now leaves /etc/crash/majestic.dump when it dies of a fatal signal: the registers at the fault, a slice of the stack, the memory map and each module's build-id. A bundle carrying one is filed as class 'user', kind 'signal' (migration 032), maintainers' only: never on /crashes, and its sender is told the signal, not where. It arrives under a provisional signature (module and offset), with its dump in crash_dumps. The firmware role's new symbolize package, woken by NOTIFY crash_symbolize, writes an ELF core from the dump and unwinds it with gdb-multiarch against the executable and debuginfo majestic's CI publishes by build-id. Library frames are named from the firmware build's own rootfs, unsquashed from its release tarball. Where gdb stops (most camera libraries have no unwind tables) a scan of the stack for return addresses that follow a call finds the probable callers. The event is then refiled under its backtrace's signature and the dump, being majestic's memory, is deleted. Tested against dumps of a stand-in program that faulted on a gk7205v300 and on x86_64: in its own code, in a stripped library and in libc. The service image and CI gain gdb-multiarch and squashfs-tools.
PR Summary by QodoIngest and symbolize majestic crashes for maintainer triage
AI Description
Diagram
High-Level Assessment
Files changed (45)
|
Code Review by Qodo
1.
|
![qodo-free-for-open-source-projects[bot]](https://avatars.githubusercontent.com/in/1068000?s=60&v=4){kind=small}
- The public list and a signature's public page never count majestic's events, whatever they are merged into; a kernel signature and majestic's cannot be merged. - A triage decided under a provisional signature goes with the crash to the bug it is refiled under when that one is undecided; a crash found bogus while it waited is not refiled. - A backtrace with no frame of majestic's own leaves the crash under its provisional signature: strlen alone would make every caller one bug. - Events leaving one provisional signature lock it, so the last one out deletes it. - What gdb, unsquashfs or a library's symbols say is stripped of NULs and invalid UTF-8 before PostgreSQL sees it; a dump whose result cannot be stored counts as a failed try, and the drain stops until the next tick. - A scan start off the stack returns nothing (a crafted SP panicked it), and a panic in the symbolizer is that dump's failure, not the role's. - A firmware build that cannot be had right now is waited for; the last try makes the backtrace without its libraries. - The rootfs image is rootfs.squashfs[.<soc>], never its .md5sum.
Collaborator
Author
|
Addressed Qodo's review in a4365d5, with a test for each:
|
The redactor took OpenIPC's own version, 2.6.10.05, for a dotted quad and replaced it in meta.json with a keyed hash, so a maintainer could not read which firmware a camera ran. An octet is never written with a leading zero; now the pattern refuses one. The triage page also says it holds majestic's crashes beside the kernel's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Kernel crashes reach the Club already (#411, #413). This adds the streamer's own crashes. majestic now writes
/etc/crash/majestic.dumpwhen it dies of SIGSEGV, SIGBUS, SIGILL, SIGFPE or SIGABRT. The dump holds the registers at the fault, a slice of the faulting thread's stack, the memory map, and each loaded module's build-id. A release build has no unwind tables, so the camera can't produce its own backtrace; the site produces it.Ingest (
internal/crashes)majestic.dump, either in a tar.gz withmeta.jsonor as the bare body, is filed as classuser, kindsignal(migration 032)./crashesor/api/v1/crashes/{id}. The sender and the member's own list show the signal (SIGSEGV (NULL pointer)), never where in majestic it happened.crash_dumps.crash_bundleskeeps the dump with its stack removed.si_code <= 0, and the newsender=in the header is not majestic's own pid) counts as caused and pays nothing. majestic aborting itself counts as a bug.Symbolizer (
internal/crashes/symbolize, firmware role)NOTIFY crash_symbolize, plus a 10-minute sweep, one dump at a time.NT_PRSTATUSin each architecture'self_gregset_torder (ARM, MIPS, x86_64, AArch64),NT_AUXVto place the PIE, the stack, and the modules' code.gdb-multiarchloads the executable and debuginfo majestic's CI publishes by build-id (MAJESTIC_SYMBOLS_BASE, the S3buildid/tree). A small Python script prints frames as JSON. Host init files, debuginfod and auto-load are all off.meta.json. The rootfs is taken from that build's release tarball via the existing release cache, withunsquashfsextractinglibandusr/libonly. Symlinks are followed inside the image, never into the host's/lib.RELEASE_CACHE_ROOT/symbols, swept after 30 days unused. There is no new volume.UI
/club/crashesshows the backtrace withfile:line, probable frames marked, and the symbolization status.Tests
testdata/dump/toy.c, withgen.sh) that writes the same dump. It faulted on the lab gk7205v300 (ARM, musl, Thumb-2,-Os, no unwind tables in the executable) and on x86_64, in four ways: its own code, a stripped library, libc, andraise(). The fixtures were generated in an empty environment and checked to hold no host or lab details.store←parse_level←mainat their lines. The stripped library's frame is named from its dynamic symbols, with the caller below it. The libc frame is named by its module.unsquashfs.gdb-multiarchandsquashfs-tools. These tests fail rather than skip whenCIis set, as the ffmpeg ones do.Checked locally: Go vet and tests (all packages), frontend lint, typecheck and 710 tests, and the locale export.
Depends on majestic's recorder landing in a nightly. The WebUI sender and the firmware's respawn come after this.