Collect the kernel crashes cameras recover from, rank them, and pay Club stars for them

CLAUDE.md

@@ -208,6 +208,26 @@ restores the image but never the schema, so keep migrations additive.
   `wall_notices` for the next run. `openipc club wall-revoke` takes a faked
   camera's stars back. The owner's name on the camera's wall page is opt-in;
   the leaderboard (`/club/leaderboard`) is opt-out (migration 027).
+- `internal/crashes` -- **kernel crashes cameras recovered from**: the
+  bundle the firmware's `S98crashlog` makes of pstore's records after an
+  oops or a panic, which the WebUI offers its owner. Sent by the WebUI to
+  `POST /api/v1/crashes` (the contract is `internal/crashes/CRASH.md`), or
+  downloaded and sent by a member from `/club`. Filed under a **signature**
+  -- the top five backtrace frames, `fn [module]` without offsets -- so one
+  bug from many cameras is one row (migration 030); the records pstore keeps
+  twice (`Oops#1`, `Panic#2`) are one crash, and the same records sent again
+  by one camera are one event. Ranked by kind, how many cameras and whether today's
+  firmware still has it; public at `/crashes` (signatures only), triaged by
+  maintainers at `/club/crashes` or `openipc crashes status`. Logs are
+  redacted (MACs and IPs keyed-hashed) and maintainers' only; bundles are
+  kept as sent in PostgreSQL (`crash_bundles`), never served; events and
+  bundles are guarded like owner reports. Stars go into `crash_stars`, the
+  third ledger, paid by the nightly `settle` only: 1 per camera and bug, 3 to
+  the first reporter once confirmed, 5 more when fixed, at most 10 a month;
+  a crash asked for (sysrq) pays nothing. The ledger is net per bug, through
+  merges: `bogus` takes back what a bug and the signatures merged into it
+  paid, and undoing it lets the settlement pay again. A crash with a MAC is
+  the camera's linked owner's, whoever sent it.
 - `internal/tools`, `internal/nfsro` — **ipctool for stock firmware**, which
   has no curl and no TLS. ipctool's release job pushes each build to
   `PUT /api/v1/tools/{name}` (OIDC, `internal/tools/PUSH.md`); nginx serves

data/locales/boards.en.yml

@@ -46,6 +46,14 @@ en:
       eyebrow: 'OpenIPC Club'
       title: 'Who helps the most'
       lede: 'Stars for documenting camera boards and for keeping cameras on the Open Wall, from the members who chose to be listed.'
+    club_crashes:
+      eyebrow: 'OpenIPC Club'
+      title: 'Crash triage'
+      lede: 'Kernel crashes cameras sent, worst first, with what it takes to reproduce each one.'
+    crashes:
+      eyebrow: 'OpenIPC firmware'
+      title: 'Kernel crashes from the field'
+      lede: 'When a camera running OpenIPC crashes, it keeps the kernel''s last words and offers them to its owner. Sent here, the same bug from many cameras becomes one row, ranked by how badly and how widely it hurts, so it can be fixed.'
   nav:
     boards: Camera boards
   cameras:
@@ -605,3 +613,72 @@ en:
       lb_empty_30d: 'Nobody on the leaderboard earned stars in the last 30 days.'
       lb_note: 'Every member with stars appears here unless they opted out on their club page. Ranked by stars, never by how many cameras someone has.'
       lb_join: 'Join the OpenIPC Club'
+      side_from_crashes: 'From crash reports'
+      lb_crashes: 'Crashes'
+    crashes:
+      list_label: 'Kernel crashes, worst first'
+      empty: 'No crash has been sent yet.'
+      kind_bootloop: 'Boot loop'
+      kind_panic: 'Kernel panic'
+      kind_oops: 'Oops'
+      kind_bug: 'BUG'
+      kind_warning: 'Warning'
+      in_irq: 'in an interrupt'
+      status_open: 'Not yet looked at'
+      status_confirmed: 'Confirmed'
+      status_fixed: 'Fixed'
+      status_wontfix: 'Will not be fixed'
+      status_bogus: 'Not a real crash'
+      current: 'In current firmware'
+      cameras:
+        one: '{count} camera'
+        other: '{count} cameras'
+      events:
+        one: '{count} crash'
+        other: '{count} crashes'
+      last_seen: 'last {date}'
+      issue: 'The issue tracking it'
+      col_soc: 'SoC'
+      col_sensor: 'Sensor'
+      col_build: 'Kernel · firmware · majestic'
+      col_cameras: 'Cameras'
+      signature: 'Signature {id}'
+      firmware: 'firmware {list}'
+      how_title: 'Your camera crashed?'
+      how_1: 'After a crash, the camera''s web interface shows "This camera recovered from a crash" on every page. Open the crash report.'
+      how_2: 'Press Send to OpenIPC, or download the crash log and send it from'
+      how_club: 'your OpenIPC Club page'
+      how_3: 'Link the camera to the Club on the Open Wall, and the crashes it sends earn you stars.'
+      how_privacy: 'What is sent is the kernel''s log of the crash, the firmware version and the chip. The camera''s MAC and every IP address in it are replaced with hashes; the logs are read only by OpenIPC''s maintainers. Nothing is sent unless you send it.'
+      mine_title: 'My crash reports'
+      all_link: 'All crashes'
+      mine_lede: 'Each bug a camera of yours hits earns {report} ★ once settled; if you were the first to send it, {first} ★ more when a maintainer confirms it and {fixed} ★ when it is fixed. At most {cap} ★ a month from crashes; a crash you caused on purpose earns nothing.'
+      file_label: 'Crash log from the camera''s web interface (crashlog_*.tar.gz)'
+      mac_label: 'The camera''s MAC, if it is linked on the Open Wall (optional)'
+      send: 'Send the crash log'
+      sent: 'Received, filed under'
+      sent_again: 'This crash was already received, under'
+      sent_self: 'Received, but this crash was asked for (sysrq), so it earns nothing:'
+      self_inflicted: 'caused on purpose'
+      mine_empty: 'No crash sent yet, from you or your linked cameras.'
+      triage_link: 'Crash triage'
+      triage_forbidden: 'Only OpenIPC''s maintainers triage crashes.'
+      merged_into: 'merged into {id}'
+      taken_back: 'Saved; {n} ★ taken back.'
+      saved: 'Saved.'
+      f_status: 'Status'
+      f_fixed_in: 'Fixed in (firmware version)'
+      f_issue: 'Issue'
+      f_merge: 'Same bug as signature'
+      f_note: 'Note'
+      f_save: 'Save'
+      d_hardware: 'Hardware'
+      d_firmware: 'Firmware'
+      d_kernel: 'Kernel'
+      d_builds: 'Matching builds'
+      d_task: 'Task'
+      d_interrupted: 'interrupted'
+      d_uptime: 'Uptime'
+      d_cmdline: 'Command line'
+      d_anomalies: 'Before it'
+      d_before: 'Warnings'

data/locales/boards.ru.yml

@@ -46,6 +46,14 @@ ru:
       eyebrow: 'Клуб OpenIPC'
       title: 'Кто помогает больше всех'
       lede: 'Звёзды за описанные платы камер и за камеры на Open Wall — у участников, которые согласились попасть в список.'
+    club_crashes:
+      eyebrow: 'Клуб OpenIPC'
+      title: 'Разбор падений'
+      lede: 'Падения ядра, присланные камерами, — сначала самые тяжёлые, с тем, что нужно, чтобы воспроизвести каждое.'
+    crashes:
+      eyebrow: 'Прошивка OpenIPC'
+      title: 'Падения ядра с реальных камер'
+      lede: 'Когда камера с OpenIPC падает, она сохраняет последние слова ядра и предлагает их владельцу. Присланные сюда, одинаковые падения со многих камер становятся одной строкой, отсортированной по тому, насколько тяжело и насколько широко они бьют, — чтобы их можно было исправить.'
   nav:
     boards: Платы камер
   cameras:
@@ -646,3 +654,76 @@ ru:
       lb_empty_30d: 'За последние 30 дней никто в рейтинге не получил звёзд.'
       lb_note: 'Здесь каждый участник со звёздами, если он не отказался от этого на своей странице клуба. Порядок — по звёздам, а не по числу камер.'
       lb_join: 'Вступить в клуб OpenIPC'
+      side_from_crashes: 'За отчёты о падениях'
+      lb_crashes: 'Падения'
+    crashes:
+      list_label: 'Падения ядра, сначала самые тяжёлые'
+      empty: 'Пока никто не прислал ни одного падения.'
+      kind_bootloop: 'Циклическая перезагрузка'
+      kind_panic: 'Kernel panic'
+      kind_oops: 'Oops'
+      kind_bug: 'BUG'
+      kind_warning: 'Предупреждение'
+      in_irq: 'в прерывании'
+      status_open: 'Ещё не разобрано'
+      status_confirmed: 'Подтверждено'
+      status_fixed: 'Исправлено'
+      status_wontfix: 'Исправлено не будет'
+      status_bogus: 'Не настоящее падение'
+      current: 'В текущей прошивке'
+      cameras:
+        one: '{count} камера'
+        few: '{count} камеры'
+        many: '{count} камер'
+        other: '{count} камеры'
+      events:
+        one: '{count} падение'
+        few: '{count} падения'
+        many: '{count} падений'
+        other: '{count} падения'
+      last_seen: 'последнее {date}'
+      issue: 'Задача, где это чинят'
+      col_soc: 'SoC'
+      col_sensor: 'Сенсор'
+      col_build: 'Ядро · прошивка · majestic'
+      col_cameras: 'Камер'
+      signature: 'Сигнатура {id}'
+      firmware: 'прошивка {list}'
+      how_title: 'Ваша камера упала?'
+      how_1: 'После падения веб-интерфейс камеры на каждой странице пишет «This camera recovered from a crash». Откройте отчёт о падении.'
+      how_2: 'Нажмите «Send to OpenIPC» или скачайте лог падения и отправьте его со'
+      how_club: 'своей страницы в клубе OpenIPC'
+      how_3: 'Привяжите камеру к клубу на Open Wall — и падения, которые она присылает, принесут вам звёзды.'
+      how_privacy: 'Отправляется журнал ядра о падении, версия прошивки и чип. MAC камеры и все IP-адреса в нём заменяются хешами; журналы читают только мейнтейнеры OpenIPC. Без вашего действия ничего не отправляется.'
+      mine_title: 'Мои отчёты о падениях'
+      all_link: 'Все падения'
+      mine_lede: 'Каждая ошибка, на которую попала ваша камера, приносит {report} ★ после расчёта; если вы прислали её первым — ещё {first} ★, когда мейнтейнер подтвердит, и {fixed} ★, когда её исправят. Не больше {cap} ★ в месяц за падения; падение, вызванное нарочно, ничего не приносит.'
+      file_label: 'Лог падения из веб-интерфейса камеры (crashlog_*.tar.gz)'
+      mac_label: 'MAC камеры, если она привязана на Open Wall (необязательно)'
+      send: 'Отправить лог падения'
+      sent: 'Получено, отнесено к'
+      sent_again: 'Это падение уже было получено:'
+      sent_self: 'Получено, но это падение вызвано нарочно (sysrq), поэтому ничего не приносит:'
+      self_inflicted: 'вызвано нарочно'
+      mine_empty: 'Пока ни одного падения — ни от вас, ни от привязанных камер.'
+      triage_link: 'Разбор падений'
+      triage_forbidden: 'Разбирать падения могут только мейнтейнеры OpenIPC.'
+      merged_into: 'объединено с {id}'
+      taken_back: 'Сохранено; списано {n} ★.'
+      saved: 'Сохранено.'
+      f_status: 'Состояние'
+      f_fixed_in: 'Исправлено в (версия прошивки)'
+      f_issue: 'Задача'
+      f_merge: 'Та же ошибка, что сигнатура'
+      f_note: 'Заметка'
+      f_save: 'Сохранить'
+      d_hardware: 'Железо'
+      d_firmware: 'Прошивка'
+      d_kernel: 'Ядро'
+      d_builds: 'Подходящие сборки'
+      d_task: 'Задача ядра'
+      d_interrupted: 'прервана'
+      d_uptime: 'Время работы'
+      d_cmdline: 'Командная строка'
+      d_anomalies: 'Перед этим'
+      d_before: 'Предупреждения'

deploy/nginx/sites-available/org.openipc

@@ -455,6 +455,26 @@ server {
         add_header X-Served-By go always;
     }
 
+    # Kernel crashes cameras recovered from (service/internal/crashes): the
+    # crash log a camera's WebUI sends (a tar.gz of pstore's records, at most
+    # 256 KB), and the public list of signatures, which the web role lets
+    # caches keep for five minutes. The upload is limited per address and per
+    # camera by the web role itself.
+    location ^~ /api/v1/crashes {
+        client_max_body_size 512k;
+        limit_req zone=boards_search burst=20 nodelay;
+        limit_req_status 429;
+        gzip on;
+        gzip_proxied any;
+        gzip_types application/json;
+        proxy_pass http://127.0.0.1:3002;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Request-Id $request_id;
+        add_header Strict-Transport-Security max-age=15768000;
+        add_header X-Served-By go always;
+    }
+
     # The OpenIPC Club (service/internal/club): signing in, the send form,
     # members' own reports and the maintainers' review. Its session cookie's
     # path is this prefix, so nothing else on the site ever receives it, and

deploy/nginx/sites-available/org.openipc.dev

@@ -368,6 +368,27 @@ server {
         add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
     }
 
+    # Kernel crashes cameras recovered from (service/internal/crashes): the
+    # crash log a camera's WebUI sends (a tar.gz of pstore's records, at most
+    # 256 KB), and the public list of signatures, which the web role lets
+    # caches keep for five minutes. The upload is limited per address and per
+    # camera by the web role itself.
+    location ^~ /api/v1/crashes {
+        client_max_body_size 512k;
+        limit_req zone=boards_search burst=20 nodelay;
+        limit_req_status 429;
+        gzip on;
+        gzip_proxied any;
+        gzip_types application/json;
+        proxy_pass http://127.0.0.1:3012;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Request-Id $request_id;
+        add_header Strict-Transport-Security max-age=15768000;
+        add_header X-Served-By go always;
+        add_header X-Robots-Tag "noindex, nofollow, noarchive" always;
+    }
+
     # The OpenIPC Club (service/internal/club): signing in, the send form,
     # members' own reports and the maintainers' review. Its session cookie's
     # path is this prefix, so nothing else on the site ever receives it, and

frontend/apps/site/src/components/club/Club.tsx

@@ -17,6 +17,7 @@ import {
 } from '../../lib/club';
 import { size } from '../../lib/reports';
 import Cameras from './Cameras';
+import Crashes from './Crashes';
 import IpctoolCode from './IpctoolCode';
 import { STATUS_TONE, Stars } from './parts';
 
@@ -273,6 +274,7 @@ function MemberPage({ member, ways, locale, t, onChange }: {
           <dl class="m-0 mt-2 grid gap-0.5 text-[13px]">
             <div class="flex justify-between gap-2"><dt class="text-body-secondary">{t('club.side_from_reports')}</dt><dd class="m-0"><Stars n={member.report_stars ?? member.stars} /></dd></div>
             <div class="flex justify-between gap-2"><dt class="text-body-secondary">{t('club.side_from_wall')}</dt><dd class="m-0"><Stars n={member.wall_stars ?? 0} /></dd></div>
+            <div class="flex justify-between gap-2"><dt class="text-body-secondary">{t('club.side_from_crashes')}</dt><dd class="m-0"><Stars n={member.crash_stars ?? 0} /></dd></div>
           </dl>
         </div>
         {cams && (
@@ -304,6 +306,7 @@ function MemberPage({ member, ways, locale, t, onChange }: {
           )
           : <button type="button" class="w-fit cursor-pointer p-0 text-[13px] text-brand-blue underline" onClick={() => setNaming(true)}>{t('club.rename')}</button>}
         {member.maintainer && <a class="site-btn site-btn-dark site-btn-sm w-fit" href={pathFor(locale, '/club/review')}>{t('club.review_link')}</a>}
+        {member.maintainer && <a class="site-btn site-btn-outline-secondary site-btn-sm w-fit" href={pathFor(locale, '/club/crashes')}>{t('crashes.triage_link')}</a>}
         {telegram && (
           <div class="grid gap-1.5 text-[13px] text-body-secondary">
             <span>{member.quiet ? t('club.bot_muted') : t('club.bot_on')}</span>
@@ -323,6 +326,7 @@ function MemberPage({ member, ways, locale, t, onChange }: {
         {cams
           ? <Cameras data={cams} locale={locale} t={t} onChange={() => { void loadCams(); onChange(); }} />
           : !error && <div class="m-4 h-24 animate-pulse rounded bg-surface-alt" />}
+        <Crashes locale={locale} t={t} onChange={onChange} />
         <div class="flex flex-wrap items-baseline justify-between gap-x-4 gap-y-1 border-b border-hairline px-4 py-3">
           <h2 id="club-mine" class="m-0 text-lg font-semibold">{t('club.mine_title')}</h2>
           <span class="flex flex-wrap items-baseline gap-x-4 gap-y-1 text-sm">