Skip to content

Restart containers automatically when a camera stops producing fresh images #423

Closed as not planned

Description

@MateoLostanlen

This issue is part of the epic pyronear/roadmap#16 (Engine robustness).

Follow-up to #422. Root causes get fixed there, this is the safety net for the next unknown one.

Problem

Every current health signal said "healthy" during the 2026-09-21 outage on bellefosse:

  • Camera API /health returns 200 even with a patrol thread blocked forever.
  • Engine healthcheck greps confidence|Nighttime detected in engine.log, which matches while the engine sleeps as "night" in daylight.
  • The "engine.log moved in the last 70 min" check from fix(compose): restart the engine when its log goes stale #419 catches a frozen process, not an engine that logs one line per hour while doing nothing.

The one signal that covers a stuck thread, an unreachable camera, a wrong night detection and any future bug is the outcome: a camera that has not produced a fresh image for N minutes during daytime.

Proposal

  1. Camera API: keep a last_capture_at per camera. /health returns 503 when a patrol is expected to run and the last stored image is older than 5 min. Compose healthcheck becomes curl -sf /health || kill -9 -1 so restart: always brings the container back (Docker never restarts an unhealthy container by itself).
  2. Engine: keep the 70 min log check from fix(compose): restart the engine when its log goes stale #419, and add: unhealthy when is_day has been false for more than 2 h while the sun is up (sunrise/sunset already in the cache from get_sunset_sunrise.sh).
  3. Optional, server side: a Grafana alert on last_active_at per camera in the alert API, as a third net that does not depend on the Pi.

Trade-off

A camera that is really down would restart the camera API every 5 min, which also interrupts the other camera's patrol for about 20 s. Either accept it, or skip the check for cameras that do not answer ping.

Activity

  1. MateoLostanlen commented on Oct 22, 2026

    @MateoLostanlen
    MemberAuthor

    Merged into #446.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions