Production-readiness: telemetry, run lifecycle, GTFS-RT, and full cleanup

.env.dev

@@ -3,4 +3,13 @@ DEBUG=True
 DJANGO_SERVE_STATIC=True
 LOG_LEVEL=DEBUG
 CREATE_SUPERUSER=True
-RUN_MIGRATIONS=True
+RUN_MIGRATIONS=True
+
+# MQTT Configuration
+MQTT_HOST=telemetry-broker
+MQTT_PORT=1883
+
+# ETA model registry (dev default; seed once with seed_baseline_model)
+MODEL_REGISTRY_DIR=eta_models
+ETA_MAX_STOPS=10
+ETA_DEFAULT_UNCERTAINTY_S=120

.env.example

@@ -25,6 +25,10 @@ REDIS_PASSWORD=redispassword
 # Google Maps API Key
 API_TOKEN=
 
+# MQTT Configuration
+MQTT_HOST=telemetry-broker
+MQTT_PORT=1883
+
 # HiveMQ Configuration
 HIVEMQ_LOG_LEVEL=INFO
 
@@ -54,4 +58,10 @@ FLOWER_DOMAIN=tasks.databus.simovilab.com
 DOCS_DOMAIN=docs.databus.simovilab.com
 
 # Certificate in production
-CERT_RESOLVER=letsencrypt
+CERT_RESOLVER=letsencrypt
+
+# ETA model registry (used by gtfs_eta; set to a writable directory)
+# Seed the baseline model once: MODEL_REGISTRY_DIR=eta_models ./.venv/bin/python -m gtfs_eta.seed_baseline_model
+MODEL_REGISTRY_DIR=eta_models
+ETA_MAX_STOPS=3
+ETA_DEFAULT_UNCERTAINTY_S=120

.gitignore

@@ -197,3 +197,12 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+# AI tooling artifacts (local only)
+graphify-out/
+llms.txt
+llms-full.txt
+CONTEXT_SYNC_NOTES.md
+
+# Published GTFS-RT / Schedule output (runtime artifacts)
+backend/feed/files/

Makefile

@@ -0,0 +1,17 @@
+.PHONY: lint typecheck test
+
+# ruff needs no Django settings (no settings import), so it runs locally
+# against the backend project without going through Docker.
+lint:
+	cd backend && uv run ruff check .
+
+# mypy loads the django-stubs plugin, which imports databus.settings; that
+# module reads env vars via python-decouple and fails to import outside the
+# dev container, so mypy must run inside it.
+typecheck:
+	docker compose -f compose.dev.yml run --rm orchestrator uv run mypy .
+
+# pytest-django also needs the real settings/DB, so it runs inside the dev
+# container too.
+test:
+	docker compose -f compose.dev.yml run --rm orchestrator uv run pytest -q

backend/README.md

@@ -9,3 +9,61 @@ Django apps:
 - `runs`: Run data (_anima machinae_)
 - `website`: Miscellaneous website features (e.g., admin interface, users, etc.)
 - `feed`: GTFS data handling and processing
+
+## Workspace layout (uv)
+
+This project is a `uv` workspace (see `pyproject.toml` `[tool.uv.workspace]`):
+
+- `gtfs-django`, `gtfs-io` — workspace members, editable installs from the
+  local `backend/gtfs-django/` and `backend/gtfs-io/` directories.
+- `gtfs-eta` — **not** a workspace member. It's an editable *path* dependency
+  (`[tool.uv.sources]`) pointing at `backend/gtfs-eta`, which is a symlink
+  (`gtfs-eta -> ../../gtfs-eta`) to the sibling `simovilab/gtfs-eta` repo
+  checked out next to `databus/`. `compose.dev.yml` bind-mounts
+  `../gtfs-eta:/gtfs-eta` read-write so the OS-resolved symlink target exists
+  inside the container too. `gtfs-eta` is excluded from ruff/mypy/pytest here
+  — it's a separately-maintained codebase with its own lint baseline and test
+  suite.
+
+## Celery services
+
+Four services share this codebase (see `Dockerfile` build targets and
+`compose.dev.yml`):
+
+| Service | Build target | Role | Queue |
+|---|---|---|---|
+| `orchestrator` | `dev` | Django HTTP + admin + REST API | — (not a worker) |
+| `realtime-engine` | `realtime-engine` | Celery worker: MQTT ingestion bootstep, lifecycle events, HTTP polling, staleness scan | `realtime_engine` |
+| `schedule-engine` | `schedule-engine` | Celery worker: builds the two GTFS-RT feeds + the daily GTFS Schedule zip | `schedule_engine` |
+| `scheduler` | `scheduler` | Celery Beat (`databus/celery.py` `beat_schedule`) | — |
+
+Queues are assigned per-task via `@shared_task(queue=...)`, not Celery
+`task_routes`; each worker is started with a matching `-Q <queue>` flag.
+
+## Code quality
+
+Root `Makefile` targets:
+
+```bash
+make lint       # cd backend && uv run ruff check .   — runs locally, no Docker needed
+make typecheck  # docker compose -f compose.dev.yml run --rm orchestrator uv run mypy .
+make test       # docker compose -f compose.dev.yml run --rm orchestrator uv run pytest -q
+```
+
+`typecheck` and `test` run inside the dev container because Django settings
+read env vars via `python-decouple`, which fails outside the container.
+`lint` needs no Django settings import, so it runs locally against the
+backend project.
+
+Ruff enforces missing-docstring rules (`D1`, i.e. `D100`-`D107`) in addition
+to its default `E`/`F` rules. Mypy runs with the `django-stubs` plugin
+(`mypy_django_plugin.main`). Both exclude `migrations/`, `gtfs-eta/`, and
+`.venv/`.
+
+## Tests
+
+445 tests (`pytest --collect-only`, `backend/`):
+
+```bash
+docker compose -f compose.dev.yml run --rm orchestrator uv run pytest -q
+```

backend/api/README.md

@@ -0,0 +1,72 @@
+# API · public REST layer (DRF)
+
+- **Purpose**: the orchestrator's control plane. Exposes run registration and lifecycle
+  transitions, read-only GTFS Schedule resources, two lookup endpoints that back the
+  run-registration UI cascade, and the static realtime OpenAPI schema. Django models: none —
+  `api` is a REST layer over `runs`, `operations`, and `feed` models.
+- **Key modules**:
+  - `views.py` — all ViewSets/APIViews (operations, runs, GTFS Schedule, auxiliary GTFS)
+  - `serializers.py` — DRF serializers, including `CreateRunSerializer` / `RunUpdateSerializer`
+  - `urls.py` — router registrations + custom paths
+  - `realtime.yml` — static GTFS Realtime/MQTT OpenAPI/AsyncAPI schema, served as a file download
+
+## Run registration & lifecycle
+
+- `POST /api/create-run/` (`CreateRunViewSet`) — validates the payload, resolves
+  `vehicle_id`/`operator_id`, creates the `Run` row (implicit `RUN_REQUESTED`), then drives it
+  through `RunLifecycleEvents.VALIDATE_RUN` → `RunLifecycleEvents.INITIALIZE_RUN` via
+  `RunLifecycleService`. Any failed step returns the failing stage (`serialization`,
+  `operational_validation`, `registration`, `gtfs_validation`, `initialization`) with a matching
+  HTTP status (`api/views.py:199-279`).
+- `GET /api/runs/<uuid:run_id>/state/` — current `run_lifecycle_state`.
+- `POST /api/runs/<uuid:run_id>/update/` (`RunUpdateViewSet`) — advances a run's FSM. The `event`
+  field takes the **lowercase** `RunLifecycleEvents` value (e.g. `run_confirmed_by_operator`, not
+  the enum member name); `RunUpdateSerializer` validates against `RunLifecycleEvents` and any
+  extra `details` are flattened into the payload before `process_event` (`api/views.py:303-366`).
+- `GET /api/runs/<uuid:run_id>/history/` — ordered `RunLifecycleTransition` audit log.
+
+## GTFS Schedule + registration-UI lookups
+
+- Router-registered read/write resources for `agency`, `stops`/`geo-stops`, `shapes`/`geo-shapes`,
+  `routes`, `calendars`, `calendar-dates`, `trips`, `stop-times`, `fare-attributes`, `fare-rules`,
+  `feed-info` (all backed by `feed.models`, unauthenticated).
+- `GET /api/service-today/?date=YYYY-MM-DD` — active GTFS `service_id`s for a date.
+- `GET /api/which-shapes/?route_id=` — step 1 of the registration cascade: distinct
+  `GeoShape`s used by a route's stops in the current (`is_current=True`) feed.
+- `GET /api/find-trips/?route_id=&service_id=&shape_id=` — step 2: candidate trips, each tagged
+  with its matching `Run`'s lifecycle state (or `"UNKNOWN"`).
+
+## Operations resources
+
+Router-registered ViewSets over `operations.models`: `company`, `operator`, `data-provider`,
+`vehicle` (filterable by `company`), `equipment`, `equipment-log` (filterable by `equipment`,
+`data_provider`, `vehicle`), plus `position`, `stop-status`, `occupancy`, `congestion` from
+`runs.models`. All except `CompanyViewSet` use `TokenAuthentication`.
+
+## Auth & docs
+
+- `POST /api/login/` — username/password → DRF auth `Token` + operator basic info.
+- Global default authentication is `TokenAuthentication` (`databus/settings.py` `REST_FRAMEWORK`).
+- `GET /api/docs/schema/` — serves the **static** file `api/realtime.yml` as a download
+  (`api/views.py:89-94`); this is a hand-maintained schema, not the dynamic drf-spectacular route.
+- `GET /api/docs/` — ReDoc page (`SpectacularRedocView`, clickjacking-exempt so it can be
+  embedded). Note `drf_spectacular`'s dynamic OpenAPI generation is installed (`SPECTACULAR_SETTINGS`
+  in `databus/settings.py`) but the `docs/schema/` route intentionally serves the static YAML
+  instead of the generated schema — the dynamic route is unfinished.
+
+## Configuration
+
+No app-specific env vars; auth/schema settings come from `databus/settings.py`
+(`REST_FRAMEWORK`, `SPECTACULAR_SETTINGS`).
+
+## Tests
+
+```
+docker compose -f compose.dev.yml run --rm orchestrator uv run pytest api/ -q
+```
+`api/tests/` covers `company`, `find-trips`, and `which-shapes`; `make test` runs the full suite.
+
+## Docs
+
+- [REST API](../../docs/content/interfaces/rest-api.md)
+- [Run lifecycle states](../../docs/content/runs/lifecycle-states.md)

backend/api/admin.py

@@ -1,3 +1,3 @@
-from django.contrib import admin
+"""Django admin registrations for the api app (none; api has no models of its own)."""
 
 # Register your models here.

backend/api/apps.py

@@ -1,6 +1,10 @@
+"""Django app configuration for the api app."""
+
 from django.apps import AppConfig
 
 
 class ApiConfig(AppConfig):
+    """Configure the api app (DRF ViewSets/serializers exposing GTFS and run data)."""
+
     default_auto_field = "django.db.models.BigAutoField"
     name = "api"

backend/api/models.py

@@ -1,3 +1,3 @@
-from django.db import models
+"""Django models for the api app (none; api is a REST layer over feed/runs/operations models)."""
 
 # Create your models here.