fix: harden Redis-backed delivery, public APIs, and production service settings by Felipequesada21 · Pull Request #71 · simovilab/infobus

Felipequesada21 · 2026-09-03T17:44:24Z

Summary

This branch addresses production-readiness gaps around unauthenticated Redis access, unbounded Redis Streams, mutable public REST resources, and fragile external-feed requests. It also makes the production Compose topology more explicit by adding the stream consumer, pinning selected image tags, and keeping the RabbitMQ management service on the internal network only.

The system now carries configured Redis credentials through Celery, Channels, cache, and direct Redis clients; applies stream caps; exposes registered model APIs as read-only; throttles public endpoints; and bounds external requests while redacting credentials and query strings from feed URLs in logs. These changes are covered by the targeted application suites in this branch.

What changed

Redis credentials and stream bounds

Redis credentials are configured for the Celery broker, Channels layer, and Django cache, and direct clients in run and update services receive the configured password. backend/infobus/settings.py:139, backend/infobus/settings.py:193, backend/infobus/settings.py:230, backend/runs/services/lifecycle.py:56, backend/updates/consumers.py:17
Lifecycle events and dead-letter entries are capped using configured stream maximum lengths. backend/runs/services/lifecycle.py:549, backend/runs/services/state.py:115, backend/updates/client.py:43
The production Compose definition adds a dedicated Redis Streams consumer, pins RabbitMQ, Redis, and documentation-server images, and adds health-check start periods. compose.prod.yml:115, compose.prod.yml:140, compose.prod.yml:154, compose.prod.yml:184, compose.prod.yml:257

Public API hardening

Registered model endpoints use ReadOnlyModelViewSet, preventing create, update, and delete actions through those public viewsets. backend/api/views.py:59, backend/api/views.py:265, backend/api/views.py:443, backend/api/tests.py:83
Anonymous, realtime, and geometry throttles are configured; realtime and geometry views declare their scopes; and the bundled schema download is restricted to GET. backend/infobus/settings.py:211, backend/api/views.py:72, backend/api/views.py:295, backend/api/views.py:507, backend/api/tests.py:142

Feed polling and diagnostic safety

Schedule and realtime HTTP requests use the configured timeout, and feed URL logging removes credentials and query strings. backend/infobus/settings.py:174, backend/engine/tasks.py:94, backend/engine/tasks.py:99, backend/feed/services/schedule.py:49, backend/feed/services/schedule.py:67, backend/infobus/utils.py:6
Stop-time-update decoding records malformed Redis payload type and length before returning an empty result. backend/updates/builders/stop/stop_time_updates.py:41, backend/updates/builders/stop/stop_time_updates.py:49, backend/updates/builders/stop/stop_time_updates.py:58

Verification

Targeted Django tests passed: updates 87, runs 13, engine 9, and api 5, for 114 total tests. No test failures occurred.
ruff check . is failing with 54 errors, including F401 django.shortcuts.render at updates/views.py:1; 10 findings are marked fixable by Ruff.
docker compose -f compose.prod.yml config --quiet passed but reported unresolved variable names REDIS_PASSWORD, CONTEXT_DOMAIN, KNOWLEDGE_DOMAIN, ORCHESTRATOR_DOMAIN, DOCS_DOMAIN, and UI_DOMAIN. docker compose -f compose.dev.yml config --quiet passed without warnings.
Live WebSocket delivery was observed for mbta.route.vehicle_positions.by_route.Red.by_direction.0: an initial snapshot was followed by messages at approximately 28-second intervals.

Known issues not addressed in this PR

PROD-D-01 — WebSocket origin validation and authentication are absent, blocking production. The ASGI application routes WebSockets directly through URLRouter, and the consumer accepts connections immediately; neither an origin validator nor an authentication wrapper is present. backend/infobus/asgi.py:22, backend/updates/consumers.py:39, backend/updates/routing.py:5 This requires an explicit access-control design and is outside this production-settings change set.
The transit_system topic segment is not validated against TransitSystem.code. TopicKey.parse checks topic shape and non-empty segments only, while projection validation does not perform this database lookup; this batch also validated a syntactically valid topic with an unknown system successfully. backend/updates/topics.py:20, backend/updates/topics.py:37, backend/updates/planner.py:25, backend/feed/models.py:38 Adding that lookup changes subscription semantics and is outside the branch.
All 9 Feed records have a null transit_system. The model permits the nullable relation, and the read-only count returned 9. backend/feed/models.py:121, backend/feed/models.py:128 Backfilling or constraining existing schedule data is outside this branch.
FeedMessage.timestamp uses auto_now. This records save time rather than an immutable ingestion timestamp. backend/feed/models.py:143, backend/feed/models.py:157 Changing the data model and its historical meaning is outside this PR.
The scheduled-trip query leaves service_id commented out. backend/feed/services/queries.py:191 Restoring service filtering needs schedule-data validation and is outside this branch.
Celery tasks have no time_limit, retry, or backoff configuration. The task decorators in engine/tasks.py are bare @shared_task declarations, and a repository search found no task-level time_limit, retry, or backoff configuration. backend/engine/tasks.py:46, backend/engine/tasks.py:67, backend/engine/tasks.py:127 Delivery policy requires separate operational design.
/health/ does not verify dependencies. It returns OK without checking PostgreSQL, Redis, RabbitMQ, or upstream services. backend/infobus/urls.py:23 Dependency health policy is outside the branch.
feed_stoptimeupdate has no retention path. The ingestion code bulk-creates rows and the scheduled task exports them to Parquet without deleting database rows. backend/feed/services/realtime.py:220, backend/engine/tasks.py:257, backend/feed/services/data.py:289 The previously cited ~11.8 million rows/day estimate is not supported by this batch: a read-only count by FeedMessage.timestamp measured 46,726,888 rows in the last 24 hours. Retention and archival policy are outside this PR.

Additional API and Redis hardening

GeoJSON stop and shape endpoints now use GeoJsonPagination, preserving FeatureCollection responses under global
pagination.
DRF_NUM_PROXIES is configured so throttling identifies real client IPs behind Traefik.
Redis-backed DRF throttles fail open when Redis is unreachable, preventing cache outages from returning API 500 responses.
Development Redis now requires REDIS_PASSWORD and authenticates its healthcheck; production Compose and Django both fail
fast if the password is missing.
RouteStopView now uses the geometry throttle scope.
drf-spectacular is configured as DRF’s default schema class, resolving the deployment-check schema error.
Feed polling logs now record only the exception type, avoiding sensitive URLs embedded in request exception messages.
.env.example now documents cache, timeout, throttling, pagination, proxy, and Redis Stream settings.

Additional verification

Full Django suite passed: 122 tests, zero failures.
infobus URL-redaction tests passed: 5 tests.
/api/geo-stops/ and /api/geo-shapes/ were verified to return GeoJSON FeatureCollection payloads.
Both resilient throttle classes were verified to allow requests when Redis raises ConnectionError.
manage.py check --deploy no longer reports drf_spectacular.E001.
ruff check . still reports 54 pre-existing findings; no new findings were introduced in modified files.

Notes for the reviewer

Agregar una clave local real a .env:

REDIS_PASSWORD=una-clave-local-larga-y-unica

Luego recrear Redis y levantar de nuevo:

docker compose -f compose.dev.yml up -d --force-recreate memory
./scripts/dev.sh

REDIS_PASSWORD must be defined on the server. With it unset, the production Compose command expands to --requirepass "", leaving Redis without authentication. compose.prod.yml:185

Add outbound HTTP timeouts, redact credentials from logged feed URLs, bound Redis stream growth, remove invalid API filter fields, and log silent decoding failures. - api: drop filterset_fields that do not exist on FareAttribute and reduce FareRule to route_id. Both routes returned HTTP 500 for any query-parameter request because django-filter validates the whole field list before applying any filter. - infobus: add redact_url helper and HTTP_REQUEST_TIMEOUT_SECONDS, REDIS_EVENTS_STREAM_MAXLEN and REDIS_DEAD_LETTER_STREAM_MAXLEN settings. - engine: add the missing timeout to the vehicle positions request and replace the two repeated literals with the setting. Redact URLs in the three error logs. - feed: add timeouts to the schedule HEAD and GET requests, use an aware timestamp in the import log, and redact the schedule URL. - runs, updates: bound the events and dead-letter streams with approximate MAXLEN. Measured 4.1M entries accumulated over 29 days with no trimming policy. - updates: log a warning when a stop time updates document fails to decode, instead of silently returning an empty list. Two tests added. Test suites: updates 84, runs 12, engine 8, all passing. No migrations, no model changes, no public contract additions.

SECURE_CONTENT_TYPE_NOSNIFF and SECURE_REFERRER_POLICY were declared with config() defaults below Django's own, which actively suppressed two response headers that the framework emits out of the box. - SECURE_CONTENT_TYPE_NOSNIFF: default False -> True (Django default) - SECURE_REFERRER_POLICY: default None -> "same-origin" (Django default) Both settings remain overridable through the environment; no .env file is modified. The remaining seven settings in the block keep insecure defaults on purpose, so that local development over plain HTTP is not broken; their production values belong in .env.prod. Verified in dev after restarting orchestrator: /health/ now returns X-Content-Type-Options: nosniff and Referrer-Policy: same-origin. manage.py check --deploy drops security.W006 and security.W022 (8 issues -> 6). Test suites unchanged: updates 84, runs 12, engine 8.

Three independent, declarative changes to compose.prod.yml. No production deployment exists yet, so these were verified by configuration render only. - Pin floating image tags to exact versions, verified to exist in the registry: rabbitmq:4-management -> 4.3.5-management, redis:7-alpine -> 7.4.11-alpine, nginx:alpine -> 1.31.4-alpine. Redis deliberately stays on the 7.x line; the move to 8.x is a separate change that must ship together with the streams-consumer service. The knowledge service builds its image locally and has no registry tag to pin. - Add start_period: 60s to the broker, database and memory healthchecks. These had no start period, leaving roughly 50s of grace. The setting only makes a healthcheck more permissive and is a precondition for adding a healthcheck to orchestrator, which takes about 67s to boot. - Remove the traefik_proxy network and the eight Traefik labels from the broker service, so the RabbitMQ management UI is no longer published on a public domain. The service stays on the internal network; engine and scheduler reach it unchanged through depends_on. This does not retire RabbitMQ, which remains pending a product decision. Verified with docker compose -f compose.prod.yml config: render completes, broker declares only the internal network and no labels, orchestrator, user-interface, context, knowledge and docs keep traefik_proxy, and the BROKER_DOMAIN warning is gone because the labels that referenced it are.

- Replace API ModelViewSets with ReadOnlyModelViewSets - Restrict the schema endpoint to GET requests - Configure anonymous and scoped throttling for realtime and geometry endpoints - Add pagination defaults for REST responses - Configure Redis-backed Django cache with optional password support

Add API tests for read-only router methods, paginated list responses, geometry throttling, and schema GET-only access. Isolate cache-backed throttling with LocMemCache to avoid development Redis writes.

Adds the streams-consumer service to the production manifest, following the engine/scheduler pattern, and moves memory from redis:7.4.11-alpine to redis:8.10.1-alpine, which bundles RedisJSON. Both changes ship together: the consumer path calls RedisJSON commands, so deploying the consumer against Redis 7 would fail after dispatch and leave every entry pending, since XACK happens after dispatch. The consumer still cannot authenticate against a password-protected Redis (backend/updates/client.py:83-88), and the Channels layer omits credentials (backend/infobus/settings.py:226-233). The service must not be deployed until REDIS_PASSWORD reaches both.

Propaga REDIS_PASSWORD al broker de Celery y a Channels; conserva el comportamiento de desarrollo cuando la contraseña está vacía.

Pass REDIS_PASSWORD (or None when unset) to the ten direct Redis clients in runs and updates, preserving existing decode_responses behavior.

Restore GeoJSON pagination for geometry endpoints and align route-stop throttling. Harden Redis configuration across dev and production: - require Redis authentication in dev and fail fast in production - add resilient DRF throttles that fail open if Redis is unavailable - respect client IPs behind Traefik with DRF_NUM_PROXIES Prevent feed URL leaks in error logs, configure drf-spectacular's schema class, and document new Redis, throttling, pagination, and stream settings. Add coverage for URL redaction edge cases.

…g and logging

Felipequesada21 added 15 commits September 1, 2026 18:40

test: cover read-only endpoints and throttling

ae8168f

Add API tests for read-only router methods, paginated list responses, geometry throttling, and schema GET-only access. Isolate cache-backed throttling with LocMemCache to avoid development Redis writes.

fix: missing space

cf4459c

docs(settings): correct event stream retention estimate to ~14 hours

2756bd9

docs(api): describe viewsets as read-only instead of CRUD

710e4af

fix: pass Redis credentials to the Celery broker and channel layer

eea2f62

Propaga REDIS_PASSWORD al broker de Celery y a Channels; conserva el comportamiento de desarrollo cuando la contraseña está vacía.

fix: fpass credentials to direct Redis clients in runs and updates

78e311e

Pass REDIS_PASSWORD (or None when unset) to the ten direct Redis clients in runs and updates, preserving existing decode_responses behavior.

fix: pass credentials to the WebSocket consumer Redis client

eabf125

test(runs): verify Redis client credentials and decoding mode

4636ce0

test(engine): verify the Celery broker URL carries Redis credentials

5597453

test(updates): verify Redis client credentials and channel layer config

7cdcfeb

Felipequesada21 requested review from Kroenenn, dotjae and fabianabarca September 3, 2026 17:44

Felipequesada21 self-assigned this Sep 3, 2026

Felipequesada21 added the enhancement New feature or request label Sep 3, 2026

Felipequesada21 and others added 6 commits September 4, 2026 12:02

Fix pagination ordering for GTFS viewsets

1b54444

fix: enhance vehicle positions and alerts fetching with error handlin…

63594d9

…g and logging

Make GTFS Schedule import atomic and validate required members

46656bd

Isolate schedule import failures per feed publisher

2e58495

Add tests for atomic GTFS Schedule import

4ca0156

Navigation Menu

Uh oh!

fix: harden Redis-backed delivery, public APIs, and production service settings - #71

fix: harden Redis-backed delivery, public APIs, and production service settings#71
Felipequesada21 wants to merge 21 commits into
mainfrom
fix/production-settings

Felipequesada21 commented Sep 3, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Felipequesada21 commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Redis credentials and stream bounds

Public API hardening

Feed polling and diagnostic safety

Verification

Known issues not addressed in this PR

Additional API and Redis hardening

Additional verification

Notes for the reviewer

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Felipequesada21 commented Sep 3, 2026 •

edited

Loading