Skip to content

Release v0.2.3 - #91

Merged
an1ndra merged 36 commits into
mainfrom
Develop
Nov 20, 2026
Merged

an1ndra merged 36 commits into
mainfrom
Develop

Conversation

@an1ndra

@an1ndra an1ndra commented Nov 20, 2026

Copy link
Copy Markdown
Owner

Release v0.2.3.

Notes in changelog/v0.2.3.md. Highlights since v0.2.2:

  • path-spring-ai: a Spring AI ChatModel/EmbeddingModel over the gateway, as a sibling module under a new path-parent POM
  • native Gemini for AI Studio and Vertex: thinking budgets, safety settings, cached content, service-account auth
  • daily digest email; bucketed /admin/stats; from/to on the usage export
  • Prometheus rules with the chart; path_errors_total, path_fallbacks_total, path_provider_circuit_open, path_alerts_sent_total
  • webhook retries and bounded alert queues; provider error-rate alert; shared Valkey circuit/concurrency state
  • scopes in /admin/settings; email alerting and console mail settings; SSO proxy auth
  • release images are scanned before the version tag exists
  • the pentest and audit-log security fixes

Migrations V19 and V20 run on startup. The Postgres partitioning script is opt-in and is not run by the upgrade.

Verified locally: ./mvnw verify (251 tests, both modules), the full smoke suite (157/157), promtool check rules, and helm template with every toggle.

The console is a separate repository (Vite/React, static), so this repository documents the
contract instead: token auth with per-person scoped tokens, the CORS setting for a separate
hostname versus routing both behind one, and the warning that /admin must not be exposed
without Access, a VPN or an allowlist.
SSO (Google, GitHub, enterprise OIDC), 2FA and passkeys belong to an identity provider, not
to an LLM gateway. GATEWAY_PROXY_AUTH=true makes /admin/** trust the identity header an
authenticating proxy injects (Authentik, Keycloak, oauth2-proxy, Cloudflare Access), so
those features arrive with the IdP that already implements them properly.

- fails closed: no header means 401 proxy_identity_missing, even if the admin token is sent
- GATEWAY_PROXY_AUTH_SCOPES=read gives a viewer group that cannot write
- audit entries are attributed to the proxy identity, not "shared-admin"; resolved per
  request from the principal
- /admin/settings reports auth_mode so the console can skip the token form
- a WARN at startup repeats that the gateway must not be reachable except through the proxy
- docs/SSO.md: Authentik + Traefik forward-auth, Keycloak/oauth2-proxy, Cloudflare Access,
  and the spoofing caveat for each

Tests: 170 (6 new) covering closed-by-default, token-not-enough, identity access, audit
attribution and read-only scopes.
The insertion anchor did not match, so the WARN that tells an operator the gateway now
trusts a header was compiled out. Assert the anchor, and verify the line appears in a real
startup log rather than assuming the edit landed.

    WARN DataSeeder - Admin auth is DELEGATED to a reverse proxy (X-Forwarded-User):
    the gateway trusts that header for /admin/**. It must not be reachable except
    through the proxy.

Verified with the gateway running in proxy mode: no header -> 401 proxy_identity_missing,
token alone -> 401, identity header -> 200 and auth_mode=proxy, and a provider created
through the proxy is audited to anindra@example.com.
provider.create and provider.update pass the whole input record to audit.record(), which
serialised it verbatim — so the plaintext provider key landed in admin_audit.details, a
table readable through GET /admin/audit. That defeats the AES-GCM encryption at rest the
key is stored with.

Sanitised centrally in AuditService: any field whose name matches api_key, apiKey,
password, secret, token, authorization, new_key or private becomes "***", at any depth.
Doing it in the service rather than at the call sites means a future admin endpoint cannot
forget.

Verified against a running gateway: a provider created with sk-live-SHOULD-NEVER-APPEAR
audits as "api_key":"***" while the credential itself still works (hint ****PEAR).

Anyone who created providers before this commit has keys in their audit table: rotate
those provider keys.

Tests: 172 (2 new).
Alerts were webhook-only, which is fine for a Slack channel and useless for the operator
who is not looking at it.

- GATEWAY_ALERT_EMAIL_TO (+ SPRING_MAIL_*) adds an email channel next to the webhook. They
  are independent: either, both or neither. Fire-and-forget on a virtual thread; a failing
  mailbox logs a warning and never touches the request.
- one message per alert, subject "[Path] <event>", body as "key: value" lines
- budget alerts keep their deduplication, so a busy project still sends one email a period
- POST /admin/alerts/test sends a test alert and reports which channels are configured, so
  the config is proven before you depend on it
- ObjectProvider<JavaMailSender>: the gateway still boots with email alerting off

Verified against a fake SMTP server: subject "[Path] alert.test", To ops@example.com, body
carrying the payload. Tests: 177 (5 new).
Alert mail was environment-only, which means a restart to change a recipient.

- GET/PUT /admin/mail-settings: host, port, username, password, from, recipients, enabled.
  Stored in a single row, winning over SPRING_MAIL_*, which keeps working as the
  config-file route; the view reports which source is in force.
- the SMTP password is AES-GCM encrypted with the master key, write-only in the API
  (has_password, never the value); blank keeps it, "-" clears it
- enabling requires a host and at least one recipient, checked before saving
- /admin/alerts/test sends through whatever is configured, console or environment
- audit records the change without the password

Found while clicking through it: PUT was missing from the CORS methods, so the console's
save failed in the browser with "Failed to fetch" while curl worked. The error message the
console produced named CORS as a possible cause, which is how it was diagnosed.

Tests: 183 (11 new). Verified end to end against a fake SMTP server: save in the UI, test
email delivered, subject "[Path] alert.test".
- Dockerfile gains curl in the runtime stage and a HEALTHCHECK on /actuator/health, so
  Dokploy and Docker can act on readiness instead of only "container is running"
- deploy/dokploy/compose.yml: gateway + Postgres (+ optional Valkey), health-gated
  depends_on, stop_grace_period above the drain timeout, volumes for the database and the
  master key
- deploy/dokploy/README.md: both apps end to end, the environment variables that matter,
  same-hostname routing that removes CORS, and the pre-public checklist (Access, API docs,
  master key, drain)

The console keeps its own Dockerfile and /healthz (path-admin) and is deployed as a second
Dokploy application.
'required:' on env_file needs compose >= 2.24, and this should not fail on an older Dokploy
compose. Provider keys go in as environment variables or through the console.
…pose file

Dokploy created no router for the compose service: the gateway container came up
with zero traefik.* labels, so path.hacksubset.com returned 404 while the app was
healthy. Declaring the labels here makes routing part of the deployment instead of
a UI step that has to be repeated after every domain change.

TLS uses the server default certificate, which is the Cloudflare Origin certificate
when Cloudflare proxies the hostname.
SPRING_MAIL_HOST is passed through as an empty string by the Dokploy compose, which
still counts as 'set' and activates Spring's mail health indicator. It then probes an
empty host, gets Connection refused, and /actuator/health returns DOWN — so Docker
marks the gateway unhealthy and the stack restarts it for a channel that is optional.

Mail is configured from the console (settings/alert-email) via JavaMailSender built at
runtime, so the stock indicator never reflects it anyway.
DataSeeder ran ensureProviders() unconditionally on every boot, so a fresh
deployment showed openai, anthropic, groq and gemini in the console even though
nobody had configured them, plus a ten-row route table. That reads as demo data
in what is supposed to be an empty, real console.

gateway.seed-defaults (GATEWAY_SEED_DEFAULTS, default true so tests and the local
quickstart are unchanged) turns it off. Deleting the rows is not enough on its own:
the seeder recreates them whenever the provider table is empty.
1. Admin auth limiter bypass (CWE-307). The per-address counter trusted the first
   X-Forwarded-For hop, which a client supplies, so rotating it (or arriving from a
   rotating Cloudflare edge) reset the counter on every request and thousands of token
   guesses were evaluated against a nominal cap of 10. X-Forwarded-For is now ignored
   unless gateway.client-ip-header names a trusted edge header (CF-Connecting-IP), and a
   process-wide cap (gateway.admin-auth.global-max-failures, default 200 per window)
   refuses all admin authentication once hit, whatever address it claims.

2. Unauthenticated Actuator. /actuator, /metrics, /prometheus and /health/providers
   answered anonymously, exposing the framework version, datasource internals, repository
   names and provider topology. Only GET /actuator/health is public now, and it reports
   status alone; everything else under /actuator needs the admin token.

3. Unauthenticated /v1/models. The catalogue names every model and alias, so
   gateway.public-models now defaults to false. Opting back in still works
   (PublicModelsTest).

Tests: 188 pass, including the rotating-address case that reproduces the bypass.
…in bypass

Follow-up to the pentest fixes:

- The global failure cap was fail-closed and therefore a denial-of-service against the
  operator: one scanner spending 200 guesses locked the console for everyone in the
  window (observed live). It now defaults to 0 (off); the per-address cap is the control
  that matters once the client address can be trusted.
- That trust required more than config: the origin answers on its own public IP, so
  Cloudflare -- and its bot challenge -- can be skipped entirely, and CF-Connecting-IP
  forged. The gateway router now carries the cf-only middleware (Cloudflare's published
  ranges), declared here so a redeploy cannot silently drop it.

Per-address limiting now works: one source gets 10 attempts per window, and the operator
is not collateral damage.
Direct-to-origin access is what made the pentest findings exploitable and makes
client-IP headers forgeable. The compose label is in the repo; the middleware and the
console's router are not, so this documents both the setup and the fact that a Dokploy
domain change silently drops cf-only from the console router.
…er cache API responses

Pentest findings #132 and #124.

#132 -- the data plane answered 'missing_api_key', 'invalid_api_key' and 'key_disabled'
for the three kinds of rejection, so a client could test whether a key existed, and each
guess cost one database lookup with no budget: 193k guesses were measured at ~474/s with
zero 429s, heavy enough to induce a proxy outage. All rejections now return one identical
'error' object, and ClientAuthLimiter caps rejected keys per address per minute before the
lookup happens. The global cap is off by default for the same reason the admin one is:
a fail-closed global cap is a denial-of-service against legitimate clients.

#124 -- a CDN caches by URL, so a static extension on an authenticated URL is stored and
served to anonymous clients for hours. NoStoreFilter now marks /v1, /admin and /actuator
responses no-store before the chain runs, so error paths are covered too and the edge has
nothing to keep.

FailureWindow is shared by both auth gates; AdminAuthLimiter keeps its public shape.

Tests: 192 pass (oracle, limiter, and no-store now covered).
A missing Authorization header skipped the hashed lookup while an unknown key performed
one, so the bodies were identical but the cost was not: 'no header' was consistently
faster. The lookup now runs for every rejection, and a test asserts the two responses are
byte-for-byte equal.

Found while reading the auth path against the pentest's timing-side-channel track, not
from a report.
The wrapper is distributionType=only-script, so there is no maven-wrapper.jar for the IDE's
Maven server to use. It resolves its own home, bootstraps Maven without the plexus artifact
bindings, and dies with 'Guice/MissingImplementation: ArtifactHandlerManager'. Recording the
symptom and the fix (point Maven home at the extracted 3.10.1) because it costs an hour to
rediscover.
A single fire-and-forget POST lost an alert on a 500, a 429 or a two-second
network blip, and the operator believed they were monitored.

- three attempts with jittered exponential backoff (250ms, 1s, 4s)
- retry only 429, 5xx and connect/timeout; never a 4xx
- honour Retry-After (seconds or HTTP date), capped at 30s
- two workers and a 64-entry queue; a full queue drops and counts instead
  of blocking the caller or growing without bound
- path_alerts_sent_total{outcome=delivered|retried|dropped}

Closes #87
A read-only token could read everything but could not write, and the console
had no way to know: it showed New key, Delete and Revoke, and the operator
found out through a 403 after the click.

GET /admin/settings now returns subject (token name or SSO identity) and the
caller's scopes with can_write, for both token and proxy auth. Scopes and a
name are not credentials.

Closes #85
The console built its requests-over-time chart from the last 1000 usage rows,
so above a thousand calls in the window the chart was a sample, and every
client reimplemented the same bucketing.

GET /admin/stats?window=24h&bucket=1h now returns a series of {from, requests,
cost_usd, failures, p95_latency_ms}. Buckets are 1m, 5m, 1h or 1d; an invalid
value is a 400 naming the accepted ones. Empty buckets are zero-filled, the
window is clamped to 500 points, and the whole series comes from one query.
Omitting bucket leaves the response unchanged.

Closes #84
The only provider alert was provider.circuit_open, which needs consecutive
failures. A provider failing 30% of calls with a success in between never
tripped it, so requests were silently retried onto a fallback while cost and
latency rose.

A one-minute check now reads the routing EWMA, aggregates it per provider by
sample count, and fires provider.error_rate_high above
GATEWAY_ALERT_ERROR_RATE (0.25) with at least GATEWAY_ALERT_MIN_SAMPLES (20)
recent calls. One alert per provider per hour; provider.error_rate_normal
closes it once the rate falls under half the threshold.

Closes #83
The metrics existed but no recording or alerting rules did, so every operator
had to derive the PromQL, and most did not.

- deploy/prometheus/path-rules.yml: fallback >20%, cache hit ratio under half
  its 7-day average, spend on track for 2x yesterday, error rate >2%, provider
  parked >15m; each with a for-duration and a summary/runbook annotation
- Helm prometheusRule.enabled renders the same rules as a PrometheusRule
- CI runs promtool check rules and renders the toggle
- new path_fallbacks_total counter and path_provider_circuit_open gauge supply
  the two rules that had no metric
- docs/OPERATIONS.md explains each alert and what to do

Closes #86
/actuator/prometheus has been gated behind the admin token since the pentest
fix, but this check still called it anonymously, so the end-to-end suite
reported a 401 failure that had nothing to do with the code under test.

Found while running the smoke suite for #84 and #86.
AlertService now has a second constructor for the bounded-queue test, so the
container needs to be told which one to use.
The EWMA is what makes the rate recent; the sample count only proves enough
traffic has been seen to trust it. Say so instead of implying a sliding
window that does not exist.
path_requests_total only knows success/error, so the 5xx alert shipped in #86
was really an overall error-rate rule: a route returning 400s and a provider
falling over with 500s looked identical.

recordError now takes the status that produced the failure and increments
path_errors_total{provider,class=4xx|5xx|other,code}. The alert rule divides
the 5xx series by all requests, so a client-side 400 can no longer page.

Closes #88
Email started one virtual thread per alert, so a hung SMTP server grew one
thread per alert with no ceiling and no way to see that mail was lost.

Email now shares the webhook's shape: two workers, a 64-entry queue, daemon
threads, and a drop that is counted. path_alerts_sent_total gains a channel
tag (webhook|email); email reports delivered/dropped only, because retrying an
SMTP send can duplicate a message the server already accepted.

Closes #89
/admin/stats read cache.semantic_hits from a LongAdder in the JVM: it reset on
every restart, differed per replica and never reached Prometheus. Every hit
was already a usage row, but nothing recorded which tier served it.

- usage_records.cache_tier (V19): exact or semantic, null otherwise
- recordCacheHit sets it and tags path_cache_hits_total{model,tier}
- /admin/stats and per-project stats count semantic hits from the window
- the in-process counter is gone; the row is the source of truth

Verified against the smoke upstream: semantic_hits=1, and
path_cache_hits_total{tier="semantic"}=1 for the same window.

Closes #90
Threshold alerts only fire when something is already wrong, and nobody opens
a dashboard every morning, so a slow spend drift is noticed on the invoice.

One email a day (GATEWAY_DIGEST_CRON, default 08:00 gateway time) summarises
yesterday: requests with the day-over-day delta, spend, tokens with the cache
hit ratio and savings, failures with the top error code, the slowest model's
p95, the top three models by spend, and any provider parked at send time.

- silently off without the alert email channel; nothing is sent for a day
  with no traffic, and a mailbox failure only logs a warning
- GATEWAY_DIGEST=false disables it
- POST /admin/digest/test?date=... sends it on demand and returns the text,
  so the schedule can be checked before the first morning
- rendered text is pinned by a unit test for the documented numbers

Closes #82
The workflow pushed the version tag and latest before Trivy ran, so a failed
scan still published the version: v0.2.0 and v0.2.1 each burned a number that
way.

- build and push only ghcr.io/<repo>:sha-<commit>
- Trivy gates that tag
- on success, docker buildx imagetools create copies the same digest to
  <tag> and latest; nothing is rebuilt
- the recorded digest is the one that was scanned
- sha- is kept for traceability

Closes #80
usage_records grows forever, and the nightly retention delete competes with
the indexes that /admin/stats scans.

- deploy/postgres/partitions.sql converts usage_records to monthly range
  partitions (same columns, identity and indexes, plus a DEFAULT partition)
  and ships path_ensure_usage_partitions for a monthly cron; documented as a
  maintenance-window step, not a Flyway migration, because it rewrites the
  table
- /admin/usage/export accepts from/to, so the exact slice a purge is about to
  delete can be archived for finance; window still works when they are absent
- docs/OPERATIONS.md covers retention, the archive loop, the dump-first
  procedure and detaching an archived month

Retention itself already existed (GATEWAY_USAGE_RETENTION, nightly purge) and
is unchanged.

Closes #61
With three replicas one parked a dead provider while the others kept sending
it traffic, and per-key concurrency caps multiplied by the replica count.

- CircuitBreakerStore: memory (default) or Valkey, one hash per provider with
  state/open_until/failures and a TTL, so an OPEN entry expires even if the
  replica that opened it dies
- OPEN -> HALF_OPEN is a Lua compare-and-set: exactly one replica claims the
  probe, and the claim itself expires after GATEWAY_BREAKER_PROBE_LEASE
- ConcurrencyStore: semaphores (default) or a Valkey sorted set of leased
  permits; releasing pops one member, a killed pod's permits age out
- ChatService/StreamService count availability with isParked() so the probe
  claim is not consumed by the pre-flight count
- GATEWAY_SHARED_STATE_BACKEND=valkey opts in; the chart sets it when its
  Valkey is enabled
- docs/LIMITS.md no longer lists the gap

Tests: two breaker instances over one store park together, exactly one claims
the probe, success closes and failure reopens for both; two limiters over one
store enforce a single cap. The Valkey scripts need a real server and are not
covered by the H2 suite; the memory path is unchanged.

Closes #74
A Java team using Spring AI could not point its ChatModel at the gateway
without hand-writing an HTTP client, which is why the Maven Central artifact
was hard to justify: there was no library-shaped thing to add.

- the build becomes a reactor: path-parent (pom), path (the gateway, still
  io.github.an1ndra:path) and path-spring-ai
- PathChatModel implements ChatModel: messages, options, tool definitions and
  tool responses, non-streaming tool calls and SSE content streaming
- PathEmbeddingModel implements EmbeddingModel over /v1/embeddings
- X-Path-Cache, X-Gateway-Provider and X-Gateway-Cost-Usd land in the
  ChatResponse metadata; errors surface as PathException with status and code
- idempotency(true) sends Idempotency-Key so a caller retry is not charged twice
- the gateway never depends on Spring AI; only the adapter does
- Dockerfile builds -pl path-gateway only, so the image stays the same size
- CI builds and tests both modules; release publishes all three artifacts
- docs/SPRING-AI.md and a README entry

Tests run against a stub HTTP server: request shape, tool round-trip, streaming
assembly, PathException, embeddings.

Closes #62
The compatible Gemini endpoint covers chat and streaming but silently drops
thinkingConfig, safetySettings and cached-content pricing, which is the
opposite of the 422 contract the Anthropic adapter keeps.

- GEMINI and VERTEX provider types with a native generateContent adapter
- thinkingConfig (or reasoning_effort as a shortcut) and safetySettings are
  honoured; a parameter Gemini cannot honour is 422 unsupported_parameter,
  never dropped
- thinking tokens are counted in completion (billable output) and broken out
  as reasoning_tokens, with cachedContentTokenCount priced as cache reads
- a blocked prompt maps to 400 content_blocked naming the category
- Vertex: service-account JSON as the encrypted credential, RS256 assertion
  exchanged for a cached OAuth token, project/location columns (V20)
- tool calls and tool responses round-trip; images are rejected, not ignored
- model sync and the probe handle the native types instead of pretending they
  are OpenAI-shaped

Tests: request translation, 422, response normalisation, the exact cost
arithmetic (thinking priced as output), blocked category, streaming, and a
Vertex OAuth exchange against a stub Google that verifies the RS256 signature
and asserts the token is cached.

Closes #75
GitHub now forces Node 20 actions onto Node 24 and warns on every Helm job:
'azure/setup-helm@1a275c3 ... is being forced to run on Node.js 24'.

v5.1.0 is the same action on node24. The version input is unchanged, so Helm
stays pinned at v3.16.3. Every other action in the workflows already targets
node24 or is composite.
Bumps the reactor to 0.2.3 and adds the release notes: the Spring AI adapter,
native Gemini (AI Studio and Vertex), the daily digest, bucketed stats,
archive slices, Prometheus rules, shared Valkey state, the alerting changes
and the security fixes since v0.2.2.

unreleased.md is reset to the empty template.
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Scorecard details
PackageVersionScoreDetails
actions/azure/setup-helm 2f5e7bc67f1f9972ab16657bb795f36d8f04de0e UnknownUnknown
maven/com.code-intelligence:jazzer-junit 0.30.0 UnknownUnknown
maven/com.h2database:h2 UnknownUnknown
maven/com.knuddels:jtokkit 1.1.0 UnknownUnknown
maven/io.micrometer:micrometer-registry-prometheus UnknownUnknown
maven/io.micrometer:micrometer-tracing-bridge-otel UnknownUnknown
maven/io.opentelemetry:opentelemetry-exporter-otlp UnknownUnknown
maven/org.cyclonedx:cyclonedx-maven-plugin 2.9.3 UnknownUnknown
maven/org.flywaydb:flyway-core UnknownUnknown
maven/org.flywaydb:flyway-database-postgresql UnknownUnknown
maven/org.postgresql:postgresql UnknownUnknown
maven/org.projectlombok:lombok UnknownUnknown
maven/org.springdoc:springdoc-openapi-starter-webmvc-ui 2.8.17 🟢 5
Details
CheckScoreReason
Maintained🟢 1030 commit(s) and 12 issue activity found in the last 90 days -- score normalized to 10
Code-Review⚠️ 2Found 5/25 approved changesets -- score normalized to 2
Binary-Artifacts🟢 10no binaries found in the repo
Security-Policy🟢 10security policy file detected
Dangerous-Workflow🟢 10no dangerous workflow patterns detected
Token-Permissions⚠️ 0detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies⚠️ 0dependency not pinned by hash detected -- score normalized to 0
CII-Best-Practices⚠️ 0no effort to earn an OpenSSF best practices badge detected
Fuzzing⚠️ 0project is not fuzzed
License🟢 10license file detected
SAST⚠️ 0SAST tool is not run on all commits -- score normalized to 0
Packaging🟢 10packaging workflow detected
Branch-Protection⚠️ 0branch protection not enabled on development/release branches
Signed-Releases⚠️ -1no releases found
maven/org.springframework.boot:spring-boot-maven-plugin UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-actuator UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-data-jpa UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-data-redis UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-mail UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-test UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-validation UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-web UnknownUnknown
maven/com.fasterxml.jackson.core:jackson-databind UnknownUnknown
maven/io.projectreactor:reactor-core UnknownUnknown
maven/io.projectreactor:reactor-test UnknownUnknown
maven/org.springframework.ai:spring-ai-bom UnknownUnknown
maven/org.springframework.ai:spring-ai-client-chat UnknownUnknown
maven/org.springframework.ai:spring-ai-model UnknownUnknown
maven/org.springframework.boot:spring-boot-starter-test UnknownUnknown

Scanned Files

  • .github/workflows/ci.yml
  • path-gateway/pom.xml
  • path-spring-ai/pom.xml
  • pom.xml

@an1ndra
an1ndra merged commit 5044096 into main Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant