Working notes for agents and contributors. Read this before changing code; the gotchas at the bottom each cost real debugging time.
A provider-agnostic LLM gateway. One HTTP surface in front of OpenAI, Anthropic, Groq and any OpenAI-compatible endpoint, with model routing, automatic fallback, per-key credits, rate limits and cache-aware token metering.
The client-facing contract is the OpenAI chat-completions shape because that is what SDKs and tools already implement. That is a compatibility decision, not a vendor dependency: the gateway translates to each provider's native protocol.
# unit + integration tests (85)
./mvnw verify
# full end-to-end suite against a mock upstream (no real keys)
python3 tools/mock-upstream.py 9099 &
GATEWAY_ADMIN_TOKEN=test-admin java -jar path-gateway/target/path-*.jar &
python3 tools/smoke_test.py
# run locally
export OPENAI_API_KEY=... ANTHROPIC_API_KEY=... GATEWAY_ADMIN_TOKEN=...
./mvnw -pl path-gateway spring-boot:runThe seeder prints a demo sk-gw-... key on the first boot only.
./mvnw is a Maven reactor, not a single project:
| Module | Artifact | Contents |
|---|---|---|
path-gateway/ |
io.github.an1ndra:path |
the gateway jar, the runnable Spring Boot application |
path-spring-ai/ |
io.github.an1ndra:path-spring-ai |
a Spring AI ChatModel/EmbeddingModel over the gateway |
The gateway is the published root artifact and never depends on Spring AI; the adapter is
the only module that knows about it. ./mvnw verify builds both. The Dockerfile builds only
-pl path-gateway, so the image does not carry the adapter's dependencies. tools/ and
deploy/ stay at the repository root.
com.anindra.path
├── api/ ChatCompletionsController (public surface), errors, Durations
├── admin/ AdminController + services (providers, routes, keys, sync, stats)
├── config/ GatewayProperties, AuthFilter, CORS
├── domain/ JPA entities: Provider, ModelRoute, ClientKey, UsageRecord
├── gateway/ the engine: adapters, routing, streaming, metering
│ ├── ProviderAdapter wire-format contract
│ ├── OpenAiCompatibleAdapter OpenAI + Groq + vLLM + Together + ...
│ ├── AnthropicAdapter Messages API, tools, cache tokens
│ ├── ChatService non-streaming path with fallback
│ ├── StreamService SSE path; owns commit semantics
│ ├── ProviderCircuitBreaker per-provider OPEN/HALF_OPEN/CLOSED, Retry-After
│ ├── IdempotencyService Idempotency-Key claims and replay
│ ├── ConcurrencyLimiter per-key in-flight permits
│ ├── RateLimitStore in-memory or Valkey RPM limiter
│ ├── ResponseCacheService exact-match replay, opt-in per route
│ ├── DrainState shutdown drain + readiness
│ └── (config) RequestIdFilter X-Request-Id + MDC correlation
│ ├── RouteResolver public model id → ordered chain (aliases, weights)
│ ├── ModelAliasService cheap/fast/long-context/vision
│ ├── EmbeddingsService /v1/embeddings for OPENAI-type providers
│ └── (admin) AuditService, AlertService, PromptLogService, TenancyService
│ ├── CreditGuard pre-flight balance/worst-case check
│ ├── UsageService cost breakdown, aggregates, metering
│ └── TokenUsage normalised token accounting
├── repo/ Spring Data repositories
├── secret/ AES-GCM for stored provider credentials
└── seed/ first-boot bootstrap
AuthFilterresolvesAuthorization: Bearer sk-gw-...→ClientKey, applies the RPM limiter, or checksX-Admin-Tokenfor/admin/**.ChatRequestSpec.parsevalidates the body and keeps the original JSON for passthrough.RouteResolverturns the public model id (plusmodels[]andprovider.order) into an ordered chain of(route, provider).CreditGuardrefuses the call if the balance cannot cover a worst-case estimate.ChatService/StreamServicewalk the chain: one attempt per route, adapter translates,UpstreamClientsends raw HTTP.UsageServiceprices the call, writes aUsageRecord, charges the key.
- No AI SDKs. The gateway speaks HTTP + SSE with Jackson. SDKs own the request model and drop unknown fields, which breaks passthrough; they also add retries that corrupt fallback accounting. See docs/LIMITS.md.
promptTokensmeans total prompt, including cache reads and cache writes:billableInput = prompt - cached - cacheWrite. OpenAI reports cache as a subset; the Anthropic adapter sums it. Keep this invariant when adding providers.- Cost is 8 decimals, each component rounded before summing. Do not "fix" the
smoke test's
1e-8tolerance by widening it further. - Admin JSON is snake_case, with camelCase accepted via
@JsonAliason the input records. Responses are always snake_case maps. - Errors use the OpenAI envelope:
{"error":{"message","type","code","param"}}. - Never log a credential. Admin views show
credential_hint(last 4) only. - Streaming commits on the first byte. Fallback is possible only before that;
afterwards the client gets an SSE error object and
[DONE].
| Task | Where |
|---|---|
| New provider (OpenAI-shaped or Anthropic-shaped) | No code: POST /admin/providers + routes, or GATEWAY_UPSTREAM_<SLUG>_KEY |
| New wire format | Implement ProviderAdapter + StreamTranslator, add tests |
| New public endpoint | api/, reuse ApiException + the error envelope |
| New metered field | TokenUsage → both adapters → UsageRecord → UsageService → /v1/usage and /admin/usage |
| Admin action | AdminService + a thin AdminController method |
src/main/resources/db/migration/ is the only source of truth. Hibernate validates on
startup; a mismatch fails the boot. Tests run the same migrations against in-memory H2.
README.md is the landing page. Details live in:
| File | Contents |
|---|---|
docs/API.md |
endpoints, errors, idempotency, cache, embeddings |
docs/ARCHITECTURE.md |
request flow, routing, metering fields |
docs/CONFIGURATION.md |
every environment variable and profile |
docs/OPERATIONS.md |
admin API, metrics, migrations, alerts |
docs/LIMITS.md |
design notes and known gaps |
docs/TESTING.md |
test suites and the mock upstream |
docs/ROADMAP.md |
delivered work and the open stories |
changelog/ |
one release-notes file per tag, published as the release body |
Keep README short: a feature can be one bullet there and a section in the matching doc.
GatewayApiTest— boots the full app on in-memory H2: auth, catalogue, credits, admin token, snake_case inputs.AnthropicAdapterTest— request translation (system, images, tools, tool results), response translation, stream events, cache tokens.UsageCostTest— cost breakdown, cache pricing, missing prices, usage parsing.ProviderCircuitBreakerTest— threshold, half-open probe, rate-limit parking.IdempotencyServiceTest— replay, conflicts, concurrent claims, key scoping.ConcurrencyLimiterTest— per-key permits, no over-release.TokenEstimateTest— real tokenization with a heuristic fallback.ResponseCacheEligibilityTest/ResponseCacheServiceTest— cache rules, scoping, expiry.CreditGuardTest— budget windows and max_tokens clamping.DrainStateTest— in-flight accounting and refusal while draining.tools/smoke_test.py— 104 checks againsttools/mock-upstream.py. The mock can simulate failures by model id:boom(500),ratelimit(429),unauthorized(401),badrequest(400),slow,empty.
Any behaviour change needs a unit test and, if it touches routing/streaming/billing, a smoke check. Never claim a fix without running both.
Developis the default branch; direct pushes are allowed.mainis release-only: PR required,Build and unit tests+End-to-end smoke testmust pass, no bypassing (admins included), no force-push, no deletion.- Maven Central coordinates are
io.github.an1ndra:path(GitHub namespace); the Java package stayscom.anindra.path. Publishing lives behind-P release(plain jar + sources + javadoc, signed); seedocs/PUBLISHING.md. Never rundeployagainst a used version — Central is immutable. After a-P releasebuild the jar intarget/is the plain library, so rebuild before running the smoke suite. - CI pins
ubuntu-26.04(latest LTS) instead ofubuntu-latest; bump it deliberately. - The Dockerfile fetches Maven itself, not through
./mvnw. The wrapper shells out to curl/unzip, whicheclipse-temurindoes not fully ship; inside the image it downloaded a file that failed its own SHA-256 check and broke every container build. The build stage installs those tools and verifies the pin from.mvn/wrapper/maven-wrapper.propertiesexplicitly. Do not "simplify" it back to./mvnwwithout building the image. .trivyignore.yamlmust be passed to Trivy explicitly. The YAML form (the one that supportsexpired_at) is not auto-detected; the release workflow setsignorefile. Every entry carries astatementand a deadline, so an accepted CVE fails the gate again on that date instead of living forever.- Every action is pinned to a commit SHA, with the tag in a trailing comment
(
actions/checkout@3d3c42e5... # v7.0.1). A mutable tag can be repointed at any commit, which is exactly the supply-chain hole this closes. Bump by merging the Dependabotgithub-actionsPR, orpinact runlocally; never replace a SHA with@vNagain. - Release: copy
changelog/unreleased.mdtochangelog/vX.Y.Z.md, commit, then push av*.*.*tag → jar + SBOM + checksums + multi-archghcr.io/an1ndra/path. The workflow publisheschangelog/<tag>.mdverbatim as the release body and fails if the file is missing or empty (tools/changelog.py). - The image is scanned before it is published. The workflow pushes only
ghcr.io/<repo>:sha-<commit>; Trivy gates that tag; only then doesdocker buildx imagetools createcopy the same digest to<tag>andlatest. A failed scan therefore leaves asha-tag and an unused version number, not a released image. The digest in the workflow summary is the one that was scanned. Do not move the version tag back into the build step.
Every open story carries exactly one area/ label and one effort/ label, so the
queue is filterable by label:area/routing or by size.
| Label | Meaning |
|---|---|
area/routing |
model selection, fallback chains, aliases |
area/billing |
credits, budgets, idempotent charging |
area/metering |
usage records, reporting, token counting |
area/providers |
wire formats, adapters, provider support |
area/security |
PII, guardrails, auth, secrets |
area/ops |
deployment, scaling, cluster behaviour |
area/api |
public and admin API surface |
effort/s |
roughly a day or less |
effort/m |
roughly two to five days |
effort/l |
a week or more, or needs a design decision |
blocked |
waiting on something outside this repository |
A story may carry two area/ labels when it genuinely spans two (for example a
billing change that also adds an endpoint). Never three, and never none. effort/*
is a guess, not a commitment — re-tag when reality disagrees.
-
IntelliJ: point Maven home at a real install, not "Use Maven wrapper". The wrapper here is
distributionType=only-script(nomaven-wrapper.jar), and IntelliJ's Maven server resolves its own home instead of runningmvnw. It then bootstraps Maven without the plexus artifact bindings and fails withGuice/MissingImplementation: No implementation for ArtifactHandlerManager was bound. Set Settings -> Build Tools -> Maven -> Maven home to the wrapper's extracted 3.10.1 (~/.m2/wrapper/dists/apache-maven-3.10.1/<hash>) or to the IDE's bundled Maven. The CLI./mvnwis unaffected. -
Flyway owns the schema;
ddl-autoisvalidate. AddV<n>__name.sqlfor every entity change and never edit an applied migration.V1__baseline.sqlis written in portable SQL:bigint generated by default as identity,timestamp(6) with time zone, enums asvarchar+ check constraint (native enum types would not port between H2 and Postgres). Existing pre-Flyway databases baseline at V1 automatically. -
The circuit breaker is in-memory per node. Like the rate limiter, state is not shared between instances; a restart clears it. That is deliberate for v1.
-
Idempotency claims are written before the upstream call. If the process dies mid-request the row stays PENDING until
GATEWAY_IDEMPOTENCY_TTL, so retries get 409idempotency_in_progress. The purge job only removes expired rows. -
Spend windows use stored usage, not a counter.
daily_limit_usdandmonthly_limit_usdcompare againstsum(cost_usd)since UTC midnight / month start, so they survive restarts and stay correct across instances. -
Shutdown drains for
GATEWAY_DRAIN_TIMEOUT.DrainStateflips readiness and refuses new chat requests while streams finish; keep the value below your orchestrator's termination grace period. -
System-only conversations are rejected. Anthropic needs a non-empty message list;
ChatRequestSpec.parsereturns 400 instead of forwarding an empty array. The property test found this; keep the guard. -
tools/load-test.jsis a k6 script, not part of CI (it needs a running gateway and a real key). -
allowed_modelsis enforced inCreditGuard, before any provider call. Globs end with*; blank means every model. Aliases are checked by their alias name, so add the alias to the allowlist too if a key should usecheap. -
Prompt logging is a PII store. Off by default; when enabled it redacts provider keys and bearer tokens, but it still stores user prompts. Retention is enforced by a scheduled purge.
-
Audit and alert writes must never break the operation. Both swallow their own failures and log a warning.
-
Aliases are computed from the route table on every resolve, so editing a price can change what
cheapmeans. There is no alias table to drift out of sync. -
Canary weights must stay deterministic per key.
RouteResolver.weightedhasheskeyIdand the model; changing that hash reshuffles every client's assignment. -
Tracing is off by default (
GATEWAY_TRACE_SAMPLE=0). Raising the probability without settingOTEL_EXPORTER_OTLP_ENDPOINTproduces export warnings, not traces. -
The
prodprofile switches logs to ECS JSON.requestIdcomes fromRequestIdFilter's MDC entry;traceId/spanIdare added by the tracing bridge. -
RPM limiting defaults to memory; set
GATEWAY_RATE_LIMIT_BACKEND=valkeyin a cluster. Budget reservations are database-backed either way, so money is safe without Valkey. -
Admin tokens are scoped. The bootstrap env token is read+write; DB tokens are read-only unless
scopescontainswrite. -
Weighting is opt-in. A chain of default-weight (100) routes keeps priority order; canary shuffling only happens once a weight differs from 100. Otherwise fallback would become a lottery.
-
Rate limiting is in-memory by default, one node. Running two instances doubles the effective RPM limit.
-
H2 rejects
AUTO_SERVER=TRUEtogether withDB_CLOSE_ON_EXIT. Don't re-add it. -
Admin credentials are resolved DB → config → env. A key stored via the admin API silently wins over
OPENAI_API_KEY. -
Anthropic rejects parameters it has no equivalent for with 422
unsupported_parameter(seed,logit_bias, penalties,logprobs,n > 1, non-textresponse_format). Do not "fix" a failing call by dropping the field; the divergence is what the 422 exists to prevent.usermaps tometadata.user_id. -
The response cache only serves explicit
temperature: 0, tool-free requests on a route withcache_enabled. Entries are keyed per client key so tenants cannot read each other's completions. Do not relax the temperature rule: a cached sample at temperature 1 is a correctness bug. -
Token estimates use jtokkit when the upstream model has a known encoding, and fall back to
chars / 4. The estimate only pre-authorises spend; billing always uses the provider-reported usage. -
usage_estimated=truemeans the provider returned nousageblock and the counts were inferred from text length. Never present those as observed. -
The demo key is unlimited and is created when the key table is empty. Set
GATEWAY_BOOTSTRAP_DEMO_KEY=falseanywhere real. -
tools/smoke_test.pyneeds a clean DB for the demo-key path; otherwise it creates its own key through the admin API.
main requires the CI, CodeQL and Security checks. .gitleaks.toml allowlists the
placeholder credentials in tests and docs — do not broaden it to hide a real key.
Release artifacts carry checksums.txt and the wrapper pins the Maven distribution
SHA-256.
Never commit .env, data/, *.mvdb, master keys, or provider credentials. Stored
credentials are AES-GCM encrypted; data/master.key decrypts them and must be backed
up, never shared.