diff --git a/ICONS.md b/ICONS.md index 091f7eb..26f5421 100644 --- a/ICONS.md +++ b/ICONS.md @@ -27,7 +27,7 @@ to storefront visitors — nominative use — and remains the property of its respective owner. Inclusion here is not an endorsement by those projects, and these files are **not** covered by this repository's `LICENSE`. See `NOTICE`. -All 13 items carry an icon; none currently ship screenshots. +All 14 items carry an icon; none currently ship screenshots. | Item | Source | Method | License note | |---|---|---|---| @@ -36,6 +36,7 @@ All 13 items carry an icon; none currently ship screenshots. | `label-studio` | | GitHub org avatar | Label Studio maintainer (HumanSignal) GitHub org avatar; nominative use. | | `langflow` | | GitHub org avatar | Langflow mark via the langflow-ai GitHub org avatar; nominative use. | | `litellm` | | GitHub org avatar | LiteLLM maintainer (BerriAI) GitHub org avatar; nominative use. | +| `litellm-stack` | | GitHub org avatar | Same BerriAI mark as `litellm`, copied byte-for-byte rather than re-fetched, since the item lists the same upstream software; nominative use. | | `meilisearch` | | GitHub org avatar | Meilisearch mark via the meilisearch GitHub org avatar; nominative use. | | `mlflow` | | GitHub org avatar | MLflow mark via the mlflow GitHub org avatar; nominative use. | | `n8n` | | GitHub org avatar | n8n mark via the n8n-io GitHub org avatar; nominative use. | diff --git a/items/litellm-stack/blueprint.yaml b/items/litellm-stack/blueprint.yaml new file mode 100644 index 0000000..e5c701c --- /dev/null +++ b/items/litellm-stack/blueprint.yaml @@ -0,0 +1,39 @@ +specVersion: v1 +kind: BLUEPRINT +metadata: + slug: litellm-stack + version: 1 +spec: + components: + # Producers first: map order is graph order, and reading the two nodes the + # wires come from before the node they arrive at is what makes the + # connections below resolvable on a single pass down the file. + db: + component: ./components/postgres.yaml + size: general.standard.small + connections: {} + cache: + component: ./components/redis.yaml + size: general.standard.small + connections: {} + web: + component: ./components/litellm.yaml + # The single-container `litellm` item runs on general.economy.nano because + # a stateless proxy is mostly idle sockets. This one is not that: with + # DATABASE_URL set, the first boot runs the Prisma schema setup, and every + # request afterwards carries a key lookup and a spend write. + size: general.standard.small + connections: + databaseUrl: + fromRole: db + fromOutput: dsn + redisUrl: + fromRole: cache + fromOutput: url + # Derived rather than authored. The merged USER inputs are exactly the install + # form this item wants — the database user and name, both defaulted, beside + # three secrets the platform mints — and an authored block would have to + # restate all five to add nothing. Neither wired input appears there, which is + # the point: a CONNECTION input is satisfied by its wire and never reaches the + # form. What is left is a form with no required field on it. + parameters: {} diff --git a/items/litellm-stack/components/litellm.yaml b/items/litellm-stack/components/litellm.yaml new file mode 100644 index 0000000..c423b67 --- /dev/null +++ b/items/litellm-stack/components/litellm.yaml @@ -0,0 +1,177 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + # Same pin as the single-container `litellm` item, and it is the tip of + # LiteLLM's stable track rather than merely a recent tag: the project + # publishes `vX.Y.Z-stable` releases beside its ordinary ones, and + # v1.83.14-stable.patch.3 is the highest of them. + ref: ghcr.io/berriai/litellm:v1.83.14-stable.patch.3 + # No `command`. The image's ENTRYPOINT is `docker/prod_entrypoint.sh`, whose + # last line is `exec litellm "$@"`, and a Musher `command` overrides the + # image's CMD while inheriting its ENTRYPOINT — so every word written here + # arrives as a `litellm` CLI argument. The stock CMD (`--port 4000`) is + # already the one this component wants. See the `redisUrl` input below for + # what that argument-only surface costs. + endpoints: + primary: + containerPort: 4000 + protocol: HTTP + visibility: PUBLIC + health: + readiness: + # /health/liveliness (LiteLLM's spelling) stays the gate even though this + # deployment now has the database that would make /health/readiness + # meaningful. That route is declared with + # `dependencies=[Depends(user_api_key_auth)]`, so once LITELLM_MASTER_KEY + # is set it answers 401 to a probe carrying no key, and the deployment + # would never come up. /health/liveliness carries no such dependency. + path: /health/liveliness + endpoint: primary + # Longer than the stateless item's 30s. With DATABASE_URL set, the CLI + # runs the Prisma schema setup *before* uvicorn binds the port + # (proxy_cli.run_server), so on a first deploy nothing answers on 4000 + # until the migration chain has finished against an empty database. + initialDelaySeconds: 120 + timeoutSeconds: 5 + successThreshold: 1 + failureThreshold: 10 + envVars: + # Lets the proxy keep model definitions, and the config rows below, in the + # Postgres this blueprint deploys rather than in a config file this + # container has no way to receive. It is also what makes the Admin UI's + # "Add model" flow work at all. + - key: STORE_MODEL_IN_DB + value: + type: LITERAL + value: 'True' + # LITELLM_MODE defaults to "DEV", and "DEV" is the one value that makes + # the CLI call load_dotenv() on startup — reading a stray .env out of the + # working directory. Setting it leaves the platform as the only thing that + # can put settings into this container. + - key: LITELLM_MODE + value: + type: LITERAL + value: PRODUCTION + # DISABLE_SCHEMA_UPDATE is deliberately NOT set. Upstream sets it on proxy + # replicas because their Helm chart runs migrations in a separate job; + # this blueprint has no such job, so the proxy applying its own schema at + # startup is the only thing that creates the tables the database exists + # for. + contract: + inputs: + masterKey: + schema: + type: STRING + isSensitive: true + isRequired: true + suppliedBy: USER + ui: + label: Master key + target: + envVarKey: LITELLM_MASTER_KEY + generator: + byteLength: 32 + encoding: ALPHANUMERIC + description: >- + Auto-generated proxy admin credential. It authenticates the admin API, + mints virtual keys, and is the Admin UI password. Copy it from the + deployment's Configuration tab. + # Distinct from the master key and far less forgiving. LITELLM_SALT_KEY + # encrypts the provider credentials the proxy stores in Postgres, so + # changing it after any credential has been saved leaves every stored + # credential undecryptable — upstream's own guidance is "set once, never + # change it". It is generated rather than typed for the same reason the + # master key is, and it exists in this item and not in the single-container + # `litellm` item because there is no encrypted-at-rest credential without a + # database to hold it. + saltKey: + schema: + type: STRING + isSensitive: true + isRequired: true + suppliedBy: USER + ui: + label: Salt key + target: + envVarKey: LITELLM_SALT_KEY + generator: + byteLength: 32 + encoding: ALPHANUMERIC + description: >- + Auto-generated key encrypting provider API keys at rest in the + database. Rotating it makes every credential already stored + unreadable, so treat it as fixed for the life of the deployment. + # The wire that turns a stateless proxy into a re-billable one. Virtual + # keys, teams, budgets and the spend ledger are all rows; without this + # input none of the tables exist and the proxy is a router with a single + # shared credential. + # + # DATABASE_URL is the whole of the database configuration this container + # needs — the CLI reads it directly, appends its connection-pool query + # params, and hands it to Prisma. (LiteLLM will also assemble one from + # DATABASE_HOST/PORT/USER/PASSWORD/NAME, which is the same value spelled + # in five inputs instead of one wire.) + databaseUrl: + schema: + type: STRING + format: CONNECTION_STRING + semanticType: POSTGRES + isSensitive: true + isRequired: true + suppliedBy: CONNECTION + ui: null + target: + envVarKey: DATABASE_URL + description: >- + Postgres connection string the proxy stores virtual keys, teams, + budgets and spend in. Wired from the database node in this blueprint. + # WHAT THIS WIRE DOES, AND THE ONE THING IT DOES NOT. + # + # It does deliver the address. REDIS_URL is read by + # `_redis_kwargs_from_environment()`, which maps every `REDIS_` + # variable onto a redis-py argument, and RedisCache drops a null host, + # port and password before calling `get_redis_client()` — so this URL + # alone is a complete connection, with no REDIS_HOST/REDIS_PORT beside it. + # + # It does not, on its own, switch Redis on. `redis_usage_cache` — the + # client the proxy shares tpm/rpm counters, spend and router state through + # — is assigned in exactly one place, `ProxyConfig._init_cache`, reached + # only from the `litellm_settings.cache is True` branch of `load_config`. + # No environment variable reaches that flag — not in this release, and not + # in the current one, which was read to check whether the gap had closed. + # + # And a config file cannot be shipped here: as the `command` note above + # records, everything this component can pass the container is a `litellm` + # CLI argument, so there is nowhere to write a `config.yaml` for `--config` + # to name. + # + # What closes the gap is the other wire. With STORE_MODEL_IN_DB set, + # `ProxyConfig.save_config` writes `litellm_settings` to the `config` table + # in Postgres and `_update_config_from_db` merges it back at every start, + # so a one-time POST of `{"litellm_settings": {"cache": true}}` to + # /config/update, authenticated with the master key, is durable and + # applies to every replica. `cache_params` is left empty on purpose: an + # empty block is what sends LiteLLM to the environment, and the + # environment is where this wire has already put the URL. + redisUrl: + schema: + type: STRING + format: CONNECTION_STRING + semanticType: REDIS + isRequired: true + suppliedBy: CONNECTION + ui: null + target: + envVarKey: REDIS_URL + description: >- + Redis connection URL backing the shared response cache, cross-replica + rate-limit counters and router state. Wired from the cache node in + this blueprint; enable caching once from the Admin UI or + /config/update and the setting persists in the database. + outputs: {} diff --git a/items/litellm-stack/components/postgres.yaml b/items/litellm-stack/components/postgres.yaml new file mode 100644 index 0000000..5846fec --- /dev/null +++ b/items/litellm-stack/components/postgres.yaml @@ -0,0 +1,147 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: postgres:17.10-alpine + endpoints: + primary: + containerPort: 5432 + protocol: TCP + visibility: PRIVATE + envVars: + - key: PGDATA + value: + type: LITERAL + value: /var/lib/postgresql/data/pgdata + volumes: + data: + sizeGib: 10 + mountPath: /var/lib/postgresql/data + contract: + inputs: + # Generated rather than typed, and the `dsn` output below is the reason. + # A `${{ params. }}` token is substituted verbatim — the platform + # percent-encodes nothing — so a password carrying `@`, `/`, `:`, `#` or + # `?` composes a URI that parses wrong or not at all, and the failure + # surfaces inside Prisma minutes into a deploy rather than on the form + # that accepted it. ALPHANUMERIC draws from base62, which contains no + # character reserved in a URI, so the value is safe in the userinfo + # position by construction rather than by a rule the user has to obey. + # + # Nothing outside the mesh connects to this database, so there is no + # password for a person to choose: the proxy receives it over a wire, and + # the deployment's Configuration tab is where to read it for a manual + # `psql`. That leaves this item's install form asking for nothing at all. + postgresPassword: + schema: + type: STRING + isSensitive: true + isRequired: true + suppliedBy: USER + ui: + label: Database password + target: + envVarKey: POSTGRES_PASSWORD + generator: + byteLength: 32 + encoding: ALPHANUMERIC + description: >- + Auto-generated password for this database's user. It reaches the proxy + inside the connection string below rather than by being typed twice. + Copy it from the deployment's Configuration tab if you want to connect + to the database by hand. + # Typed, and interpolated into `dsn` just as the password is — so the same + # percent-encoding hazard applies, with the difference that this one has a + # person on the other end of it. The pattern is the intersection that + # costs nothing: every character in it is URL-safe, and it is also exactly + # what Postgres accepts as an unquoted identifier, so it rejects no name a + # role could legally have. + postgresUser: + schema: + type: STRING + default: postgres + pattern: '^[A-Za-z_][A-Za-z0-9_]{0,62}$' + isRequired: false + suppliedBy: USER + ui: + label: Database user + target: + envVarKey: POSTGRES_USER + postgresDb: + schema: + type: STRING + default: app + pattern: '^[A-Za-z_][A-Za-z0-9_]{0,62}$' + isRequired: false + suppliedBy: USER + ui: + label: Database name + target: + envVarKey: POSTGRES_DB + outputs: + address: + schema: + type: STRING + semanticType: POSTGRES + description: >- + Mesh-internal `host:port` of the database endpoint (for example + `db-myapp.internal:5432`), resolvable only by workloads in the same + deployment. Wire it into a consuming component's CONNECTION input to + point that workload at this database. + valueFrom: DERIVED + value: null + # The whole URI rather than the address, for a consumer that takes one + # DSN and no separate credential settings. LiteLLM is that consumer: + # `DATABASE_URL` is the only database setting its container reads without + # a config file, and Prisma parses it as a URI. + # + # DECLARED, and DECLARED does not mean static. The value below is a + # template the platform interpolates once the node is placed: + # `${{ params. }}` against this node's own resolved inputs, + # `${{ self.privateAddress. }}` against its own addressing. So + # the host and port that are unknowable at authoring time still arrive + # here — writing them is not the same as knowing them now. + # + # DERIVED is the narrower thing, and the reason `address` above is one and + # this is not: the platform assigns every DERIVED output the same value, + # the node's public URL if it has one and its private `host:port` + # otherwise. Neither is a connection string. `format` and `semanticType` + # do not close the gap — they are wiring-compatibility tags the publish + # gate reads to check a connection's two ends agree, and they synthesise + # nothing. An earlier revision of this file had that backwards, and the + # deployment it produced handed LiteLLM a `DATABASE_URL` of + # `db-litellm-stack.internal:5432`, which Prisma rejects with P1012 + # before the proxy binds a port at all. + # + # Every token resolves from this node and nothing else — its own USER + # inputs, its own placement — which is what keeps the output legal: + # component §6.2 forbids an output that depends on a value the component + # received over an *inbound connection*, and none of these arrive that + # way. Blueprint §4.2's own worked example wires a Postgres node's + # `connectionString`, so a composed DSN is the shape the contract was + # written for. + # + # Sensitive because it embeds POSTGRES_PASSWORD. `address` is not, and + # that difference is the reason to keep both: a consumer that only needs + # to reach the server should take the address and not a copy of the + # password. + dsn: + schema: + type: STRING + format: CONNECTION_STRING + semanticType: POSTGRES + isSensitive: true + description: >- + Ready-to-use `postgresql://` URI for this database — the mesh-internal + host and port above, carrying the user, password and database name + this node was deployed with. Wire it into a consuming component's + CONNECTION input where that workload wants one connection string + rather than the parts. + valueFrom: DECLARED + value: >- + postgresql://${{ params.postgresUser }}:${{ params.postgresPassword }}@${{ self.privateAddress.primary }}/${{ params.postgresDb }} diff --git a/items/litellm-stack/components/redis.yaml b/items/litellm-stack/components/redis.yaml new file mode 100644 index 0000000..5e32e6a --- /dev/null +++ b/items/litellm-stack/components/redis.yaml @@ -0,0 +1,77 @@ +specVersion: v1 +kind: COMPONENT +metadata: + version: 1 +spec: + workload: + kind: SERVICE + source: + type: IMAGE + ref: redis:8.8.1-alpine + command: redis-server --appendonly yes + endpoints: + primary: + containerPort: 6379 + protocol: TCP + visibility: PRIVATE + volumes: + data: + sizeGib: 5 + mountPath: /data + contract: + inputs: {} + outputs: + address: + schema: + type: STRING + semanticType: REDIS + description: >- + Mesh-internal `host:port` of the cache endpoint (for example + `cache-myapp.internal:6379`), resolvable only by workloads in the same + deployment. Wire it into a consuming component's CONNECTION input to + point that workload at this cache. + valueFrom: DERIVED + value: null + # The `redis://` form of the address above, for a consumer that takes one + # URL rather than a host and a port. LiteLLM is that consumer: its Redis + # settings are read out of the environment by `_redis_kwargs_from_environment`, + # which maps every `REDIS_` variable onto a redis-py argument, and + # `url` is the one argument that carries host and port together. + # + # DECLARED for the reason postgres.yaml's `dsn` sets out at length: a + # DECLARED value is a template the platform interpolates after placement, + # while a DERIVED one is assigned the node's bare address and nothing + # else. A scheme is not part of an address, so `redis://` has to be + # written here — there is nowhere else it can come from. + # + # No credentials, and that is a property of this component rather than an + # omission: the server runs with no `requirepass` on a PRIVATE endpoint, + # so there is no password for the URI to carry. A future component that + # sets one would have to mark this output sensitive — a URI may embed the + # password inline, which is why LiteLLM's own settings API redacts a + # Redis `url` alongside `password`. It is also why this output takes no + # `${{ params.* }}` token and postgres.yaml's `dsn` takes three. + # + # ONE CAVEAT THE TEMPLATE CANNOT FIX. A SERVICE that declares no + # ENDPOINT_URL output has a `url` one appended at publish, and the check + # that suppresses it matches on the key *and* the ENDPOINT_URL format — so + # this CONNECTION_STRING output does not suppress it, and the appended + # entry lands after this one and takes the key. Until + # musher-dev/platform#2589 closes that, REDIS_URL reaches the proxy as the + # bare address whatever is written here. It is inert either way: LiteLLM + # reads REDIS_URL only once caching is switched on, which is a deliberate + # act described on the `redisUrl` input in litellm.yaml. DATABASE_URL is + # unaffected — `dsn` collides with no auto-derived key. + url: + schema: + type: STRING + format: CONNECTION_STRING + semanticType: REDIS + description: >- + Ready-to-use `redis://` URI for this cache — the mesh-internal host + and port above, with no credentials, since this deployment runs Redis + unauthenticated on a private endpoint. Wire it into a consuming + component's CONNECTION input where that workload wants one URL rather + than a host and a port. + valueFrom: DECLARED + value: redis://${{ self.privateAddress.primary }} diff --git a/items/litellm-stack/listing.yaml b/items/litellm-stack/listing.yaml new file mode 100644 index 0000000..2c41353 --- /dev/null +++ b/items/litellm-stack/listing.yaml @@ -0,0 +1,71 @@ +specVersion: v1 +kind: LISTING +metadata: + slug: litellm-stack + version: 1 +spec: + listingKind: BLUEPRINT + displayName: LiteLLM Stack + summary: LiteLLM with its own Postgres and Redis — virtual keys, teams, and per-client spend that outlive a restart + description: | + ## What it is + The LiteLLM proxy deployed the way it is meant to run in production: + with a database behind it and a shared cache beside it. LiteLLM fronts + 100+ LLM providers (OpenAI, Anthropic, Bedrock, Vertex, Mistral, + Azure, local Ollama) behind one OpenAI-compatible endpoint. Postgres + is what turns that into an accountable service — virtual API keys, + teams, budgets, rate limits and a per-key spend ledger are rows, and + without a database none of them exist. Redis is the shared state a + second replica needs: response cache, cross-replica rate-limit + counters, and router health. + + ## When to use + - You are re-billing model access to clients or teams and need + per-key budgets and a spend ledger you can invoice from. + - You want to issue and revoke API keys per customer without + handing out a provider key or redeploying. + - You are running more than one proxy replica and need rate limits + and cached responses to be shared rather than per-instance. + - You want provider fallbacks and load balancing in front of your + model traffic, with the routing state kept outside the container. + + ## Defaults + Three nodes. `db` runs `postgres:17.10-alpine` on a private endpoint + with a 10 GiB volume; `cache` runs `redis:8.8.1-alpine` with + append-only persistence on a 5 GiB volume; `web` runs + `ghcr.io/berriai/litellm:v1.83.14-stable.patch.3` on port 4000 behind + the platform's HTTPS edge. The database connection string and the + Redis URL are wired between the nodes, so neither is typed at + install. The three secrets the stack runs on — the database password, + the master key, and the credential-encryption salt key — are + auto-generated at deploy; copy them from the deployment's + Configuration tab. Nothing on the install form is required. LiteLLM + applies its own schema to the empty database on first boot, which is + why the first deploy takes a few minutes longer than a restart. + + Configure provider upstreams from the dashboard; model definitions + are stored in the database rather than a config file, so they survive + a re-deploy. Response caching is the one thing that is not on by + default: LiteLLM has no environment variable for the switch, so turn + it on once — from the Admin UI, or with a single POST to + `/config/update`. Either way the setting is written to the same + Postgres and applies on every start, to every replica, using the + Redis URL already in the environment. + + Rotating the salt key after any provider credential has been saved + makes those credentials unreadable. Treat it as fixed. + category: AI_ML + lifecycleStage: BETA + tags: + - ai + - llm + - proxy + - gateway + - openai-compatible + - postgres + - redis + homepageUrl: https://www.litellm.ai + sourceRepoUrl: https://github.com/BerriAI/litellm + supportUrl: https://docs.litellm.ai + license: MIT + icon: media/icon.png diff --git a/items/litellm-stack/media/icon.png b/items/litellm-stack/media/icon.png new file mode 100644 index 0000000..4a54e89 Binary files /dev/null and b/items/litellm-stack/media/icon.png differ