From 29ed5f3b1f788cc61c52cff594ce0a756881aef3 Mon Sep 17 00:00:00 2001 From: khanhphamj Date: Mon, 9 Nov 2026 06:48:31 +0700 Subject: [PATCH 1/4] Docs re-read: correct skills where the AgentBase docs differ (inbound auth header stripping, rate-limit fallback, memory limit, MCP, vCR) Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 12 ++++++++++++ skills/agentbase-build-auth/SKILL.md | 7 ++++--- skills/agentbase-build-auth/references/patterns.md | 1 + skills/agentbase-build-deploy/SKILL.md | 12 ++++++------ skills/agentbase-build-frontend/SKILL.md | 2 +- skills/agentbase-build-llm/SKILL.md | 8 ++++---- skills/agentbase-build-mcp/SKILL.md | 8 ++++---- skills/agentbase-build-mcp/references/governance.md | 6 ++++-- skills/agentbase-build-memory/SKILL.md | 6 +++--- .../agentbase-build-memory/references/compression.md | 2 +- skills/agentbase-build-web/SKILL.md | 2 +- skills/agentbase-build/references/api-contract.md | 2 +- skills/agentbase-build/references/architecture.md | 1 + skills/agentbase-build/references/checklist.md | 1 + skills/agentbase-build/references/decision-guide.md | 3 ++- .../agentbase-build/references/platform-coverage.md | 7 ++++--- .../agentbase-build/references/private-networking.md | 3 ++- skills/agentbase-build/references/security.md | 2 +- 18 files changed, 53 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9817b2a..2bc2d96 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,18 @@ review can tell which standard a project was built from. ## Unreleased +- **Docs re-read (2026-11): skills corrected where the AgentBase docs say otherwise** (docs-only; no template code changed). + Runtime Inbound Auth `IAM Permissions` / `JWT` **strips `Authorization` before the container** (only *No authorization* + forwards it) and the Console now defaults a new Runtime to IAM: `agentbase-build-deploy`, `-auth`, `-web`, `-frontend` and + `api-contract.md` no longer tell the agent to read the user JWT from `Authorization` behind a Runtime JWT (send it also in + `X-GreenNode-AgentBase-Custom-User-Token`; the Expo template and the BFF `jwt` mode still send one header, not run live). + `-llm`: a Protect & Govern limit on an API key is shared by all its models, so a fallback model does not bypass it; the + Sidecar LLM Proxy is mentioned in the MCP Gateway docs. `-memory`: event pagination `from` ≤ 5000 is now documented; no + `{sessionId}` in the namespace template; strategies and expiry are set at store creation. `-mcp`: gateway JWT settings + (Discovery URL / JWKS, Principal claim), connector Header key / prefix, default scopes, M365 VNG bundle, Slack needs a + Custom provider, 22 policy operators, conditions combine with AND. Also: Outbound IP of a Private Runtime, vCR + `backendName` / robot username / expiry, runtime env vars "non-sensitive config only", stateless contract, ephemeral + filesystem (vStorage S3), Policy Group deletion detaches from all gateways, and a Built-in Tools row in the decision guide. - **Deploy: find out how the customer can build before building.** `agentbase-build-deploy` Step 3 now starts with `scripts/detect_build_env.sh` (read-only; no login, no secret printed): which engine is installed and answering (Docker Desktop, Rancher Desktop, OrbStack, Colima, Podman, nerdctl, Finch, buildah), `linux/amd64` support, git host and CI config diff --git a/skills/agentbase-build-auth/SKILL.md b/skills/agentbase-build-auth/SKILL.md index 074887d..a99b6ce 100644 --- a/skills/agentbase-build-auth/SKILL.md +++ b/skills/agentbase-build-auth/SKILL.md @@ -35,10 +35,11 @@ Role-based access (RBAC): put `roles`/`groups` in `AUTH_FORWARD_CLAIMS` and chec ## AgentBase Runtime endpoint security -The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): **IP Access Control** (allowed source CIDRs) and **Inbound Auth type** — `IAM Permissions` (GreenNode IAM token), `JSON Web Tokens` (Discovery URL or inline JWKS), or `No authorization` (public: anyone with the URL reaches the container — what we observed on runtimes created before this setting, 2026-10). +The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): **IP Access Control** (allowed source CIDRs) and **Inbound Auth type** — `IAM Permissions` (GreenNode IAM token; the **default** for a new Runtime in the Console), `JSON Web Tokens` (Discovery URL or inline JWKS), or `No authorization` (public: anyone with the URL reaches the container — what we observed on runtimes created before this setting, 2026-10). +- **In `IAM Permissions` and `JSON Web Tokens` mode the platform strips `Authorization` (and `x-agentbase-endpoint-inbound-header-authorization`) before the request reaches the container** — only `No authorization` forwards them ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime), *Header Handling on Incoming Requests*). The agent can read a user token only from another header, so the caller sends it there too, and the agent sets `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`. `AUTH_TOKEN_HEADER=Authorization` works only behind `No authorization`; behind IAM/JWT the agent would see no token and answer 401 to every request. - The agent **still authenticates itself** (`jwt`/`api_key`) whatever the Runtime setting — defense in depth, and `Settings` blocks `AUTH_MODE=none` outside local. -- Runtime `JWT` with the same IdP as `AUTH_*` ⇒ bad tokens are rejected before the container (cheaper), and pattern **A. Direct** works. It validates the **key source** (discovery/JWKS) only — it **never replaces `AUTH_AUDIENCE` / `AUTH_ALLOWED_CLIENT_IDS`**: a token the same IdP issued to another app passes it, the agent's own checks stop it. +- Runtime `JWT` with the same IdP as `AUTH_*` ⇒ bad tokens are rejected before the container (cheaper); pattern **A. Direct** then needs the app to send the user JWT **twice** (`Authorization` for the platform, the custom header for the agent) — the Expo template sends one header and the BFF's `jwt` mode sends only `Authorization`, so with a Runtime JWT adapt the client (or use the IAM runtime + BFF). It validates the **key source** (discovery/JWKS) only — it **never replaces `AUTH_AUDIENCE` / `AUTH_ALLOWED_CLIENT_IDS`**: a token the same IdP issued to another app passes it, the agent's own checks stop it. - Runtime `IAM Permissions` ⇒ a mobile/web app cannot call it directly ⇒ pattern **B. BFF**. - Add IP Access Control for server-to-server callers (BFF, other agents) when their egress IPs are fixed. - GreenNode IAM does not publish a JWKS (404) ⇒ IAM tokens cannot be used as the agent's inbound JWTs. @@ -55,7 +56,7 @@ The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.a ## Calling the runtime endpoint: 2 patterns Read `references/patterns.md`. Summary: -- **A. Direct (default):** app → runtime endpoint with the user JWT in `Authorization`. Use when the endpoint does not require IAM in `Authorization`. +- **A. Direct (default):** app → runtime endpoint with the user JWT in `Authorization`, read by the agent (`AUTH_TOKEN_HEADER=Authorization`) only when the Runtime is `No authorization`. Behind a Runtime JWT the app sends the token twice (see *Runtime endpoint security*); behind Runtime IAM use B. - **B. BFF:** app → your BFF (verifies JWT) → runtime endpoint with an IAM token; the user JWT goes in a custom header (`AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`). Use when the endpoint requires IAM, or you need your own rate-limiting/billing. **Never** embed IAM credentials in a mobile app. Choose the pattern from the Runtime's **Inbound Auth type** (above); verify: `/invocations` **without** a token ⇒ 401 **and with a valid user token ⇒ 200** (a setup that rejects everyone also passes the first check). diff --git a/skills/agentbase-build-auth/references/patterns.md b/skills/agentbase-build-auth/references/patterns.md index 0510031..cf39bba 100644 --- a/skills/agentbase-build-auth/references/patterns.md +++ b/skills/agentbase-build-auth/references/patterns.md @@ -8,6 +8,7 @@ Mobile app ──(Authorization: Bearer )──▶ AgentBase Runtime e - Backend: `AUTH_MODE=jwt`, `AUTH_TOKEN_HEADER=Authorization`. - Frontend: `EXPO_PUBLIC_AUTH_TOKEN_HEADER=Authorization`. +- **Only behind a Runtime with Inbound Auth `No authorization`**: with `JWT` or `IAM Permissions` the platform strips `Authorization` before the container, so the agent would see no token (every request 401). Behind a Runtime JWT the app must send the user JWT in `Authorization` (for the platform) **and** in `X-GreenNode-AgentBase-Custom-User-Token` (for the agent, `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`); the Expo template sends one header — that needs a client change. Not run against a live Runtime JWT yet: verify on dev (valid token ⇒ 200). - Pros: simple, fewer hops. Cons: no dedicated rate-limit layer. - Web (Expo web): set `CORS_ALLOW_ORIGINS='["https://app.example.com"]'`. - Verify with **both**: no token ⇒ 401, **a valid user token ⇒ 200** (a misconfiguration that rejects every token also passes the first check). diff --git a/skills/agentbase-build-deploy/SKILL.md b/skills/agentbase-build-deploy/SKILL.md index 7c02d07..273a762 100644 --- a/skills/agentbase-build-deploy/SKILL.md +++ b/skills/agentbase-build-deploy/SKILL.md @@ -34,7 +34,7 @@ The scaffold commits one **non-secret** config file per environment: `src/backen | `MCP_GATEWAY_URL`, `HITL_TOOLS`, `MEMORY_MAX_CONCURRENCY` | per environment (separate dev/prod gateways) | `≈ 10 / max replicas` | | `APP_TIMEZONE`, `LLM_TASK_MAX_TOKENS`, `LLM_THINKING_BUDGET`, `LTM_MIN_SCORE`, `LTM_FACT_MAX_CHARS`, `REFLECTION_MODE`, `REFLECTION_TIMEOUT_S` | as evaluated (empty = app default) | same as the eval config | -**Which secrets may be in the runtime env file** (rendered from the CI secret store, or the user's git-ignored `.env.`): `LLM_API_KEY`, `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY`, `FEEDBACK_SIGNING_KEY`, static keys the agent presents to its own MCP servers / A2A peers (`${VAR}` in `mcp_servers.json` / `a2a_agents.json`). **Never** there: credentials tools use to call external services (SaaS API keys, OAuth tokens) — those live in AgentBase Identity (`/agentbase-build-identity`); MCP Gateway outbound credentials live in Access Control. Never `GREENNODE_CLIENT_ID/SECRET/AGENT_IDENTITY` (injected) or `GREENNODE_ENDPOINT_URL` (reserved; set `A2A_PUBLIC_URL`). Remind the user to review the env file themselves; **do not** print secret-bearing files into chat. +**Which secrets may be in the runtime env file** (rendered from the CI secret store, or the user's git-ignored `.env.`): `LLM_API_KEY`, `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY`, `FEEDBACK_SIGNING_KEY`, static keys the agent presents to its own MCP servers / A2A peers (`${VAR}` in `mcp_servers.json` / `a2a_agents.json`). **Never** there: credentials tools use to call external services (SaaS API keys, OAuth tokens) — those live in AgentBase Identity (`/agentbase-build-identity`); MCP Gateway outbound credentials live in Access Control. Never `GREENNODE_CLIENT_ID/SECRET/AGENT_IDENTITY` (injected) or `GREENNODE_ENDPOINT_URL` (reserved; set `A2A_PUBLIC_URL`). Remind the user to review the env file themselves; **do not** print secret-bearing files into chat. The Runtime docs describe the Environment Variables field as "Non-sensitive config only" ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): the platform has no secret store for it, and anyone who can read the runtime's versions (Console, API, `grn` with `-o json`) sees the values. This list is the deliberate, minimal exception — tell the user, and keep any credential a tool uses for an external service out of it. ## Step 3 — Build the image @@ -70,11 +70,11 @@ Standard Dockerfile (`src/backend/Dockerfile`): multi-stage `python:3.13-slim` * - **Agent identity — one per environment** (`-dev`, `-staging`, `-prod`; each runtime gets its own service account). Never share an identity between environments and never put the prod identity in a developer's `.greennode.json`: any dev or `api_key` caller could then fetch another environment's per-user 3LO tokens / delegated keys. Bind each runtime to its environment's identity, holding that environment's providers (`/agentbase-build-identity`); otherwise `@agent_api_key` / `@user_access_token` return 404 or "can't retrieve credential". The Runtime injects it as `GREENNODE_AGENT_IDENTITY`. Per-user helpers refuse unless `AUTH_MODE=jwt` (`IDENTITY_TRUST_ASSERTED_USER=false`, the default). - With per-user credentials, `IDENTITY_CALLBACK_URL` in `.env.` must be this environment's page and must be listed in the identity's `allowedReturnUrls`. -- **Network mode** — `PUBLIC` by default. `VPC` (VPC + Subnet + optional Route CIDRs; needs VPC Peering and a flavor that supports `agent-runtime-vpc`) only when agent code must reach an internal API or a self-hosted service; internal MCP servers go through a Private MCP Gateway instead — see `/agentbase-build` `references/private-networking.md`. +- **Network mode** — `PUBLIC` by default. `VPC` (VPC + Subnet + optional Route CIDRs; needs VPC Peering and a flavor that supports `agent-runtime-vpc`) only when agent code must reach an internal API or a self-hosted service; internal MCP servers go through a Private MCP Gateway instead — see `/agentbase-build` `references/private-networking.md`. A Private runtime reaches your side from its **Outbound IP** (runtime detail page, [manage-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/manage-runtime)): add it to the allow list of any firewall, security group or database that filters by IP. - **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): - - **Inbound Auth type** — `JWT` (same IdP as `AUTH_*`; lets the app call directly), `IAM Permissions` (server callers / BFF only), or `No authorization` (public — the agent still verifies tokens itself). Prefer JWT or IAM for production. + - **Inbound Auth type** — `IAM Permissions` (the Console default for a new Runtime, "secure by default"; server callers / BFF only), `JWT` (same IdP as `AUTH_*`), or `No authorization` (public — the agent still verifies tokens itself). Prefer IAM or JWT for production. A runtime created with `grn` / the API may not get the Console default: read the new version's inbound mode after creating it (a CI smoke test that calls `/health` without a token only works on a `No authorization` endpoint). - **IP Access Control** (`allowedCidrs`) — allowed source CIDRs for server-to-server callers with fixed egress (BFF, other agents). - - With Runtime JWT on `Authorization`, keep the agent's `AUTH_TOKEN_HEADER=Authorization` (same token); with Runtime IAM, the BFF sends the IAM token in `Authorization` and the user JWT in `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token` (`/agentbase-build-auth`). + - **The platform strips `Authorization` before the container in `IAM Permissions` and `JWT` mode** (and `x-agentbase-endpoint-inbound-header-authorization` too; only `No authorization` forwards them — [create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime), *Header Handling on Incoming Requests*). The agent can only read a token the caller also sends in another header: with Runtime IAM the BFF sends the IAM token in `Authorization` and the user JWT in `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`; with Runtime JWT the caller sends the same user JWT twice (`Authorization` for the platform, the custom header for the agent, `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`). `AUTH_TOKEN_HEADER=Authorization` is right only behind `No authorization` — behind IAM/JWT every request would be a 401 (`/agentbase-build-auth`). Other headers (`X-Api-Key`, cookies, `traceparent`, custom) are forwarded in every mode. **Every update is a full replacement — re-send the network and security settings.** Before `runtime.sh update`, read the latest version without printing its env (it holds secrets; > 100 versions ⇒ also read `--page 2…` up to `totalPage`): @@ -154,7 +154,7 @@ The scaffold ships `deploy/agent.yaml.tpl` (rendered by CI), `deploy/render_runt - **Credentials:** - CI uses a **dedicated service account**, never a person's. - Least privilege: the managed policies (`AgentBaseFullAccess`, `vcrFullAccess`, `AiPlatformFullAccess`) are broad; a custom IAM policy with only the actions CI needs can be built in the IAM Console (*Policies* → select actions — platform `/agentbase`, IAM setup). Validate it with one dev deploy. See `/agentbase-build` `references/iam-permissions.md`. - - Image pull/push uses a vCR robot account. + - Image pull/push uses a vCR robot account. The registry is `vcr.vngcloud.vn`; the image path is `vcr.vngcloud.vn//:` (the repository's `backendName`, shown on the repository detail page / in the API response — not its display name) and the `docker login` username is the robot's full `backendName`, `-` (e.g. `109072-deploy-bot`), not the name typed at creation ([supporting-services](https://docs.greennode.ai/ai-stack/agent-base/supporting-services)). A robot needs push + pull permissions and is created with a **Duration (days)**: note its expiry date and rotate it before then (what an expired robot does to a runtime's image pull is not documented). - **What the smoke test proves:** it calls `GET /ready` (retried for a minute): 200 = the replica can serve — and with `AUTH_MODE=jwt` can fetch the IdP's JWKS, so users can log in; 503 = it cannot (the body names `jwks`). `degraded` (HTTP 200, `failing: ["llm"]` or `backing_service`) reports a dependency that failed recently — it needs real traffic to appear, so a bad LLM key is not caught before the first request. A release from before `/ready` answers 404 and is checked on `/health`. It does not show which version serves: the deploy step checks the latest version's `imageUrl` and the tag's digest; Langfuse `release` (= tag) confirms what serves traffic. - **No automatic rollback** (a half-applied rollback is worse than a failed deploy): when a deploy fails, the last step prints the previous image and the exact dispatch (`image_tag=`, from that release's tag). - **`/health` vs `/ready`:** `/health` is the platform's liveness probe and stays 200 while the process runs — restarting a replica does not fix an IdP or LLM outage. Monitor `/ready` from outside (uptime check, alert on 503). @@ -187,7 +187,7 @@ Runtimes are billed continuously (a `STOPPED` runtime keeps its config without c 1. Runtime(s) incl. custom endpoints — `/agentbase-teardown` or `/agentbase-deploy`. 2. Memory store(s), identity + Access Control providers, LLM API keys, vCR images — `/agentbase-teardown`. 3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-` service account). -4. Policy Groups — `/agentbase-policy` (detach from the gateway first). +4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible and **detaches it from every gateway at once**; a gateway without a Policy Group answers 403 to every `tools/call` ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)) — detach and re-bind (or delete the gateway) deliberately. 5. MCP Connectors (catalog/custom) — Console. 6. The **Orphaned** runtime service account (`auto-sa-`) and gateway service accounts — revoke in IAM. 7. Inbound API keys: remove their hashes from `AUTH_API_KEY_SHA256` of the callers' environments, and the A2A keys peers hold for this agent. diff --git a/skills/agentbase-build-frontend/SKILL.md b/skills/agentbase-build-frontend/SKILL.md index 50e748a..3bfb63c 100644 --- a/skills/agentbase-build-frontend/SKILL.md +++ b/skills/agentbase-build-frontend/SKILL.md @@ -30,7 +30,7 @@ What it does: `CI=1 npx create-expo@5.0.1 --template blank-typescript@sdk- |---|---| | `EXPO_PUBLIC_AGENT_URL` | **Direct**: runtime endpoint (from `/agentbase-deploy`). **BFF**: the BFF base URL. Local backend: iOS simulator / web `http://127.0.0.1:8080`, Android emulator `http://10.0.2.2:8080` (or `adb reverse tcp:8080 tcp:8080` and keep 127.0.0.1), real device `http://:8080` — but a local backend with `AUTH_MODE=none` listens on 127.0.0.1 only (anyone on the LAN could otherwise act as any user): use `AUTH_MODE=jwt` locally (then it binds 0.0.0.0) or a tunnel. Release builds require `https://` | | `EXPO_PUBLIC_AUTH_MODE` | `jwt` · `none` (only when the backend uses `AUTH_MODE=none` locally) | -| `EXPO_PUBLIC_AUTH_TOKEN_HEADER` | Header the **app** sends `Bearer ` in: `Authorization` in both modes. Direct: = backend `AUTH_TOKEN_HEADER`. BFF: stays `Authorization` even though the backend uses `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token` — the BFF moves it there | +| `EXPO_PUBLIC_AUTH_TOKEN_HEADER` | Header the **app** sends `Bearer ` in: `Authorization` in both modes. Direct: = backend `AUTH_TOKEN_HEADER` (works only behind a Runtime with Inbound Auth *No authorization*: IAM and JWT modes strip `Authorization` before the container). BFF: stays `Authorization` even though the backend uses `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token` — the BFF moves it there | | `EXPO_PUBLIC_OIDC_ISSUER` / `CLIENT_ID` / `SCOPES` / `AUDIENCE` | IdP **public** client + PKCE (table below); register the redirect URIs below | | `EXPO_PUBLIC_OIDC_EPHEMERAL_SESSION` | `true` ⇒ iOS private browser session (no SSO cookie shared with Safari) | | `EXPO_PUBLIC_STREAMING` | `true` to use SSE | diff --git a/skills/agentbase-build-llm/SKILL.md b/skills/agentbase-build-llm/SKILL.md index 28816a7..6b2849f 100644 --- a/skills/agentbase-build-llm/SKILL.md +++ b/skills/agentbase-build-llm/SKILL.md @@ -99,14 +99,14 @@ Two layers, both answer **429**: | Layer | Scope | Configure | |---|---|---| | MaaS account limit | **10 requests/min, 14,400/day per account**, shared by all models and keys ([available-models](https://docs.greennode.ai/ai-stack/model-as-a-service/available-models)) | More only via whitelist request to GreenNode | -| **Rate Limit** (Protect & Govern) | Requests and/or tokens per period, enforced **independently per API key and per model** it's attached to; rejected as soon as either threshold is hit | Console *Protect & Govern → Rate Limit* (Root/Admin) — [rate-limit](https://docs.greennode.ai/ai-stack/agent-base/protect-govern/rate-limit) | +| **Rate Limit** (Protect & Govern) | Requests and/or tokens per period, enforced **independently per API key and per model** it's attached to (a key's cap is totalled across all its models, a model's cap across all keys); rejected as soon as either threshold is hit | Console *Protect & Govern → Rate Limit* (Root/Admin) — [rate-limit](https://docs.greennode.ai/ai-stack/agent-base/protect-govern/rate-limit) | Standard: - **One LLM API key per agent per environment** (`/agentbase-llm`), so a rate limit and usage can be attached to exactly that agent; a separate key for eval/CI. - Attach a Rate Limit to each agent key (requests/day **and** tokens/day) sized from the expected traffic × calls per turn (router + agent + tool rounds + judge). This caps runaway loops and cost. - Budget: track spend in [Usage & Budget](https://docs.greennode.ai/ai-stack/usage-budget); per-trace cost is in Langfuse. - On 429: - - **A fallback helps for a Protect & Govern limit**: that limit is per key **and per model**, so another model has its own quota. + - **A fallback helps only for a model-level Protect & Govern limit.** A limit attached to an API key is totalled across **all models** of that key, and one attached to a model across all keys ([rate-limit](https://docs.greennode.ai/ai-stack/agent-base/protect-govern/rate-limit), *Overview*); both run in parallel. A fallback model under the same key shares the key's cap. - **A fallback does not help for the account limit** (10 RPM / 14,400 per day), which is shared by every model. - The SDK backoff (0.5–8 s) is much shorter than a per-minute window: after a 429 from a per-minute limit, retries in the same minute usually fail too. More retries don't fix it; fewer calls per turn do. Lower `MAX_TOOL_ROUNDS`, disable adaptive routing / reflection, lower eval `--concurrency`, or raise the limit. - A turn that still fails returns a clear error. @@ -179,7 +179,7 @@ Re-measure on your models before changing defaults — `make check-creds` shows The prompt is `[static system prompt, *history, latest user message, per-turn context message, *this turn's tool rounds]`. The static system prompt and the tool schemas stay byte-identical. Everything that changes per turn goes in the **context system message** (`build_context_message` in `app/graph/builder.py`), inserted right after the latest user message and never persisted: current date/time, summary, recalled notes, critique. The cached prefix (system prompt + history + user message) is therefore reused across turns. Don't put timestamps, request ids or per-user data in the system prompt. MaaS returns `prompt_tokens_details.cached_tokens` (measured: 8128/8174 tokens cached on repeat) ⇒ Langfuse records `input_cache_read`. -Keep `LLM_BASE_URL` pointing at MaaS on the Runtime too. An older doc mentioned a sidecar LLM proxy on the Runtime; the current docs don't, so don't route through one unless GreenNode confirms it. +Keep `LLM_BASE_URL` pointing at MaaS on the Runtime too. The MCP Gateway docs ([mcp-gateway](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/mcp-gateway), *MCP Gateway vs Sidecar LLM Proxy*) mention a Sidecar LLM Proxy (auto-injected, `localhost:18080`, bypasses Kong so no Policy Group applies) but document no path, auth or model mapping, so don't route through it unless GreenNode confirms it. ## Troubleshooting @@ -191,7 +191,7 @@ Keep `LLM_BASE_URL` pointing at MaaS on the Runtime too. An older doc mentioned | `ProviderStreamError` / bare `openai.APIError: ` (no status), often after some tokens | Provider failed inside the stream (SSE error chunk: engine overloaded or crashed) | Falls back automatically when the tier has fallbacks; without fallbacks it surfaces as an error, so add a fallback | | WARNING `LLM tier …: worst case … > REQUEST_TIMEOUT_S` | Timeout × attempts + retry backoff exceed the request budget | Use the §4 time-budget table values | | 404 model not found (ERROR log names the env var) | Wrong `path` or model removed (MaaS may also answer 404 for a model not enabled for the key) | Fix the path; with a fallback the request still works but every call fails over first, so fix it | -| Frequent 429 | Account limit (10 RPM, shared by all models) or a Protect & Govern limit (per key and model) | Account limit: a fallback does NOT help; request a whitelist, turn off adaptive routing/reflection, lower `MAX_TOOL_ROUNDS`, lower eval `--concurrency`. Protect & Govern: a fallback to another model helps, or raise the limit | +| Frequent 429 | Account limit (10 RPM, shared by all models) or a Protect & Govern limit (key-level or model-level) | Account limit: a fallback does NOT help; request a whitelist, turn off adaptive routing/reflection, lower `MAX_TOOL_ROUNDS`, lower eval `--concurrency`. Protect & Govern: a limit on the API key is shared by every model of that key — a fallback does NOT help, raise it or cut calls; a limit on a model — a fallback to another model helps | | 504 although no call failed | Thinking models streaming for minutes (the inactivity timeout never fires) | Budget the turn from Langfuse p95 (§4); non-thinking models for internal flows, output caps | | WARNING `… hit its output cap (finish_reason=length)` | A thinking model spent the flow's `max_tokens` (§2) | Non-thinking model for that flow, or raise `LLM_TASK_MAX_TOKENS` | | Answer starts with reasoning-like text ("Let me check…", English in a Vietnamese reply) / router or judge returns prose instead of JSON | `LLM_THINKING_BUDGET` cut the thinking and the model kept thinking in the content (§6) | JSON flows: `0` (thinking off); conversation flows: remove the budget (`-1`) or raise it | diff --git a/skills/agentbase-build-mcp/SKILL.md b/skills/agentbase-build-mcp/SKILL.md index 543d34d..ac92b2b 100644 --- a/skills/agentbase-build-mcp/SKILL.md +++ b/skills/agentbase-build-mcp/SKILL.md @@ -39,9 +39,9 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou ## MCP connection procedure (Coding Agent follows this order; every mutating step needs user confirmation) -1. **Upstream credential** — Secret Providers live in **Access Control** (`/agentbase-build-identity` explains the provider types; create them with `/agentbase-identity`): create an API key / OAuth2 provider (Secret Provider). Catalog connectors can use GreenNode's **Managed** secret; Custom requires creating your own OAuth App and declaring the provider. Never let the user paste secrets into chat. +1. **Upstream credential** — Secret Providers live in **Access Control** (`/agentbase-build-identity` explains the provider types; create them with `/agentbase-identity`): create an API key / OAuth2 provider (Secret Provider). Catalog connectors can use GreenNode's **Managed** secret; Custom requires creating your own OAuth App and declaring the provider — **Slack is the docs' example of a connector with no Managed provider** ([connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector)): create the Slack OAuth App yourself, add it as an OAuth provider in Access Control, and only then does it appear in the connector's Secret Provider dropdown. Never let the user paste secrets into chat. 2. **Gateway** — `/agentbase-gateway create` or reuse an existing gateway (`GET /gateway/api/v1/gateways`): - - `inboundAuth`: **IAM** (agent calls with the runtime's IAM — recommended default) | **JWT** (forwards end-user identity; Policy uses `jwt:`, `principal.`) | NONE (lab only). + - `inboundAuth`: **IAM** (agent calls with the runtime's IAM — recommended default) | **JWT** (forwards end-user identity; the gateway validates the token with a **Discovery URL** or an inline **JWKS**, plus optional Audience / Allowed clients / Allowed scopes; Policy uses `jwt:` where the principal is the gateway's **Principal claim** field — default `sub` — and `principal.`; [manage-mcp-gateway](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/mcp-gateway/manage-mcp-gateway)) | NONE (lab only). - Network: **Public**, or **Private** (VPC, Subnet, Route CIDRs, Flavor, Replicas; needs VPC Peering) when the MCP servers live in your VPC or data center — `/agentbase-build` `references/private-networking.md`. 3. **Connector** — Console *AgentBase → MCP Connectors → Catalog → Connect* (or *Add Custom Connector*): pick the gateway and auth mode. **Name it `[a-z0-9]{3,50}` — no dash, no underscore** (e.g. `hrmcp`, not `hr-mcp`): the name becomes the policy action prefix `__`, and actions accept only letters, digits and `_` (`/agentbase-policy`), while gateway target names are `^[a-z0-9-]{3,50}$` ⇒ with a dash no per-tool policy can be written. (Derived from the two validation rules — verify once on the real API.) @@ -49,11 +49,11 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou | Auth mode | Credential | Note | |---|---|---| | OAuth 2LO / 3LO | Secret Provider (Managed / Custom) in Access Control | 3LO needs a Return URL + user consent | - | API Key 2LO / 3LO | Secret Provider (the user enters the key in the Console/Identity — never in chat) | | + | API Key 2LO / 3LO | Secret Provider (the user enters the key in the Console/Identity — never in chat) | The key travels as a header; *Advanced settings → Parameters* sets a custom **Header key** / **Header value prefix** pair (docs example `Authorization` / `Bearer`) for upstreams that expect e.g. `X-API-Key` | | Inbound forward | The agent's own inbound credential | Gateway inbound ≠ NONE. ⚠ See below | | No authorization | — | | - ⚠ **OAuth 3LO from the Console: consent is given at connect time by whoever clicks *Authorize*** ([connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector)) and the connection belongs to the **project** (`/agentbase-gateway` describes `flow=3LO` as per-end-user — verify which one you get). Until verified, assume every project member and every agent user acts upstream **as that person** (their mailbox, repos, channels) ⇒ connect with a dedicated bot/service account where possible, grant the **minimal OAuth scopes**, and put **every side-effect tool** of a 3LO connector in `HITL_TOOLS`. + ⚠ **OAuth 3LO from the Console: consent is given at connect time by whoever clicks *Authorize*** ([connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector)) and the connection belongs to the **project** (`/agentbase-gateway` describes `flow=3LO` as per-end-user — verify which one you get). Until verified, assume every project member and every agent user acts upstream **as that person** (their mailbox, repos, channels) ⇒ connect with a dedicated bot/service account where possible, grant the **minimal OAuth scopes** (a catalog connector pre-selects default scopes, at least one required — review and trim them; **M365 VNG bundles 4 services**: SharePoint, Outlook Mail, Outlook Calendar, Teams — so restrict the bundle with `allow_tools` and per-tool policy), and put **every side-effect tool** of a 3LO connector in `HITL_TOOLS`. **Inbound forward** sends the MCP server the **same credential the agent used on the gateway** ([connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector)). With gateway inbound **IAM** that is the agent's platform IAM token (runtime service account; locally your developer service account with `AgentBaseFullAccess`) — the server could call AgentBase APIs as your agent ⇒ **never use Inbound forward with IAM inbound**. With inbound **JWT** it forwards the end user's JWT ⇒ only towards **your own** MCP server that validates the same IdP (`/agentbase-build-mcp-server`), never a third-party server. diff --git a/skills/agentbase-build-mcp/references/governance.md b/skills/agentbase-build-mcp/references/governance.md index 09a752c..a37c7e0 100644 --- a/skills/agentbase-build-mcp/references/governance.md +++ b/skills/agentbase-build-mcp/references/governance.md @@ -34,7 +34,7 @@ E2E result with the agent running as ``: `stock_stock_quote` ⇒ re ## Common conditions (examples — verify before use) -The two sources disagree here too. The Console docs ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups)) show operators such as `ipInRange` and keys such as `request.client_ip` / `request.timestamp.hour`, with `in` taking `"Admin,Editor"`. `/agentbase-policy` (API reference) lists **9 operators** (`equals`, `notEquals`, `in`, `like`, `contains`, `lessThan`, `lessThanOrEqual`, `greaterThan`, `greaterThanOrEqual`), keys **`context.ip`**, **`context.input.`**, **`principal.`** only, and `in` takes a **string list**. Check the live catalog `GET https://agentbase.api.vngcloud.vn/policy/api/v1/policies/condition-operators` and read `/agentbase-policy` (`references/policy-statement.md`) before creating; the API answers 400 for an unsupported operator. +The two sources disagree here too. The Console docs ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups), *Supported operators*) list 22 operators in five groups — equality `equals`/`notEquals`; comparison `lessThan`/`lessThanOrEqual`/`greaterThan`/`greaterThanOrEqual`; string `like`/`contains`/`containsAll`/`containsAny`/`startsWith`/`endsWith`; membership `in`/`has`/`hasTag`/`is`/`memberOf`; network `ipInRange`/`isIpv4`/`isIpv6`/`isLoopback`/`isMulticast` — and show keys such as `request.client_ip` / `request.timestamp.hour`, with `in` taking `"Admin,Editor"`. `/agentbase-policy` (API reference) lists **9 operators** (`equals`, `notEquals`, `in`, `like`, `contains`, `lessThan`, `lessThanOrEqual`, `greaterThan`, `greaterThanOrEqual`), keys **`context.ip`**, **`context.input.`**, **`principal.`** only, and `in` takes a **string list**. Check the live catalog `GET https://agentbase.api.vngcloud.vn/policy/api/v1/policies/condition-operators` and read `/agentbase-policy` (`references/policy-statement.md`) before creating; the API answers 400 for an unsupported operator. | Purpose | Condition (statement JSON, API form) | Note | |---|---|---| @@ -42,7 +42,9 @@ The two sources disagree here too. The Console docs ([policy-groups](https://doc | By role | `{"in": {"principal.role": ["Admin", "Editor"]}}` | list value, not `"Admin,Editor"` | | Specific client IPs | `{"in": {"context.ip": ["203.0.113.10", "203.0.113.11"]}}` | CIDR ranges (`ipInRange`) appear only in the Console docs — verify | | Limit a tool argument | `{"lessThan": {"context.input.qty": 100}}` | `tools/call` arguments; a missing argument ⇒ deny | -| Business hours | Console docs: `request.timestamp.hour`; not in the API key list | verify before relying on it | +| Business hours | Console docs: `request.timestamp.hour`; not in the API key list | verify before relying on it. The docs' example is two conditions (`greaterThan` 9 **and** `lessThan` 17) | + +Multiple conditions in one policy are combined with **AND** (all must be true for the policy to match — [policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups), *Conditions*). For an OR, write one policy per branch. ## Governance checklist before prod diff --git a/skills/agentbase-build-memory/SKILL.md b/skills/agentbase-build-memory/SKILL.md index c76e09f..b702de6 100644 --- a/skills/agentbase-build-memory/SKILL.md +++ b/skills/agentbase-build-memory/SKILL.md @@ -20,7 +20,7 @@ Call `/agentbase-memory` to create it (following that skill's HARD GATE). Sugges - `name`: `-memory` - `eventExpiryDuration`: 30 (days of conversation retention, allowed 1–365; per data policy). Name ≤ 50 chars `^[a-zA-Z0-9._-]*$`. This does not bound a session that stays active, see "Long sessions" (Step 4). - Strategy: **one per agent** with this template: `SEMANTIC` (general facts), **or** `USER_PREFERENCE`, **or** `CUSTOM` + `customFactExtractionPrompt` when the business needs its own extraction. The app reads and writes a single namespace, `/strategies/{MEMORY_STRATEGY_ID}/actors/{actorId}`, so records produced by a second strategy on the same store are **never recalled**. If you need two, extend `AgentBaseLTM.search` to query each strategy's namespace and merge by score (the strategy id becomes a list in config), and decide which namespace `remember` writes to. The template does not do this. -- `namespaceTemplate`: **`/strategies/{memoryStrategyId}/actors/{actorId}`** — MUST match `AgentBaseLTM.namespace()` in `long_term.py`. +- `namespaceTemplate`: **`/strategies/{memoryStrategyId}/actors/{actorId}`** — MUST match `AgentBaseLTM.namespace()` in `long_term.py`. Do not add `{sessionId}`: it would put each session's records in its own namespace, and recall is an exact namespace match, so a new session would find nothing. Strategies and `eventExpiryDuration` are set when the store is created; the docs describe no update call, so changing them means a new store (confirm in the Console before promising an in-place change). - `enableAutomaticMemoryRecordGeneration`: true. The memory docs describe extraction from **conversational** events, but this template writes only binary checkpoint events. Verify it on dev (Step 3). Write into each environment's `.env`: `MEMORY_BACKEND=agentbase`, `MEMORY_ID=`, `MEMORY_STRATEGY_ID=` (the id returned by `GET /memories/{memoryId}/long-term-memory-strategies` — see [memory docs](https://docs.greennode.ai/ai-stack/agent-base/memory); the app **refuses to start** with `LTM_ENABLED=true` and `MEMORY_STRATEGY_ID` empty/`default`, because a wrong id makes recall silently return nothing); write `memory_id` into `.agentbase-state.json`. @@ -88,7 +88,7 @@ Read `references/compression.md` to pick thresholds. Mechanism: - The bridge writes every graph step as binary events: about **24 events per turn** without tools (measured on real Memory: 3 turns = 72 events), more with tool rounds and HITL. On **every turn** it lists **all** events of the session (pages of 100 per request; the template lists them twice per turn). `RemoveMessage` does not delete events; old events stay until `eventExpiryDuration`. Read volume and latency therefore grow linearly with the session's length, with or without compression. - Measured offline (real bridge + a fake Memory client, 40 turns, about 1 KB per message): turn 2 reads 46 events (43 KB). Turn 40 reads about 1,800 events in about 20 list requests: **3.6 MB with compression, 9.9 MB without**. A review on real Memory measured 207 KB → 2.6 MB per turn over 40 turns. Compression only slows the growth. -- **Possible ceiling (unverified):** an offset limit of **5000** is documented for Runtime **logs** pagination. Whether the Memory events API has the same limit is **not documented or verified**. If it does, a session with more than about 5,000 events (roughly 200 tool-less turns, fewer with tools) could not be fully read, and checkpoint loads would fail or miss older channel data. Read volume grows linearly in any case, so don't let sessions grow that long. +- **Ceiling (documented):** the Memory docs ([memory](https://docs.greennode.ai/ai-stack/agent-base/memory), *Memory Service Limits*) give a maximum `from` of **5000** for event pagination (offset-based). A session with more than about 5,000 events (roughly 200 tool-less turns, fewer with tools) can no longer be read in full, and checkpoint loads could fail or miss older channel data (what exactly happens past the limit is not documented — verify on dev before raising `SESSION_MAX_TURNS`). Read volume grows linearly in any case, so don't let sessions grow that long. - **Mitigation — session rotation, enforced by the server:** the graph counts turns in state (`turns`, at each turn start). An HTTP chat on a session with `turns ≥ SESSION_MAX_TURNS` (default 50, `0` = off) continues in a **new** `session_id`, seeded with the old `summary` and the (already compressed) messages (`service._rotate`, span `memory.session_rotated`). The response's `session_id` is the new one and the client must use it from then on — the web and mobile templates do (`chatState` adopts it). The old session is no longer written, so each turn's reads stay bounded. In-process callers (A2A, eval, jobs via `run_chat`) are not rotated: they keep their own id. A client can still rotate earlier ("new conversation", per day or per topic); LTM carries facts across sessions. - A shorter `eventExpiryDuration` only bounds sessions that go idle. The bridge writes a channel's value only when it changes, so in a session that stays active longer than the expiry, values not rewritten since (e.g. `summary`) can expire under the latest checkpoint. Prefer rotation, and verify on dev before relying on expiry. @@ -115,7 +115,7 @@ Read `references/compression.md` to pick thresholds. Mechanism: | **10 concurrent requests / IAM account** — shared across all runtime replicas | 429 `Too many concurrent streaming requests for this user. Limit: 10` | `MEMORY_MAX_CONCURRENCY` (process-level FIFO limiter shared by the bridge's sync calls and LTM's async calls) + backoff retry (bridge for checkpoints, `with_retry` for LTM search/save) | | Search query ≤ 1000 characters | 400 | truncated to `MEMORY_QUERY_MAX_CHARS` (keeps the tail) | | Search `limit` 5–200 | outside the documented range | `LTM_RECALL_LIMIT` validated 5–200 at startup; `AgentBaseLTM.search` clamps defensively, results cut to the requested limit | -| Events per session: an offset limit of 5000 (documented for Runtime logs; **unverified** for Memory events) | if it applies, a longer session can't be fully loaded | session rotation (Step 4, "Long sessions") | +| Events per session: `from` ≤ 5000 for event pagination (documented, *Memory Service Limits*; behaviour past it not verified) | a longer session can't be fully loaded | session rotation (Step 4, "Long sessions") | | Checkpoint reads occasionally very slow (seen ~7 minutes with default timeout/retry) | request hangs | `MEMORY_TIMEOUT_S=10`, `REQUEST_TIMEOUT_S`, span `memory.checkpoint_load` | Measured 12 parallel conversations (24 turns) on real Memory: limit 8 → 14 retries on 429, 18s; limit 6 → 9 retries, 24s; limit 4 → 2 retries, 53s. No turn failed thanks to retries. The server-side counter lags, so bursts still get 429 ⇒ **need both the limit and retries**. Scaling to many replicas: set `MEMORY_MAX_CONCURRENCY ≈ 10 / replica count`; heavy traffic ⇒ request a quota increase or split service accounts. diff --git a/skills/agentbase-build-memory/references/compression.md b/skills/agentbase-build-memory/references/compression.md index 6585837..1f43a06 100644 --- a/skills/agentbase-build-memory/references/compression.md +++ b/skills/agentbase-build-memory/references/compression.md @@ -28,4 +28,4 @@ Compression shrinks the **prompt** and each **new** checkpoint. It does not shri | KB read per turn, no compression | 43 | 828 | 2,780 | 9,918 | | KB read per turn, compression at 1,500 tokens | 43 | 724 | 1,672 | 3,570 | -The template writes about 23 events per tool-less turn. If the events API has the 5000 offset limit documented for Runtime logs (**unverified** for Memory), a session could no longer be fully read after roughly 200 such turns. Read cost grows with every turn either way. The remedy is **session rotation**: a new `session_id` seeded with the old `summary`. A shorter `eventExpiryDuration` only helps idle sessions. See SKILL.md Step 4, "Long sessions". +The template writes about 23 events per tool-less turn. The Memory docs limit event pagination to `from` ≤ 5000 (*Memory Service Limits*), so a session can no longer be fully read after roughly 200 such turns. Read cost grows with every turn either way. The remedy is **session rotation**: a new `session_id` seeded with the old `summary`. A shorter `eventExpiryDuration` only helps idle sessions. See SKILL.md Step 4, "Long sessions". diff --git a/skills/agentbase-build-web/SKILL.md b/skills/agentbase-build-web/SKILL.md index 717a8a1..c2d4f27 100644 --- a/skills/agentbase-build-web/SKILL.md +++ b/skills/agentbase-build-web/SKILL.md @@ -45,7 +45,7 @@ a failed run leaves `.agentbase-setup-pending` and the next run re-applies the t | `NEXT_PUBLIC_STREAMING`, `NEXT_PUBLIC_IDLE_TIMEOUT_S` | The **only** browser settings (public bundle — never a secret) | | `BFF_AGENT_URL` | Runtime endpoint (`/agentbase-deploy`) or `http://127.0.0.1:8080` (`make dev`). Production: https | | `BFF_AUTH_MODE` | `jwt` (OIDC sign-in) · `none` = local backend with `AUTH_MODE=none` (refused in production) | -| `BFF_RUNTIME_AUTH` | Runtime Inbound Auth `jwt` / *No authorization* ⇒ `jwt` (backend `AUTH_TOKEN_HEADER=Authorization`) · *IAM Permissions* ⇒ `iam` (backend `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`) + `BFF_IAM_CLIENT_ID` / `BFF_IAM_CLIENT_SECRET` | +| `BFF_RUNTIME_AUTH` | Runtime Inbound Auth *No authorization* ⇒ `jwt` (backend `AUTH_TOKEN_HEADER=Authorization`; a Runtime `JWT` strips `Authorization` before the container, so `jwt` mode does **not** work behind it — the BFF would have to send the user token in a custom header too) · *IAM Permissions* ⇒ `iam` (backend `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`) + `BFF_IAM_CLIENT_ID` / `BFF_IAM_CLIENT_SECRET` | | `BFF_OIDC_ISSUER` / `CLIENT_ID` / `CLIENT_SECRET` / `SCOPES` / `AUDIENCE` | The IdP client of this app; per-IdP values as in `/agentbase-build-frontend` (Auth0 `AUDIENCE`, Cognito without `offline_access`). Confidential client recommended | | `BFF_APP_ORIGIN` | Public origin, e.g. `https://chat.example.com` — redirect URI `/api/auth/callback`, post-logout URI `/`, CSRF Origin check | | `BFF_SESSION_SECRET` | ≥ 32 random chars (`openssl rand -base64 48`) — encrypts the session cookie. The user sets it, never in chat | diff --git a/skills/agentbase-build/references/api-contract.md b/skills/agentbase-build/references/api-contract.md index 79fdd0d..8963027 100644 --- a/skills/agentbase-build/references/api-contract.md +++ b/skills/agentbase-build/references/api-contract.md @@ -7,7 +7,7 @@ Source of truth: the docstring at the top of `src/backend/app/service.py`. Contr | Header | Required | Notes | |---|---|---| | `Content-Type: application/json` | ✔ | | -| `Authorization: Bearer ` | ✔ when `AUTH_MODE=jwt` | Header name configurable via `AUTH_TOKEN_HEADER` (e.g. when the runtime endpoint uses `Authorization` for IAM ⇒ use `X-GreenNode-AgentBase-Custom-User-Token`) | +| `Authorization: Bearer ` | ✔ when `AUTH_MODE=jwt` | Header name configurable via `AUTH_TOKEN_HEADER` (IAM and JWT Inbound Auth strip `Authorization` before the container ⇒ behind them use `X-GreenNode-AgentBase-Custom-User-Token`; `Authorization` only behind *No authorization*) | | `X-GreenNode-AgentBase-Session-Id` | ✔ for chat/resume | UUID, 1 conversation = 1 session. Format `^[A-Za-z0-9][A-Za-z0-9-]{0,127}$` (blocks `../`, `/`), invalid ⇒ 400 | | `X-GreenNode-AgentBase-Custom-Api-Key: ` | ✔ when `AUTH_MODE=api_key` | Only for trusted callers (server/BFF/other agents) | | `X-GreenNode-AgentBase-User-Id` | jwt: ✖ · **api_key: ✔** | jwt: if sent it must match `sub` (otherwise ⇒ 403). api_key: required (memory is isolated per user). Format `^[A-Za-z0-9][A-Za-z0-9._@+=-]{0,127}$`, invalid ⇒ 400. A JWT whose `sub` is outside this format ⇒ `user_id = "u-" + sha256(iss|sub)[:40]` | diff --git a/skills/agentbase-build/references/architecture.md b/skills/agentbase-build/references/architecture.md index 63154a9..acb5966 100644 --- a/skills/agentbase-build/references/architecture.md +++ b/skills/agentbase-build/references/architecture.md @@ -75,5 +75,6 @@ START → compress → recall → [route] → agent ─┬─ no tool call ─ - **Router / multi-agent within one agent:** `LLM_ADAPTIVE_ROUTING` provides a `route` node; for other business routers add a node that calls `get_llm("", schema=MyModel)` (a Pydantic model): `get_llm` applies structured output to **every model of the fallback chain** (`method="function_calling"` by default) and keeps tiers and tracing. Never call `.with_structured_output()` / `.bind_tools()` on what `get_llm` returns (tools go in `get_llm(task, tools=…)`): for internal tasks it is a wrapper runnable (`RunnableLambda`, for the trace layout), not a `ChatOpenAI`, so those methods fail — and they would bypass the fallbacks. For a cheap free-form classification, the JSON-prompt pattern of `app/llm/routing.py::classify` also works (prompt for JSON only → `re.search(r"\{.*\}", …)` → `json.loads` → validate → safe default on any error, inside `tracing.step(...)`). Add the new task to `DEFAULT_TASK_TIERS` and its module to `LLM_MODULES` (`tests/conftest.py`). Sub-agents are subgraphs compiled **without** their own checkpointer (they share the parent graph's). An agent owned / deployed separately by another team ⇒ A2A. - **RAG:** the retriever is a tool (`app/tools/`) or an MCP server; do not stuff documents into the system prompt. +- **Files the agent receives or produces:** AgentBase has no storage service and the runtime's container filesystem is ephemeral (lost on restart, scale or a new version — [faq](https://docs.greennode.ai/ai-stack/agent-base/faq)). The docs' answer is S3 via vStorage (`boto3`, endpoint per region, an S3 key kept in Access Control and fetched through the agent identity). Same rule as the state rule in `checklist.md`: nothing durable on the container disk. - **Background / batch jobs:** call `service.run_chat()` in-process (skips inbound auth) — internal jobs only. - **Multiple channels (Zalo/Telegram):** a channel is an adapter calling `/invocations`; if only a template chatbot is needed, consider OpenClaw (`/agentbase-deploy`). diff --git a/skills/agentbase-build/references/checklist.md b/skills/agentbase-build/references/checklist.md index c93a8a9..a133bd3 100644 --- a/skills/agentbase-build/references/checklist.md +++ b/skills/agentbase-build/references/checklist.md @@ -12,6 +12,7 @@ Run each item and report `PASS` / `FAIL` / `N/A` with evidence (command + short ## Runtime contract - [B] `GET /health` → 200; listens on `0.0.0.0:8080`. +- [B] Stateless: no session state only in process memory — the docs' *Service Contract* ([runtime-reference](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/runtime-reference)) says to use the Memory Service. `MEMORY_BACKEND=inmemory`, the in-memory A2A task store and per-replica caches/limits are acceptable only on one replica or with the per-replica semantics in `platform-coverage.md` → *Known limits*. - [B] `make test` green; `make lint` clean. - Image built with `--platform linux/amd64`, runs as non-root. diff --git a/skills/agentbase-build/references/decision-guide.md b/skills/agentbase-build/references/decision-guide.md index be2e8c9..329a626 100644 --- a/skills/agentbase-build/references/decision-guide.md +++ b/skills/agentbase-build/references/decision-guide.md @@ -7,7 +7,7 @@ Principle: **build minimal by default; every enabled component must have a reaso | Component | Reason | |---|---| | Runtime contract, `service.py`, standard graph | Common skeleton | -| Inbound auth (`jwt` / `api_key`) | The Runtime's own Inbound Auth (IAM / JWT / None) is optional and may be None ⇒ the agent always verifies callers itself | +| Inbound auth (`jwt` / `api_key`) | The Runtime's own Inbound Auth (IAM — the Console default — / JWT / None) may be None ⇒ the agent always verifies callers itself; behind IAM/JWT the platform strips `Authorization`, so the agent reads the user token from `X-GreenNode-AgentBase-Custom-User-Token` | | Short-term memory (checkpointer) | Multi-turn conversations; HITL also needs a checkpointer | | Compression | Insurance for long conversations; costs nothing until the threshold is exceeded | | Langfuse tracing (can be sampled) | An agent cannot be debugged without traces | @@ -79,6 +79,7 @@ Details and measurements: `/agentbase-build-llm`. | Pure logic, used only by this agent, no secrets needed | Local tool | | SaaS available in the connector catalog (GitHub, Slack, M365) | MCP Connector (no code) | | Internal system, shared by many agents, needs policy/audit | **Build an MCP server** (`/agentbase-build-mcp-server`) + Custom Connector | +| Agent must run model-written code or drive a browser | **Built-in Tools** (Code Interpreter, Browser: per-session sandboxes behind an MCP endpoint, [built-in-tools](https://docs.greennode.ai/ai-stack/agent-base/built-in-tools)) — whitelist-only rollout (contact support), pricing not published; the sandbox has outbound public internet, and sessions do not expire. Never exec model-written code in the agent's own container. This repo has no template or tested recipe for it (`platform-coverage.md`) | | Needs per-user data in the target system | MCP server using OAuth 3LO or inbound JWT forwarding so `current_user()` comes from the token | | Internal document lookup (RAG), POC / ≤ a few dozen files | Local `search_knowledge`: drop `.md/.txt` into `app/knowledge/` (auto-registered when files exist) | | Prod RAG: many documents, semantic search, frequent updates | MCP server with a vector store, **keep the tool name `search_knowledge`** (set `"prefix": false` on that server in `mcp_servers.json` — MCP tools are otherwise exposed as `_`; a name collision fails `collect_tools`) and the `[Source: …]` format | diff --git a/skills/agentbase-build/references/platform-coverage.md b/skills/agentbase-build/references/platform-coverage.md index ff94b78..d5da72f 100644 --- a/skills/agentbase-build/references/platform-coverage.md +++ b/skills/agentbase-build/references/platform-coverage.md @@ -1,6 +1,6 @@ # GreenNode AgentBase platform coverage ↔ skills -Maps each item in the AgentBase docs (docs.greennode.ai/ai-stack/agent-base, 2026-10) and the platform skill set `greennode-agentbase-skills` to the `agentbase-build-*` skills. ✅ run for real · 🧪 tested offline/locally · 📄 guidance only · ⛔ out of scope. +Maps each item in the AgentBase docs (docs.greennode.ai/ai-stack/agent-base, 2026-10; re-read 2026-11) and the platform skill set `greennode-agentbase-skills` to the `agentbase-build-*` skills. ✅ run for real · 🧪 tested offline/locally · 📄 guidance only · ⛔ out of scope. | AgentBase item | Platform skill (operations) | Build skill (code/architecture) | Status | |---|---|---|---| @@ -15,7 +15,8 @@ Maps each item in the AgentBase docs (docs.greennode.ai/ai-stack/agent-base, 202 | Memory (events, records, strategies) | `/agentbase-memory` | `agentbase-build-memory`, `-hitl` | ✅ short/long-term, isolation, HITL on real memory | | AI Platform LLM (MaaS, API key, models) | `/agentbase-llm` | `agentbase-build-llm` (tiers, flows, fallback, routing) | ✅ 10 models, real fallback, prompt cache | | Protect & Govern — Rate limit | console | `-llm` (Rate Limit per API key / per model, MaaS 10 RPM account limit, one key per agent per env) | 📄 docs | -| Sidecar LLM Proxy (`localhost:18080`) | — | `-llm` (needs verification) | 📄 not verified | +| Sidecar LLM Proxy (`localhost:18080`) | — | `-llm` (mentioned in the MCP Gateway docs only; no path/auth/model mapping documented) | 📄 not verified | +| Built-in Tools (Code Interpreter, Browser: MCP endpoint + per-session sandboxes; whitelist-only, pricing unpublished) | — | `agentbase-build` `references/decision-guide.md` row only; no template or tested recipe | 📄 docs only | | Private Networking (VPC Peering) | `/agentbase-deploy`, `/agentbase-gateway` | `agentbase-build` `references/private-networking.md`, `-deploy`, `-mcp`, `-mcp-server`; sample `greennode-samples/sample-onprem-mcp-vpn` | 📄 docs + sample | | Team & Permissions / Service Accounts | console / `/agentbase` | `agentbase-build` `references/iam-permissions.md`, `-deploy` (CI SA, reset-service-account, orphaned SAs) | 📄 docs | | GreenNode CLI (`grn`) / GreenNode MCP | docs | `-deploy` (`grn agentbase deploy up`, pinned install, manifest rules, network/security gate — grn cannot carry them), scaffold `ci.yml` + `deploy/` | 🧪 mock `grn` + YAML render · ⏳ not run against a live account | @@ -26,7 +27,7 @@ Maps each item in the AgentBase docs (docs.greennode.ai/ai-stack/agent-base, 202 ## Platform bugs/differences found during real runs (handled in the skills) -1. Runtimes created without Security Settings (Inbound Auth `No authorization`) are public ⇒ the agent must use `jwt`/`api_key`. The platform now offers Inbound Auth IAM/JWT + IP Access Control ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)); the agent still verifies tokens itself. +1. Runtimes created without Security Settings (Inbound Auth `No authorization`) are public ⇒ the agent must use `jwt`/`api_key`. The platform now offers Inbound Auth IAM/JWT + IP Access Control ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)); the agent still verifies tokens itself. The Console now defaults a new Runtime to **IAM Permissions** (what `grn` / the API default to is not documented), and in IAM and JWT mode the platform **strips `Authorization`** before the container — the user token must also travel in `X-GreenNode-AgentBase-Custom-User-Token` (`agentbase-build-auth`). Not yet run against a live Runtime JWT; the Expo template and the BFF's `jwt` mode send only `Authorization`. 2. `greennode-agentbase` 1.0.3: `insert_memory_records_directly` needs a `MemoryRecordInsertDirectlyRequest` (the official sample passes a list ⇒ TypeError); `search` returns `list[dict]`. 3. `AgentBaseMemoryEvents`: reading state right after an interrupt may miss the `__interrupt__` write, and the bridge inserts a fake ToolMessage ⇒ take the interrupt from the graph output, treat `next=('approval',)` as pending approval. 4. SDK `IAMCredentials` falls back per field ⇒ mismatched ID/secret pair ⇒ 401 ⇒ `.greennode.json` is the only source locally. diff --git a/skills/agentbase-build/references/private-networking.md b/skills/agentbase-build/references/private-networking.md index 3652def..1fcdf0d 100644 --- a/skills/agentbase-build/references/private-networking.md +++ b/skills/agentbase-build/references/private-networking.md @@ -52,6 +52,7 @@ Agent (Runtime, Public or Private) → MCP Gateway (Private, Route CIDRs include ``` - DC firewall: allow **only** the AgentBase/VPC source CIDRs to the MCP port. The sample uses the AgentBase VPC range observed for its org; confirm yours with GreenNode. +- If a firewall, security group or database on your side filters by IP, allow the Runtime's **Outbound IP** (runtime detail page → *Outbound IP*; [private-networking](https://docs.greennode.ai/ai-stack/agent-base/private-networking)) — a Private Runtime's calls originate from it, and without it they are blocked. - The MCP server **still authenticates every call** (API key or JWT, fail-closed — `/agentbase-build-mcp-server`). A private network is not authentication. - Connector outbound auth = API key / OAuth from Access Control (`/agentbase-build-identity`); the agent never holds it. @@ -60,5 +61,5 @@ Agent (Runtime, Public or Private) → MCP Gateway (Private, Route CIDRs include 1. VPC visible in the dropdown (otherwise peering is not active yet — refresh, then contact support). 2. From inside the VPC: the target answers (`curl https:///health`). 3. Gateway: `tools/list` through the connector returns the tools; `tools/call` is allowed by the Policy Group. -4. Timeouts only for private targets ⇒ missing **Route CIDRs** or a missing route/ACL on the VPC/DC side. +4. Timeouts only for private targets ⇒ missing **Route CIDRs**, a missing route/ACL on the VPC/DC side, or a firewall that does not allow the Runtime's **Outbound IP**. 5. Name resolution fails for internal hosts ⇒ DNS resolution must be enabled in the VPC (MCP Gateway docs); prefer IPs or VPC DNS names that resolve inside the VPC. diff --git a/skills/agentbase-build/references/security.md b/skills/agentbase-build/references/security.md index 510c1e1..26a49c2 100644 --- a/skills/agentbase-build/references/security.md +++ b/skills/agentbase-build/references/security.md @@ -11,7 +11,7 @@ This file describes the template's defenses. To check that a project still has t | User spoofing via headers | JWT: the User-Id header must match `sub` (otherwise ⇒ 403). api_key: only issue keys to trusted callers | | Tools doing dangerous things | `HITL_TOOLS` + Policy Group (least privilege) + scopes on the MCP server | | Prompt injection via tool / web / document content | See below | -| Secret leakage | Credentials that tools use for external services: AgentBase Identity only. The agent's own secrets (`LLM_API_KEY`, Langfuse keys, `FEEDBACK_SIGNING_KEY`, static keys for its MCP servers / A2A peers): the runtime env file, rendered from the CI secret store (committed `.env..example` holds no secrets). Never in the image (`.dockerignore`), the repo or chat (the user enters them with `make set-secret` / `gh secret set`, below); `_mask` in Langfuse; tokens never logged | +| Secret leakage | Credentials that tools use for external services: AgentBase Identity only. The agent's own secrets (`LLM_API_KEY`, Langfuse keys, `FEEDBACK_SIGNING_KEY`, static keys for its MCP servers / A2A peers): the runtime env file, rendered from the CI secret store (committed `.env..example` holds no secrets). The Runtime docs call that field "non-sensitive config only", so these values are readable by whoever can read the runtime's versions: keep the list short and scope each key (one LLM key per agent per environment). Never in the image (`.dockerignore`), the repo or chat (the user enters them with `make set-secret` / `gh secret set`, below); `_mask` in Langfuse; tokens never logged | | Races within a session (double-submitted approve, 2 tabs) | Lock per (user, session) + `interrupt_id` ⇒ 409 | | Hangs / cascading overload (Memory, MaaS, MCP) | `REQUEST_TIMEOUT_S`, Memory timeout + retry, `MEMORY_MAX_CONCURRENCY` FIFO limiter (fair between sync and async callers), `MCP_LIST_TIMEOUT_S`, negative cache for failing MCP (`MCP_FAILURE_TTL_S`, not applied to auth/per-user errors) | | Runaway cost (loops, huge questions) | `MAX_MESSAGE_CHARS`, `MAX_TOOL_ROUNDS`, hard trim, MaaS rate limit, fallback with bounded retries | From ef7228532597a36daa8984dd2efed1554e6e95cd Mon Sep 17 00:00:00 2001 From: khanhphamj Date: Mon, 9 Nov 2026 07:08:39 +0700 Subject: [PATCH 2/4] Policy Group delete does not detach; policy/group limits not enforced as documented (verified live 2026-11-09) Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 5 ++++- skills/agentbase-build-deploy/SKILL.md | 4 ++-- skills/agentbase-build-mcp/SKILL.md | 2 +- skills/agentbase-build-mcp/references/governance.md | 4 ++-- skills/agentbase-build/references/iam-permissions.md | 2 +- 5 files changed, 10 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bc2d96..68f25c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,7 +19,10 @@ review can tell which standard a project was built from. (Discovery URL / JWKS, Principal claim), connector Header key / prefix, default scopes, M365 VNG bundle, Slack needs a Custom provider, 22 policy operators, conditions combine with AND. Also: Outbound IP of a Private Runtime, vCR `backendName` / robot username / expiry, runtime env vars "non-sensitive config only", stateless contract, ephemeral - filesystem (vStorage S3), Policy Group deletion detaches from all gateways, and a Built-in Tools row in the decision guide. + filesystem (vStorage S3), a Built-in Tools row in the decision guide. **Verified live (2026-11-09, temporary resources):** deleting a Policy Group does + **not** detach it (the gateway keeps the dangling `policyGroupId`; unbind first with `PATCH` `{"policyGroupId": null}`, `application/merge-patch+json`), and the documented + policy-group limits (10/20 per group, 20 groups) were not enforced on that account (100 policies accepted, 429 `errors.quota.policy` at 101; 43 groups) — the skills say + to rely on that error, not a number. A gateway has its own identity `gw-` (removed with it), a runtime `runtime-`. - **Deploy: find out how the customer can build before building.** `agentbase-build-deploy` Step 3 now starts with `scripts/detect_build_env.sh` (read-only; no login, no secret printed): which engine is installed and answering (Docker Desktop, Rancher Desktop, OrbStack, Colima, Podman, nerdctl, Finch, buildah), `linux/amd64` support, git host and CI config diff --git a/skills/agentbase-build-deploy/SKILL.md b/skills/agentbase-build-deploy/SKILL.md index 273a762..07a0fb4 100644 --- a/skills/agentbase-build-deploy/SKILL.md +++ b/skills/agentbase-build-deploy/SKILL.md @@ -186,8 +186,8 @@ Runtimes are billed continuously (a `STOPPED` runtime keeps its config without c 1. Runtime(s) incl. custom endpoints — `/agentbase-teardown` or `/agentbase-deploy`. 2. Memory store(s), identity + Access Control providers, LLM API keys, vCR images — `/agentbase-teardown`. -3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-` service account). -4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible and **detaches it from every gateway at once**; a gateway without a Policy Group answers 403 to every `tools/call` ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)) — detach and re-bind (or delete the gateway) deliberately. +3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-` service account; its own Access Control identity `gw-` — field `agentIdentityName` — is removed with the gateway, verified live 2026-11-09). A runtime likewise gets an identity `runtime-` right after creation. +4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible. The docs say it detaches the group from every gateway at once ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)); **verified live (2026-11-09) the API does not**: `DELETE /policy/api/v1/policy-groups/{id}` returns 200 but the gateway keeps the dangling `policyGroupId` (unchanged after 60 s, state stays ACTIVE). **Unbind first**: `PATCH /gateway/api/v1/gateways/{name}` with `Content-Type: application/merge-patch+json` and `{"policyGroupId": null}` → 202, state UPDATING → ACTIVE, `policyGroupId` gone. What a gateway enforces with a dangling id was not tested — do not assume 403 or allow. (A gateway without a Policy Group answers 403 to every `tools/call`.) Binding an id that does not exist is rejected with 400 `policy_group_invalid`. 5. MCP Connectors (catalog/custom) — Console. 6. The **Orphaned** runtime service account (`auto-sa-`) and gateway service accounts — revoke in IAM. 7. Inbound API keys: remove their hashes from `AUTH_API_KEY_SHA256` of the callers' environments, and the A2A keys peers hold for this agent. diff --git a/skills/agentbase-build-mcp/SKILL.md b/skills/agentbase-build-mcp/SKILL.md index ac92b2b..5c5d5e9 100644 --- a/skills/agentbase-build-mcp/SKILL.md +++ b/skills/agentbase-build-mcp/SKILL.md @@ -72,7 +72,7 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou - per principal, **ALLOW lists of exact actions** (least privilege) — order-independent; - **never a catch-all DENY after ALLOWs** (ALLOW `x` then DENY `*`): first-match allows `x`, deny-wins blocks everything. It is redundant anyway (no match ⇒ DENY); - a DENY that carves out of a broad ALLOW goes **first** (lowest `order`) — then both semantics agree. - - Limits: **plan for 10 policies per group** (`/agentbase-policy`: max 10 per group, 20 groups per user; the Console docs say 20 per group); group name 5–50 chars `[A-Za-z0-9_]` starting with a letter; 1 gateway ↔ at most 1 group (attaching another replaces it); changes apply within ~30s. Match-all principals ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups) *Principal and Wildcard Rules*): bare **`iam`** = all IAM identities, bare **`jwt`** (or `jwt:*`) = all JWT users; Console principal *All* = everyone. `iam:*` is not in the official table (`/agentbase-policy` uses it) — prefer bare `iam`. `jwt:abc*` is **not** a prefix match, it is a literal id. + - Limits: **keep policies per group few** for review reasons (`/agentbase-policy` says max 10 per group and 20 groups per user, the Console docs say 20 per group; verified live 2026-11-09: 100 policies in one group accepted, 429 `errors.quota.policy` at 101, and at least 43 groups existed at once — the documented limits were not enforced on that account, quotas can differ per account, so check for `errors.quota.policy` instead of relying on a number); group name 5–50 chars `[A-Za-z0-9_]` starting with a letter; 1 gateway ↔ at most 1 group (attaching another replaces it); changes apply within ~30s. Match-all principals ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups) *Principal and Wildcard Rules*): bare **`iam`** = all IAM identities, bare **`jwt`** (or `jwt:*`) = all JWT users; Console principal *All* = everyone. `iam:*` is not in the official table (`/agentbase-policy` uses it) — prefer bare `iam`. `jwt:abc*` is **not** a prefix match, it is a literal id. - Conditions: operators and keys also differ between the two sources — use `references/governance.md` *Common conditions* (examples to verify with `GET /policy/api/v1/policies/condition-operators`). 5. **Declare in the agent** — `mcp_servers.json`, **one entry per connector**, URL = `connectUrl`: diff --git a/skills/agentbase-build-mcp/references/governance.md b/skills/agentbase-build-mcp/references/governance.md index a37c7e0..b796697 100644 --- a/skills/agentbase-build-mcp/references/governance.md +++ b/skills/agentbase-build-mcp/references/governance.md @@ -6,7 +6,7 @@ |---|---| | Gateway | 1 gateway / environment / domain group (e.g. `search-gw-prod`, `corp-gw-prod`). Never share a gateway between dev & prod | | Connector | 1 connector / upstream (`tavily`, `github`, `hrmcp`). Short, stable names matching `[a-z0-9]{3,50}` — the name appears in `connectUrl` **and** in policy actions `__`, which accept only letters, digits and `_`; gateway target names allow `[a-z0-9-]` ⇒ a dash (`hr-mcp`) makes per-tool actions impossible (verify once on the real API) | -| Policy Group | 1 group / gateway; each agent (principal) gets 1 ALLOW policy listing exactly the actions it needs (least privilege). **Plan for ≤ 10 policies per group** (`/agentbase-policy` limit; the Console docs say 20) ⇒ more agents ⇒ more gateways, or group agents that need the same actions | +| Policy Group | 1 group / gateway; each agent (principal) gets 1 ALLOW policy listing exactly the actions it needs (least privilege). **Keep policies per group few** (review effort, not a hard limit: `/agentbase-policy` says 10, the Console docs say 20; live 2026-11-09 one account accepted 100 per group and answered 429 `errors.quota.policy` at the 101st — quotas differ per account, so rely on that error, not on a number) ⇒ more agents ⇒ more gateways, or group agents that need the same actions | | Policy design | Order-independent (the Console docs say first match wins, `/agentbase-policy` says deny wins — see SKILL step 4): ALLOW lists only; no catch-all DENY (no match is already DENY); a carve-out DENY goes first | | Principal | Agent on Runtime: `iam:` of the runtime service account. User via JWT: `jwt:` or a `principal.` condition. Match-all (official *Principal and Wildcard Rules*): bare `iam` = every IAM identity, bare `jwt` / `jwt:*` = every JWT user, *All* = everyone; `iam:*` is not in the official table (prefer `iam`), `jwt:abc*` is a literal id | | Connector outbound auth | Prefer API Key / OAuth from a Secret Provider. **Inbound forward** re-sends the agent's gateway credential to the MCP server: never on an IAM-inbound gateway (that is the agent's platform IAM token); on a JWT-inbound gateway only towards your own server validating the same IdP | @@ -54,7 +54,7 @@ Multiple conditions in one policy are combined with **AND** (all must be true fo - [ ] Each prod agent has its own principal (dedicated runtime service account) and only the actions it needs. - [ ] Side-effect tools are both restricted by Policy and go through `HITL_TOOLS`; on OAuth **3LO** connectors (consent of whoever clicked *Authorize*, shared by the project) every side-effect tool is in `HITL_TOOLS` and the OAuth scopes are minimal. - [ ] Connector names match `[a-z0-9]{3,50}`; each `allow_tools` list equals the Policy actions of that connector for this agent. -- [ ] Policies give the same result under first-match and deny-wins (ALLOW lists, no catch-all DENY, carve-out DENY first); ≤ 10 per group. +- [ ] Policies give the same result under first-match and deny-wins (ALLOW lists, no catch-all DENY, carve-out DENY first); few per group (documented 10–20, see above). - [ ] Upstream secrets live in Identity (Managed/Custom provider), rotated periodically. - [ ] Traces show `mcp.policy_denied` = 0 on the main flow (if > 0 ⇒ missing policy or the prompt calls the wrong tool). - [ ] Inbound NONE only in labs. diff --git a/skills/agentbase-build/references/iam-permissions.md b/skills/agentbase-build/references/iam-permissions.md index a63a917..c42952e 100644 --- a/skills/agentbase-build/references/iam-permissions.md +++ b/skills/agentbase-build/references/iam-permissions.md @@ -18,7 +18,7 @@ Only the three `*FullAccess` managed policies are documented for AgentBase. For ## Teams - **One developer service account per person** — never a shared one: audit logs and revocation stay per person (someone leaves ⇒ revoke one account), and the Memory API's limit of 10 concurrent requests applies **per account**, so a shared account makes developers (and eval runs) throttle each other. -- **Separate dev and prod MCP Gateways**, each with its own Policy Group (one group per gateway; plan for 10 policies per group — `/agentbase-policy` — although the MCP docs mention 20): dev policies can then allow developer principals without touching prod. +- **Separate dev and prod MCP Gateways**, each with its own Policy Group (one group per gateway; keep policies per group few — `/agentbase-policy` says 10, the MCP docs 20, one account accepted 100 on 2026-11-09; check `errors.quota.policy`): dev policies can then allow developer principals without touching prod. - Per environment: its own memory store, LLM API key (Rate Limit), runtime (its own `auto-sa-…` principal) and GitHub Environment secrets. ## Team roles (summary of the matrix) From 9afbef85a7d3d0c75c329dae3b9e0bcad13e2232 Mon Sep 17 00:00:00 2001 From: khanhphamj Date: Mon, 9 Nov 2026 07:09:36 +0700 Subject: [PATCH 3/4] Policy limits: last two stale '10 per group' spots in agentbase-build Co-Authored-By: Claude Sonnet 5.5 --- skills/agentbase-build/SKILL.md | 2 +- skills/agentbase-build/references/checklist.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/skills/agentbase-build/SKILL.md b/skills/agentbase-build/SKILL.md index 23ed6a9..cb25018 100644 --- a/skills/agentbase-build/SKILL.md +++ b/skills/agentbase-build/SKILL.md @@ -100,7 +100,7 @@ Show progress as `Step X/9`. Each step checks whether it is already done (idempo 2. **Scaffold** — `/agentbase-build-scaffold`: create `src/backend` (+ `--with-frontend` for a mobile UI, `--with-web` for a web UI) via the script, `uv sync`, `make test` must be green. "Off" components stay in the code and are **disabled via env** — never delete their modules (they import each other; only `app/identity.py` may go). Then the user runs `make set-secret` in their own terminal (Enter = the required keys: IAM `client_id`/`client_secret`, then `LLM_API_KEY` if they already have one — see *Credentials*) and `make check-creds` shows OK for IAM. 3. **LLM** — `/agentbase-build-llm`: get an API key via `/agentbase-llm` run from `src/backend` (its `api-keys create` / `--save-env` writes `LLM_API_KEY` into `./.env` itself, never shown; an existing key: the user runs `make set-secret NAME=LLM_API_KEY` in their terminal; `make check-creds` must now show OK for LLM); the user picks a model for each tier (reasoning/large/small) + fallback; map flows → tiers; decide on adaptive routing. 4. **Memory** — `/agentbase-build-memory`: create a memory store via `/agentbase-memory`, set `MEMORY_ID`, tune strategy/namespace and the context-compression threshold. -5. **Tools / MCP / A2A** — if a new MCP server is needed for an internal system: `/agentbase-build-mcp-server` (deploy it first). If another agent is needed (decision guide §5): `/agentbase-build-a2a`. Then `/agentbase-build-mcp`: write local tools; create/select an MCP Gateway (`/agentbase-gateway` — separate dev and prod gateways), connect MCP Connectors (catalog or custom), create a Policy Group (`/agentbase-policy`) with an ALLOW for the **developer principal** (`make check-creds`) for local runs — each environment's **runtime principal** is added after its first deploy (Step 9b) — and declare each `connectUrl` in `mcp_servers.json`. Plan for **10 policies per group** (`/agentbase-policy`; the MCP docs say 20) and one group per gateway. Mark side-effect tools in `HITL_TOOLS` (`/agentbase-build-hitl`). +5. **Tools / MCP / A2A** — if a new MCP server is needed for an internal system: `/agentbase-build-mcp-server` (deploy it first). If another agent is needed (decision guide §5): `/agentbase-build-a2a`. Then `/agentbase-build-mcp`: write local tools; create/select an MCP Gateway (`/agentbase-gateway` — separate dev and prod gateways), connect MCP Connectors (catalog or custom), create a Policy Group (`/agentbase-policy`) with an ALLOW for the **developer principal** (`make check-creds`) for local runs — each environment's **runtime principal** is added after its first deploy (Step 9b) — and declare each `connectUrl` in `mcp_servers.json`. Keep policies per group **few** (review effort; `/agentbase-policy` says 10, the MCP docs 20, one account accepted 100 and answered 429 `errors.quota.policy` at 101 on 2026-11-09 — check that error, not a number) and one group per gateway. Mark side-effect tools in `HITL_TOOLS` (`/agentbase-build-hitl`). 6. **Auth** — `/agentbase-build-auth`: configure inbound JWT (JWKS/issuer/audience). Tools calling external services with a key or OAuth ⇒ `/agentbase-build-identity` (providers via `/agentbase-identity`, helpers in `app/identity.py`). 7. **Tracing + Eval** — install the official Langfuse skill (`npx skills add langfuse/skills --skill langfuse`) next to this set, then `/agentbase-build-tracing` (Langfuse v4 keys — the user runs `make set-secret NAME=langfuse` first —, prompt management, model prices; run a few requests and `lf_trace.py --latest 3 --audit` until it reports no FAIL) and `/agentbase-build-eval` (write the dataset from the spec, `make eval` meets the threshold). 8. **Frontend** (if any) — mobile `/agentbase-build-frontend`, web `/agentbase-build-web`. diff --git a/skills/agentbase-build/references/checklist.md b/skills/agentbase-build/references/checklist.md index a133bd3..76dd1fb 100644 --- a/skills/agentbase-build/references/checklist.md +++ b/skills/agentbase-build/references/checklist.md @@ -30,7 +30,7 @@ Run each item and report `PASS` / `FAIL` / `N/A` with evidence (command + short - [B] Side-effect tools are in `HITL_TOOLS` (or the user confirmed in writing that it is not needed). - [B] Each local side-effect tool passes `idempotency_key(config, tool_call_id)` to the system it changes (or that system dedupes on a business key) — HITL alone does not stop a double run across replicas (`/agentbase-build-hitl`). - Prod MCP goes through the MCP Gateway, inbound `IAM` or `JWT`; secrets live in Identity (Secret Provider). -- [B] The Gateway has a **Policy Group** attached (none attached ⇒ every tools/call returns 403); there is an ALLOW policy with the exact action `__` for each principal that calls it: the developer's `iam:` (`make check-creds`, local runs) and **each environment's runtime** `iam:` (runtime logs `IAM principal …`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured — `/agentbase-build` Step 9b). Dev and prod use separate gateways/groups (≤ 10 policies per group). +- [B] The Gateway has a **Policy Group** attached (none attached ⇒ every tools/call returns 403); there is an ALLOW policy with the exact action `__` for each principal that calls it: the developer's `iam:` (`make check-creds`, local runs) and **each environment's runtime** `iam:` (runtime logs `IAM principal …`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured — `/agentbase-build` Step 9b). Dev and prod use separate gateways/groups (few policies per group — documented 10–20, observed 100 accepted on 2026-11-09; check `errors.quota.policy`). - [B] `mcp_servers.json` points to **each connector's connectUrl** (`/`), not the gateway root, and every entry declares `auth` explicitly. - [B] `HITL_TOOLS` patterns all match a tool (`tools.collect.output.hitl_unmatched` is empty). - MCP servers in a VPC / data center go through a **Private** MCP Gateway (`references/private-networking.md`); they still authenticate every call. From 09b67c5e3ab2f55232470ac586765e2075d76de2 Mon Sep 17 00:00:00 2001 From: khanhphamj Date: Mon, 9 Nov 2026 07:18:34 +0700 Subject: [PATCH 4/4] Gateway without a Policy Group is fail-open; deny shape, deny-wins, principals (verified live 2026-11-09) Co-Authored-By: Claude Sonnet 5.5 --- CHANGELOG.md | 7 +++++++ skills/agentbase-build-deploy/SKILL.md | 2 +- skills/agentbase-build-mcp/SKILL.md | 14 +++++++------- .../agentbase-build-mcp/references/governance.md | 10 +++++----- skills/agentbase-build/SKILL.md | 2 +- skills/agentbase-build/references/checklist.md | 2 +- .../references/platform-coverage.md | 2 +- 7 files changed, 23 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 68f25c0..c3dac91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,13 @@ review can tell which standard a project was built from. **not** detach it (the gateway keeps the dangling `policyGroupId`; unbind first with `PATCH` `{"policyGroupId": null}`, `application/merge-patch+json`), and the documented policy-group limits (10/20 per group, 20 groups) were not enforced on that account (100 policies accepted, 429 `errors.quota.policy` at 101; 43 groups) — the skills say to rely on that error, not a number. A gateway has its own identity `gw-` (removed with it), a runtime `runtime-`. + **Second live lab (2026-11-09, one account, one gateway, IAM inbound):** a gateway with **no Policy Group bound let every + valid IAM caller's `tools/call` through** (the skills said 403; an earlier 2026-10 run saw 403) — now a security rule: bind + a group before callers can reach the gateway; a bound group with zero policies, or a dangling id, denies; a deny arrived + as HTTP 200 with an MCP tool error `Request denied by policy.` (no 403 seen); deny wins over allow in any order (the + Console docs say first match); bare `iam` / `jwt` principals are rejected by the API with 400 (use `iam:*`, `jwt:*`, `*`); + `iam:` denies; policy changes apply in seconds, binding via `PATCH` after UPDATING → ACTIVE; targets are called + at `/`. Not tested: JWT-inbound gateways, non-owner callers, conditions, a real Runtime SA. - **Deploy: find out how the customer can build before building.** `agentbase-build-deploy` Step 3 now starts with `scripts/detect_build_env.sh` (read-only; no login, no secret printed): which engine is installed and answering (Docker Desktop, Rancher Desktop, OrbStack, Colima, Podman, nerdctl, Finch, buildah), `linux/amd64` support, git host and CI config diff --git a/skills/agentbase-build-deploy/SKILL.md b/skills/agentbase-build-deploy/SKILL.md index 07a0fb4..141fe29 100644 --- a/skills/agentbase-build-deploy/SKILL.md +++ b/skills/agentbase-build-deploy/SKILL.md @@ -187,7 +187,7 @@ Runtimes are billed continuously (a `STOPPED` runtime keeps its config without c 1. Runtime(s) incl. custom endpoints — `/agentbase-teardown` or `/agentbase-deploy`. 2. Memory store(s), identity + Access Control providers, LLM API keys, vCR images — `/agentbase-teardown`. 3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-` service account; its own Access Control identity `gw-` — field `agentIdentityName` — is removed with the gateway, verified live 2026-11-09). A runtime likewise gets an identity `runtime-` right after creation. -4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible. The docs say it detaches the group from every gateway at once ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)); **verified live (2026-11-09) the API does not**: `DELETE /policy/api/v1/policy-groups/{id}` returns 200 but the gateway keeps the dangling `policyGroupId` (unchanged after 60 s, state stays ACTIVE). **Unbind first**: `PATCH /gateway/api/v1/gateways/{name}` with `Content-Type: application/merge-patch+json` and `{"policyGroupId": null}` → 202, state UPDATING → ACTIVE, `policyGroupId` gone. What a gateway enforces with a dangling id was not tested — do not assume 403 or allow. (A gateway without a Policy Group answers 403 to every `tools/call`.) Binding an id that does not exist is rejected with 400 `policy_group_invalid`. +4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible. The docs say it detaches the group from every gateway at once ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)); **verified live (2026-11-09) the API does not**: `DELETE /policy/api/v1/policy-groups/{id}` returns 200 but the gateway keeps the dangling `policyGroupId` (unchanged after 60 s, state stays ACTIVE). **Unbind first**: `PATCH /gateway/api/v1/gateways/{name}` with `Content-Type: application/merge-patch+json` and `{"policyGroupId": null}` → 202, state UPDATING → ACTIVE, `policyGroupId` gone. With the dangling id the gateway denied every `tools/call` (still after 30 s; fail closed). A gateway with **no** group bound does not deny: verified live 2026-11-09 it let every valid IAM caller's `tools/call` through — so unbind-then-delete leaves the gateway open: delete the gateway too, or bind a replacement group at once. Binding an id that does not exist is rejected with 400 `policy_group_invalid`. 5. MCP Connectors (catalog/custom) — Console. 6. The **Orphaned** runtime service account (`auto-sa-`) and gateway service accounts — revoke in IAM. 7. Inbound API keys: remove their hashes from `AUTH_API_KEY_SHA256` of the callers' environments, and the A2A keys peers hold for this agent. diff --git a/skills/agentbase-build-mcp/SKILL.md b/skills/agentbase-build-mcp/SKILL.md index 5c5d5e9..25079c3 100644 --- a/skills/agentbase-build-mcp/SKILL.md +++ b/skills/agentbase-build-mcp/SKILL.md @@ -16,7 +16,7 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou ├─ /tavily ← MCP Connector "tavily" (outbound API Key 2LO, providerName in Identity) ├─ /stock ← MCP Connector "stock" └─ /github ← MCP Connector "github" (outbound OAuth 3LO) - │ Policy Group bound to the gateway: tools/call → ALLOW | DENY / no match ⇒ HTTP 403 + │ Policy Group bound to the gateway: tools/call → ALLOW | DENY / no match ⇒ denied (live: HTTP 200 + MCP tool error) │ ("No policy allows this request"; also seen as MCP error "Request denied by policy") ▼ MCP Server → Provider (Tavily, GitHub, …) @@ -26,7 +26,7 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou |---|---|---| | **MCP Gateway** | Proxy (Kong) — inbound auth + Policy enforcement | 1 gateway binds at most 1 Policy Group | | **MCP Connector** | A **target** of the gateway; its URL is auto-generated from the gateway (observed `/`) | Copy the connector's **Endpoint** from the Console *Connected* tab (or `connectUrl` from the API) exactly — some include a `/mcp` suffix. The agent points at **each** connector URL; the gateway root ⇒ error | -| **Policy Group** | Set of ALLOW/DENY rules for `tools/call` | **Not bound ⇒ every tools/call is 403**; `tools/list` is always allowed | +| **Policy Group** | Set of ALLOW/DENY rules for `tools/call` | **Not bound ⇒ fail-OPEN**: `tools/call` was allowed for any valid IAM caller (verified live 2026-11-09, one account, one gateway; a 2026-10 run saw 403 — do not rely on a 403). Bind a group before any caller can reach the gateway. A bound group with zero policies, or whose group was deleted (dangling id), denies everything; `tools/list` is always allowed | ## Choosing the tool type @@ -58,7 +58,7 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou **Inbound forward** sends the MCP server the **same credential the agent used on the gateway** ([connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector)). With gateway inbound **IAM** that is the agent's platform IAM token (runtime service account; locally your developer service account with `AgentBaseFullAccess`) — the server could call AgentBase APIs as your agent ⇒ **never use Inbound forward with IAM inbound**. With inbound **JWT** it forwards the end user's JWT ⇒ only towards **your own** MCP server that validates the same IdP (`/agentbase-build-mcp-server`), never a third-party server. List connectors + `connectUrl` via API (read): `GET https://agentbase.api.vngcloud.vn/gateway/api/v1/mcp-connectors?page=1&pageSize=50` (IAM token via `get_token.sh`). Item: `name`, `template.id` (`custom`, …), `connectUrl`, `outboundAuth{type, flow, provider}`, `gateway{name, endpoint, inboundAuth, state}`. -4. **Policy Group** — `/agentbase-policy` (REQUIRED, otherwise the agent gets 403 for every tool): +4. **Policy Group** — `/agentbase-policy` (REQUIRED: bind it before any caller can reach the gateway — with no group bound, every valid IAM caller could call every tool (verified live 2026-11-09, one account, one gateway); with a group but no matching ALLOW the agent's tool calls are denied). Policy changes applied within ~2–3 s; binding via `PATCH` needs the gateway to go UPDATING → ACTIVE (~10–30 s) first: - `principal`: **`iam:`** where `sub` = the `sub` claim in the agent's IAM token (the service account's user id — **NOT the client_id**). Locally: `make check-creds` prints it (`principal iam:…`). The agent also logs it at startup **when an enabled `auth: iam` server is configured** (`log_iam_principal()` in the lifespan, line `IAM principal of the agent (use in Policy Group): iam:…`; it never blocks startup). On Runtime the service account is platform-managed ⇒ deploy with the `auth: iam` server enabled, read the startup logs (`/agentbase-monitor`), then add the policy. - `actions`: **exactly** `"__"` (2 underscores) or `"*"`. Partial wildcards are not supported (`tavily__*` ❌). - `resources`: `["gateway:"]`. @@ -68,7 +68,7 @@ MCP Gateway https://gw--.agentbase-gateway.aiplatform.vngclou "actions": ["tavily__tavily_search", "tavily__tavily_extract", "stock__stock_quote"], "resources": ["gateway:sample-mcp-gw"]} ``` - - **Evaluation semantics — the sources disagree.** The Console docs ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups)) say policies run **top-to-bottom, the first match decides**; `/agentbase-policy` (API reference) says **any matching `deny` wins**. Both agree: no match ⇒ DENY, inactive policies are skipped. Until verified on your gateway, write policies whose result is the **same under both**: + - **Evaluation semantics — the sources disagree, the gateway followed one.** The Console docs ([policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups)) say policies run **top-to-bottom, the first match decides**; `/agentbase-policy` (API reference) says **any matching `deny` wins**. **Deny-wins was observed** (verified live 2026-11-09, one account, one gateway): allow-then-deny and deny-then-allow for the same principal and action both ⇒ DENY, removing the deny ⇒ ALLOW. No match ⇒ DENY, inactive policies are skipped. Still write policies whose result is the **same under both** (cheap insurance, and another gateway or a later change may differ): - per principal, **ALLOW lists of exact actions** (least privilege) — order-independent; - **never a catch-all DENY after ALLOWs** (ALLOW `x` then DENY `*`): first-match allows `x`, deny-wins blocks everything. It is redundant anyway (no match ⇒ DENY); - a DENY that carves out of a broad ALLOW goes **first** (lowest `order`) — then both semantics agree. @@ -123,7 +123,7 @@ Console *AgentBase → MCP Connectors* ([browse-connector-catalog](https://docs. ## Runtime behavior (asset `app/tools/mcp.py`) -- **Policy-deny guard** (`_guard_policy()`): a deny reaches the agent either as **HTTP 403** on `tools/call` (the documented gateway behavior, `No policy allows this request` — the adapter raises `ExceptionGroup[httpx.HTTPStatusError]` and the body never reaches the exception) or as MCP error `Request denied by policy.` (isError ⇒ `ToolException`) ⇒ either way the LLM assumes a transient error and **retries many times** (observed: 3 times). The guard walks nested exception groups and treats a 403 status or a deny marker in the error text as a deny ⇒ `POLICY_DENIED: …` (telling the LLM not to retry) + trace event `mcp.policy_denied` (WARNING, `scope=tool|connector`). With the guard: 1 call. Successful results are never scanned for deny text (prompt-injection safe). Any 403 on `tools/call` counts — a self-hosted server answering 403 is treated the same way, **except** a 403 whose `WWW-Authenticate` carries `error="insufficient_scope"` (the MCP server rejecting the token's scopes, RFC 6750) ⇒ `MCP_SCOPE_ERROR` + event `mcp.scope_error`, not a deny. +- **Policy-deny guard** (`_guard_policy()`): a deny reaches the agent as an MCP tool error — **HTTP 200 with a JSON-RPC result `{"content":[{"type":"text","text":"Request denied by policy."}],"isError":true}`** (verified live 2026-11-09, one account, one gateway; no HTTP 403 seen) ⇒ `ToolException` — or, as seen in earlier runs and in the docs, as **HTTP 403** on `tools/call` (`No policy allows this request` — the adapter raises `ExceptionGroup[httpx.HTTPStatusError]` and the body never reaches the exception) ⇒ either way the LLM assumes a transient error and **retries many times** (observed: 3 times). The guard walks nested exception groups and treats a 403 status or a deny marker in the error text as a deny ⇒ `POLICY_DENIED: …` (telling the LLM not to retry) + trace event `mcp.policy_denied` (WARNING, `scope=tool|connector`). With the guard: 1 call. Successful results are never scanned for deny text (prompt-injection safe). Any 403 on `tools/call` counts — a self-hosted server answering 403 is treated the same way, **except** a 403 whose `WWW-Authenticate` carries `error="insufficient_scope"` (the MCP server rejecting the token's scopes, RFC 6750) ⇒ `MCP_SCOPE_ERROR` + event `mcp.scope_error`, not a deny. - **Denials are per tool**: a denied tool is short-circuited for the rest of the turn (no request to the gateway, event `short_circuit=true`), while the connector's other — possibly ALLOWED — tools keep working. Once **2 distinct tools** of a connector are denied in one turn (or at the first deny with `"policy_all_or_nothing": true`), all its tools are short-circuited (a reasoning model was observed trying 8 denied tools in turn, 84s). `reset_policy_denials()` runs per request. - **Other tool failures** (HTTP 5xx, timeouts, connection errors) ⇒ the tool returns `MCP_ERROR (): tool '…' failed — HTTP 502 …; retry at most once` + event `mcp.tool_error`. The raw exception never reaches the LLM: ToolNode used to put `HTTPStatusError(… url '…?tavilyApiKey=SECRET')` + "Please fix your mistakes" into the LLM context, the checkpoint and the traces. An MCP server's own isError text passes through unchanged (it is written for the LLM). - Tool lists of `iam`/`none` servers are cached for 5 minutes; `user_jwt` is fetched per request (the user's JWT is never shared across users). @@ -137,7 +137,7 @@ Console *AgentBase → MCP Connectors* ([browse-connector-catalog](https://docs. | Symptom | Cause | Fix | |---|---|---| | `401` at `iam.api.vngcloud.vn/.../auth/token` | Mismatched ID/secret pair: the shell has `GREENNODE_CLIENT_ID` of another account while the secret comes from `.greennode.json` (SDK falls back per field) | `env \| grep GREENNODE_`; locally use only `.greennode.json`; `normalize_iam_env()` drops stray variables | -| `Request denied by policy` / 403 `No policy allows this request` / trace `mcp.policy_denied` | Gateway has no Policy Group bound, or principal/action mismatch | Bind a PG; principal = `iam:` (`make check-creds` / startup log), exact action `__`; connector name without dash | +| `Request denied by policy` / 403 `No policy allows this request` / trace `mcp.policy_denied` | Principal/action mismatch, a bound group with no matching ALLOW (or no policies), or a dangling group id. (No group bound is **not** a deny — it let calls through, see step 4) | Bind a PG; principal = `iam:` (`make check-creds` / startup log), exact action `__`; connector name without dash | | `MCP_SCOPE_ERROR` / trace `mcp.scope_error` (403 `insufficient_scope`) | The **MCP server** refused the connector's token: OAuth scopes of the connector (or `MCP_API_KEY_SCOPES`) lack what the tool's `require_scope()` / `MCP_REQUIRED_SCOPES` needs | Add the scope to the connector's OAuth scopes / the server's API-key scopes — not a Policy Group change | | `MCP_ERROR ()` / trace `mcp.tool_error` | Upstream 5xx, timeout, connection error on `tools/call` | Server logs (`log.warning` has the masked URL); the LLM retries at most once | | `MCP server 'x' skipped — … must be https://` | `auth: iam`/`user_jwt` with an `http://` URL (credential in clear text) | Use the `https://` connectUrl; `http://` only for localhost | @@ -166,6 +166,6 @@ Governance details & per-agent policy design examples: `references/governance.md - [manage-mcp-gateway](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/mcp-gateway/manage-mcp-gateway) — gateway form fields and limits - [mcp-connectors](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors) — catalog / custom connectors - [connect-a-connector](https://docs.greennode.ai/ai-stack/agent-base/mcp-connectors/connect-a-connector) — outbound auth: OAuth 2LO/3LO, API Key, Inbound forward, No auth; Managed vs Custom secret -- [policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups) — principal/action formats, conditions, first-match evaluation +- [policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups) — principal/action formats, conditions, evaluation order (docs: first match; observed: deny wins) - [manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups) — limits, attach/detach behavior - [private-networking](https://docs.greennode.ai/ai-stack/agent-base/private-networking) — Private MCP Gateway for MCP servers in your VPC / data center diff --git a/skills/agentbase-build-mcp/references/governance.md b/skills/agentbase-build-mcp/references/governance.md index b796697..5bc7387 100644 --- a/skills/agentbase-build-mcp/references/governance.md +++ b/skills/agentbase-build-mcp/references/governance.md @@ -7,8 +7,8 @@ | Gateway | 1 gateway / environment / domain group (e.g. `search-gw-prod`, `corp-gw-prod`). Never share a gateway between dev & prod | | Connector | 1 connector / upstream (`tavily`, `github`, `hrmcp`). Short, stable names matching `[a-z0-9]{3,50}` — the name appears in `connectUrl` **and** in policy actions `__`, which accept only letters, digits and `_`; gateway target names allow `[a-z0-9-]` ⇒ a dash (`hr-mcp`) makes per-tool actions impossible (verify once on the real API) | | Policy Group | 1 group / gateway; each agent (principal) gets 1 ALLOW policy listing exactly the actions it needs (least privilege). **Keep policies per group few** (review effort, not a hard limit: `/agentbase-policy` says 10, the Console docs say 20; live 2026-11-09 one account accepted 100 per group and answered 429 `errors.quota.policy` at the 101st — quotas differ per account, so rely on that error, not on a number) ⇒ more agents ⇒ more gateways, or group agents that need the same actions | -| Policy design | Order-independent (the Console docs say first match wins, `/agentbase-policy` says deny wins — see SKILL step 4): ALLOW lists only; no catch-all DENY (no match is already DENY); a carve-out DENY goes first | -| Principal | Agent on Runtime: `iam:` of the runtime service account. User via JWT: `jwt:` or a `principal.` condition. Match-all (official *Principal and Wildcard Rules*): bare `iam` = every IAM identity, bare `jwt` / `jwt:*` = every JWT user, *All* = everyone; `iam:*` is not in the official table (prefer `iam`), `jwt:abc*` is a literal id | +| Policy design | Order-independent (the Console docs say first match wins, `/agentbase-policy` says deny wins, deny-wins was observed live 2026-11-09 — see SKILL step 4): ALLOW lists only; no catch-all DENY (no match is already DENY); a carve-out DENY goes first | +| Principal | Agent on Runtime: `iam:` of the runtime service account. User via JWT: `jwt:` or a `principal.` condition. Match-all: the official *Principal and Wildcard Rules* list bare `iam` / bare `jwt`, but **the API rejected both with 400 `errors.bad_request` at policy creation** (verified live 2026-11-09) — use `iam:*` (allowed), `jwt:*` (accepted at creation; not exercised) or `*` (allowed); *All* in the Console = everyone. `iam:` ⇒ DENY: the principal is the `sub`. `jwt:abc*` is a literal id | | Connector outbound auth | Prefer API Key / OAuth from a Secret Provider. **Inbound forward** re-sends the agent's gateway credential to the MCP server: never on an IAM-inbound gateway (that is the agent's platform IAM token); on a JWT-inbound gateway only towards your own server validating the same IdP | ## Example (real observed structure, IDs replaced with placeholders) @@ -30,7 +30,7 @@ Gateway `sample-mcp-gw` (inbound IAM) has 2 connectors `tavily`, `stock` (outbou ] ``` -E2E result with the agent running as ``: `stock_stock_quote` ⇒ real data; `tavily_tavily_search` ⇒ `Request denied by policy` ⇒ guard returns `POLICY_DENIED`, agent does not retry (denials are per tool: the `stock_*` tools keep working in the same turn). The documented deny shape — HTTP 403 `No policy allows this request` (no rule matches, or no Policy Group bound) — gets the same treatment: the guard recognizes the 403 status inside the adapter's exception group (covered offline by a fake gateway in `tests/test_mcp.py`). +E2E result with the agent running as ``: `stock_stock_quote` ⇒ real data; `tavily_tavily_search` ⇒ `Request denied by policy` ⇒ guard returns `POLICY_DENIED`, agent does not retry (denials are per tool: the `stock_*` tools keep working in the same turn). Live 2026-11-09 (one account, one gateway) the deny was HTTP 200 with an MCP tool result `isError: true`, text `Request denied by policy.` — HTTP 403 was not observed; a bound group with zero policies and a bound-but-deleted group also denied, and **no group bound allowed the call**. The documented / earlier-run deny shape — HTTP 403 `No policy allows this request` — gets the same treatment: the guard recognizes the 403 status inside the adapter's exception group (covered offline by a fake gateway in `tests/test_mcp.py`). ## Common conditions (examples — verify before use) @@ -48,13 +48,13 @@ Multiple conditions in one policy are combined with **AND** (all must be true fo ## Governance checklist before prod -- [ ] Prod gateway has a Policy Group; no ALLOW policy with a match-all principal (*All* / `"*"`, `iam`, `jwt`, `jwt:*`) + `actions: ["*"]`. +- [ ] Prod gateway has a Policy Group **bound before any caller can reach it** (none bound = every valid IAM caller can call every tool, verified live 2026-11-09); no ALLOW policy with a match-all principal (*All* / `"*"`, `iam:*`, `jwt:*`) + `actions: ["*"]`. - [ ] No connector on an IAM-inbound gateway uses outbound **Inbound forward**; on JWT-inbound gateways only for your own MCP servers validating the same IdP. - [ ] `mcp_servers.json`: `auth: iam` / `user_jwt` only point at `*.agentbase-gateway.aiplatform.vngcloud.vn` (no non-gateway WARNING in the startup log, except your own JWT-validating server). - [ ] Each prod agent has its own principal (dedicated runtime service account) and only the actions it needs. - [ ] Side-effect tools are both restricted by Policy and go through `HITL_TOOLS`; on OAuth **3LO** connectors (consent of whoever clicked *Authorize*, shared by the project) every side-effect tool is in `HITL_TOOLS` and the OAuth scopes are minimal. - [ ] Connector names match `[a-z0-9]{3,50}`; each `allow_tools` list equals the Policy actions of that connector for this agent. -- [ ] Policies give the same result under first-match and deny-wins (ALLOW lists, no catch-all DENY, carve-out DENY first); few per group (documented 10–20, see above). +- [ ] Policies give the same result under first-match and deny-wins (deny-wins observed; ALLOW lists, no catch-all DENY, carve-out DENY first); few per group (documented 10–20, see above). - [ ] Upstream secrets live in Identity (Managed/Custom provider), rotated periodically. - [ ] Traces show `mcp.policy_denied` = 0 on the main flow (if > 0 ⇒ missing policy or the prompt calls the wrong tool). - [ ] Inbound NONE only in labs. diff --git a/skills/agentbase-build/SKILL.md b/skills/agentbase-build/SKILL.md index cb25018..0650a11 100644 --- a/skills/agentbase-build/SKILL.md +++ b/skills/agentbase-build/SKILL.md @@ -105,7 +105,7 @@ Show progress as `Step X/9`. Each step checks whether it is already done (idempo 7. **Tracing + Eval** — install the official Langfuse skill (`npx skills add langfuse/skills --skill langfuse`) next to this set, then `/agentbase-build-tracing` (Langfuse v4 keys — the user runs `make set-secret NAME=langfuse` first —, prompt management, model prices; run a few requests and `lf_trace.py --latest 3 --audit` until it reports no FAIL) and `/agentbase-build-eval` (write the dataset from the spec, `make eval` meets the threshold). 8. **Frontend** (if any) — mobile `/agentbase-build-frontend`, web `/agentbase-build-web`. 9. **Deploy + verify** — `make security-scan` exits 0 (`/agentbase-build-security-audit`; CI runs it before every deploy; a full audit before the first prod release). Then `/agentbase-build-deploy` (CI: the user sets the GitHub Environment secrets with `gh secret set --env `; `/agentbase-deploy`: the user runs `make set-secret ENV_FILE=src/backend/.env.`), check `/health`, send a test call, inspect the trace in Langfuse (`release` = image tag), logs via `/agentbase-monitor`. - - **9b. Grant the runtime principal** (when the agent uses an MCP Gateway with IAM inbound): the runtime has its own service account (`auto-sa-`), so a fresh deploy gets 403 on every tool. Read its principal from the runtime logs (`IAM principal … iam:`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured: `runtime.sh logs --query "IAM principal"`), add it to the environment's Policy Group with `/agentbase-policy`, record it in `.agentbase-state.json` → `environments..principal`, re-test a tool call. Repeat per environment and after a service-account reset. + - **9b. Grant the runtime principal** (when the agent uses an MCP Gateway with IAM inbound): the runtime has its own service account (`auto-sa-`), so a fresh deploy is denied on every tool (the bound group has no policy for it yet). Read its principal from the runtime logs (`IAM principal … iam:`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured: `runtime.sh logs --query "IAM principal"`), add it to the environment's Policy Group with `/agentbase-policy`, record it in `.agentbase-state.json` → `environments..principal`, re-test a tool call. Repeat per environment and after a service-account reset. Finish: run **`references/checklist.md`** (Definition of Done) and report PASS/FAIL per item. diff --git a/skills/agentbase-build/references/checklist.md b/skills/agentbase-build/references/checklist.md index 76dd1fb..7bcca98 100644 --- a/skills/agentbase-build/references/checklist.md +++ b/skills/agentbase-build/references/checklist.md @@ -30,7 +30,7 @@ Run each item and report `PASS` / `FAIL` / `N/A` with evidence (command + short - [B] Side-effect tools are in `HITL_TOOLS` (or the user confirmed in writing that it is not needed). - [B] Each local side-effect tool passes `idempotency_key(config, tool_call_id)` to the system it changes (or that system dedupes on a business key) — HITL alone does not stop a double run across replicas (`/agentbase-build-hitl`). - Prod MCP goes through the MCP Gateway, inbound `IAM` or `JWT`; secrets live in Identity (Secret Provider). -- [B] The Gateway has a **Policy Group** attached (none attached ⇒ every tools/call returns 403); there is an ALLOW policy with the exact action `__` for each principal that calls it: the developer's `iam:` (`make check-creds`, local runs) and **each environment's runtime** `iam:` (runtime logs `IAM principal …`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured — `/agentbase-build` Step 9b). Dev and prod use separate gateways/groups (few policies per group — documented 10–20, observed 100 accepted on 2026-11-09; check `errors.quota.policy`). +- [B] The Gateway has a **Policy Group** attached (none attached ⇒ **fail-open**: every valid IAM caller could call every tool, verified live 2026-11-09 — bind it before callers can reach the gateway; a bound group with no matching ALLOW denies); there is an ALLOW policy with the exact action `__` for each principal that calls it: the developer's `iam:` (`make check-creds`, local runs) and **each environment's runtime** `iam:` (runtime logs `IAM principal …`, logged at startup by `log_iam_principal()` — only when an enabled `auth: "iam"` MCP server is configured — `/agentbase-build` Step 9b). Dev and prod use separate gateways/groups (few policies per group — documented 10–20, observed 100 accepted on 2026-11-09; check `errors.quota.policy`). - [B] `mcp_servers.json` points to **each connector's connectUrl** (`/`), not the gateway root, and every entry declares `auth` explicitly. - [B] `HITL_TOOLS` patterns all match a tool (`tools.collect.output.hitl_unmatched` is empty). - MCP servers in a VPC / data center go through a **Private** MCP Gateway (`references/private-networking.md`); they still authenticate every call. diff --git a/skills/agentbase-build/references/platform-coverage.md b/skills/agentbase-build/references/platform-coverage.md index d5da72f..0d806a8 100644 --- a/skills/agentbase-build/references/platform-coverage.md +++ b/skills/agentbase-build/references/platform-coverage.md @@ -31,7 +31,7 @@ Maps each item in the AgentBase docs (docs.greennode.ai/ai-stack/agent-base, 202 2. `greennode-agentbase` 1.0.3: `insert_memory_records_directly` needs a `MemoryRecordInsertDirectlyRequest` (the official sample passes a list ⇒ TypeError); `search` returns `list[dict]`. 3. `AgentBaseMemoryEvents`: reading state right after an interrupt may miss the `__interrupt__` write, and the bridge inserts a fake ToolMessage ⇒ take the interrupt from the graph output, treat `next=('approval',)` as pending approval. 4. SDK `IAMCredentials` falls back per field ⇒ mismatched ID/secret pair ⇒ 401 ⇒ `.greennode.json` is the only source locally. -5. Each MCP Connector has its own `connectUrl` (`/`); the gateway root is unusable; policy principal = `iam:` (not client_id) — the developer's locally, the runtime service account's on AgentBase; a gateway without a Policy Group ⇒ 403 for everything; a policy deny comes back primarily as **HTTP 403** on `tools/call` ("No policy allows this request"; the MCP adapter raises an `ExceptionGroup[HTTPStatusError]`), sometimes as an MCP error "Request denied by policy." (`ToolException`) ⇒ the LLM retries unless there is a guard (`app/tools/mcp.py` handles both). +5. Each MCP Connector has its own `connectUrl` (`/`); the gateway root is unusable; policy principal = `iam:` (not client_id) — the developer's locally, the runtime service account's on AgentBase; a gateway without a Policy Group answered 403 in 2026-10 runs but **allowed every call** in a live lab on 2026-11-09 (one account, one gateway) ⇒ treat it as fail-open and bind a group first; a policy deny came back as **HTTP 403** on `tools/call` ("No policy allows this request"; the MCP adapter raises an `ExceptionGroup[HTTPStatusError]`) in 2026-10 and as **HTTP 200 with an MCP tool error "Request denied by policy."** (`ToolException`) on 2026-11-09 (deny wins over allow; empty or dangling group ⇒ deny; the gateway root is 404, a target is `/`) ⇒ the LLM retries unless there is a guard (`app/tools/mcp.py` handles both). 6. GreenNode IAM does not publish JWKS ⇒ IAM tokens cannot be used as inbound JWT. 7. Memory API: search query ≤ 1000 characters (400 if longer); checkpoint reads sometimes hang ~7 minutes with the SDK's default timeout/retry ⇒ 10s timeout, 3 retries (`MEMORY_MAX_RETRIES`), `REQUEST_TIMEOUT_S`. 8. MaaS sometimes returns 503 (glm-5.3-flash) ⇒ a fallback model is mandatory for prod.