Docs re-read: correct skills where the AgentBase docs differ (inbound auth header stripping, rate-limit fallback, memory limit, MCP)

CHANGELOG.md

@@ -8,6 +8,28 @@ review can tell which standard a project was built from.
 
 ## Unreleased
 
+- **Docs re-read (2026-11): skills corrected where the AgentBase docs say otherwise** (docs-only; no template code changed).
+  Runtime Inbound Auth `IAM Permissions` / `JWT` **strips `Authorization` before the container** (only *No authorization*
+  forwards it) and the Console now defaults a new Runtime to IAM: `agentbase-build-deploy`, `-auth`, `-web`, `-frontend` and
+  `api-contract.md` no longer tell the agent to read the user JWT from `Authorization` behind a Runtime JWT (send it also in
+  `X-GreenNode-AgentBase-Custom-User-Token`; the Expo template and the BFF `jwt` mode still send one header, not run live).
+  `-llm`: a Protect & Govern limit on an API key is shared by all its models, so a fallback model does not bypass it; the
+  Sidecar LLM Proxy is mentioned in the MCP Gateway docs. `-memory`: event pagination `from` ≤ 5000 is now documented; no
+  `{sessionId}` in the namespace template; strategies and expiry are set at store creation. `-mcp`: gateway JWT settings
+  (Discovery URL / JWKS, Principal claim), connector Header key / prefix, default scopes, M365 VNG bundle, Slack needs a
+  Custom provider, 22 policy operators, conditions combine with AND. Also: Outbound IP of a Private Runtime, vCR
+  `backendName` / robot username / expiry, runtime env vars "non-sensitive config only", stateless contract, ephemeral
+  filesystem (vStorage S3), a Built-in Tools row in the decision guide. **Verified live (2026-11-09, temporary resources):** deleting a Policy Group does
+  **not** detach it (the gateway keeps the dangling `policyGroupId`; unbind first with `PATCH` `{"policyGroupId": null}`, `application/merge-patch+json`), and the documented
+  policy-group limits (10/20 per group, 20 groups) were not enforced on that account (100 policies accepted, 429 `errors.quota.policy` at 101; 43 groups) — the skills say
+  to rely on that error, not a number. A gateway has its own identity `gw-<id>` (removed with it), a runtime `runtime-<id>`.
+  **Second live lab (2026-11-09, one account, one gateway, IAM inbound):** a gateway with **no Policy Group bound let every
+  valid IAM caller's `tools/call` through** (the skills said 403; an earlier 2026-10 run saw 403) — now a security rule: bind
+  a group before callers can reach the gateway; a bound group with zero policies, or a dangling id, denies; a deny arrived
+  as HTTP 200 with an MCP tool error `Request denied by policy.` (no 403 seen); deny wins over allow in any order (the
+  Console docs say first match); bare `iam` / `jwt` principals are rejected by the API with 400 (use `iam:*`, `jwt:*`, `*`);
+  `iam:<client_id>` denies; policy changes apply in seconds, binding via `PATCH` after UPDATING → ACTIVE; targets are called
+  at `<gateway endpoint>/<target name>`. Not tested: JWT-inbound gateways, non-owner callers, conditions, a real Runtime SA.
 - **Deploy: find out how the customer can build before building.** `agentbase-build-deploy` Step 3 now starts with
   `scripts/detect_build_env.sh` (read-only; no login, no secret printed): which engine is installed and answering (Docker
   Desktop, Rancher Desktop, OrbStack, Colima, Podman, nerdctl, Finch, buildah), `linux/amd64` support, git host and CI config

skills/agentbase-build-auth/SKILL.md

@@ -35,10 +35,11 @@ Role-based access (RBAC): put `roles`/`groups` in `AUTH_FORWARD_CLAIMS` and chec
 
 ## AgentBase Runtime endpoint security
 
-The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): **IP Access Control** (allowed source CIDRs) and **Inbound Auth type** — `IAM Permissions` (GreenNode IAM token), `JSON Web Tokens` (Discovery URL or inline JWKS), or `No authorization` (public: anyone with the URL reaches the container — what we observed on runtimes created before this setting, 2026-10).
+The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): **IP Access Control** (allowed source CIDRs) and **Inbound Auth type** — `IAM Permissions` (GreenNode IAM token; the **default** for a new Runtime in the Console), `JSON Web Tokens` (Discovery URL or inline JWKS), or `No authorization` (public: anyone with the URL reaches the container — what we observed on runtimes created before this setting, 2026-10).
+- **In `IAM Permissions` and `JSON Web Tokens` mode the platform strips `Authorization` (and `x-agentbase-endpoint-inbound-header-authorization`) before the request reaches the container** — only `No authorization` forwards them ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime), *Header Handling on Incoming Requests*). The agent can read a user token only from another header, so the caller sends it there too, and the agent sets `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`. `AUTH_TOKEN_HEADER=Authorization` works only behind `No authorization`; behind IAM/JWT the agent would see no token and answer 401 to every request.
 
 - The agent **still authenticates itself** (`jwt`/`api_key`) whatever the Runtime setting — defense in depth, and `Settings` blocks `AUTH_MODE=none` outside local.
-- Runtime `JWT` with the same IdP as `AUTH_*` ⇒ bad tokens are rejected before the container (cheaper), and pattern **A. Direct** works. It validates the **key source** (discovery/JWKS) only — it **never replaces `AUTH_AUDIENCE` / `AUTH_ALLOWED_CLIENT_IDS`**: a token the same IdP issued to another app passes it, the agent's own checks stop it.
+- Runtime `JWT` with the same IdP as `AUTH_*` ⇒ bad tokens are rejected before the container (cheaper); pattern **A. Direct** then needs the app to send the user JWT **twice** (`Authorization` for the platform, the custom header for the agent) — the Expo template sends one header and the BFF's `jwt` mode sends only `Authorization`, so with a Runtime JWT adapt the client (or use the IAM runtime + BFF). It validates the **key source** (discovery/JWKS) only — it **never replaces `AUTH_AUDIENCE` / `AUTH_ALLOWED_CLIENT_IDS`**: a token the same IdP issued to another app passes it, the agent's own checks stop it.
 - Runtime `IAM Permissions` ⇒ a mobile/web app cannot call it directly ⇒ pattern **B. BFF**.
 - Add IP Access Control for server-to-server callers (BFF, other agents) when their egress IPs are fixed.
 - GreenNode IAM does not publish a JWKS (404) ⇒ IAM tokens cannot be used as the agent's inbound JWTs.
@@ -55,7 +56,7 @@ The Runtime has **Security Settings** ([create-runtime](https://docs.greennode.a
 ## Calling the runtime endpoint: 2 patterns
 
 Read `references/patterns.md`. Summary:
-- **A. Direct (default):** app → runtime endpoint with the user JWT in `Authorization`. Use when the endpoint does not require IAM in `Authorization`.
+- **A. Direct (default):** app → runtime endpoint with the user JWT in `Authorization`, read by the agent (`AUTH_TOKEN_HEADER=Authorization`) only when the Runtime is `No authorization`. Behind a Runtime JWT the app sends the token twice (see *Runtime endpoint security*); behind Runtime IAM use B.
 - **B. BFF:** app → your BFF (verifies JWT) → runtime endpoint with an IAM token; the user JWT goes in a custom header (`AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`). Use when the endpoint requires IAM, or you need your own rate-limiting/billing. **Never** embed IAM credentials in a mobile app.
 
 Choose the pattern from the Runtime's **Inbound Auth type** (above); verify: `/invocations` **without** a token ⇒ 401 **and with a valid user token ⇒ 200** (a setup that rejects everyone also passes the first check).

skills/agentbase-build-auth/references/patterns.md

@@ -8,6 +8,7 @@ Mobile app ──(Authorization: Bearer <user JWT>)──▶ AgentBase Runtime e
 
 - Backend: `AUTH_MODE=jwt`, `AUTH_TOKEN_HEADER=Authorization`.
 - Frontend: `EXPO_PUBLIC_AUTH_TOKEN_HEADER=Authorization`.
+- **Only behind a Runtime with Inbound Auth `No authorization`**: with `JWT` or `IAM Permissions` the platform strips `Authorization` before the container, so the agent would see no token (every request 401). Behind a Runtime JWT the app must send the user JWT in `Authorization` (for the platform) **and** in `X-GreenNode-AgentBase-Custom-User-Token` (for the agent, `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`); the Expo template sends one header — that needs a client change. Not run against a live Runtime JWT yet: verify on dev (valid token ⇒ 200).
 - Pros: simple, fewer hops. Cons: no dedicated rate-limit layer.
 - Web (Expo web): set `CORS_ALLOW_ORIGINS='["https://app.example.com"]'`.
 - Verify with **both**: no token ⇒ 401, **a valid user token ⇒ 200** (a misconfiguration that rejects every token also passes the first check).

skills/agentbase-build-deploy/SKILL.md

@@ -34,7 +34,7 @@ The scaffold commits one **non-secret** config file per environment: `src/backen
 | `MCP_GATEWAY_URL`, `HITL_TOOLS`, `MEMORY_MAX_CONCURRENCY` | per environment (separate dev/prod gateways) | `≈ 10 / max replicas` |
 | `APP_TIMEZONE`, `LLM_TASK_MAX_TOKENS`, `LLM_THINKING_BUDGET`, `LTM_MIN_SCORE`, `LTM_FACT_MAX_CHARS`, `REFLECTION_MODE`, `REFLECTION_TIMEOUT_S` | as evaluated (empty = app default) | same as the eval config |
 
-**Which secrets may be in the runtime env file** (rendered from the CI secret store, or the user's git-ignored `.env.<env>`): `LLM_API_KEY`, `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY`, `FEEDBACK_SIGNING_KEY`, static keys the agent presents to its own MCP servers / A2A peers (`${VAR}` in `mcp_servers.json` / `a2a_agents.json`). **Never** there: credentials tools use to call external services (SaaS API keys, OAuth tokens) — those live in AgentBase Identity (`/agentbase-build-identity`); MCP Gateway outbound credentials live in Access Control. Never `GREENNODE_CLIENT_ID/SECRET/AGENT_IDENTITY` (injected) or `GREENNODE_ENDPOINT_URL` (reserved; set `A2A_PUBLIC_URL`). Remind the user to review the env file themselves; **do not** print secret-bearing files into chat.
+**Which secrets may be in the runtime env file** (rendered from the CI secret store, or the user's git-ignored `.env.<env>`): `LLM_API_KEY`, `LANGFUSE_PUBLIC_KEY`/`LANGFUSE_SECRET_KEY`, `FEEDBACK_SIGNING_KEY`, static keys the agent presents to its own MCP servers / A2A peers (`${VAR}` in `mcp_servers.json` / `a2a_agents.json`). **Never** there: credentials tools use to call external services (SaaS API keys, OAuth tokens) — those live in AgentBase Identity (`/agentbase-build-identity`); MCP Gateway outbound credentials live in Access Control. Never `GREENNODE_CLIENT_ID/SECRET/AGENT_IDENTITY` (injected) or `GREENNODE_ENDPOINT_URL` (reserved; set `A2A_PUBLIC_URL`). Remind the user to review the env file themselves; **do not** print secret-bearing files into chat. The Runtime docs describe the Environment Variables field as "Non-sensitive config only" ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)): the platform has no secret store for it, and anyone who can read the runtime's versions (Console, API, `grn` with `-o json`) sees the values. This list is the deliberate, minimal exception — tell the user, and keep any credential a tool uses for an external service out of it.
 
 ## Step 3 — Build the image
 
@@ -70,11 +70,11 @@ Standard Dockerfile (`src/backend/Dockerfile`): multi-stage `python:3.13-slim` *
 
 - **Agent identity — one per environment** (`<agent>-dev`, `<agent>-staging`, `<agent>-prod`; each runtime gets its own service account). Never share an identity between environments and never put the prod identity in a developer's `.greennode.json`: any dev or `api_key` caller could then fetch another environment's per-user 3LO tokens / delegated keys. Bind each runtime to its environment's identity, holding that environment's providers (`/agentbase-build-identity`); otherwise `@agent_api_key` / `@user_access_token` return 404 or "can't retrieve credential". The Runtime injects it as `GREENNODE_AGENT_IDENTITY`. Per-user helpers refuse unless `AUTH_MODE=jwt` (`IDENTITY_TRUST_ASSERTED_USER=false`, the default).
   - With per-user credentials, `IDENTITY_CALLBACK_URL` in `.env.<env>` must be this environment's page and must be listed in the identity's `allowedReturnUrls`.
-- **Network mode** — `PUBLIC` by default. `VPC` (VPC + Subnet + optional Route CIDRs; needs VPC Peering and a flavor that supports `agent-runtime-vpc`) only when agent code must reach an internal API or a self-hosted service; internal MCP servers go through a Private MCP Gateway instead — see `/agentbase-build` `references/private-networking.md`.
+- **Network mode** — `PUBLIC` by default. `VPC` (VPC + Subnet + optional Route CIDRs; needs VPC Peering and a flavor that supports `agent-runtime-vpc`) only when agent code must reach an internal API or a self-hosted service; internal MCP servers go through a Private MCP Gateway instead — see `/agentbase-build` `references/private-networking.md`. A Private runtime reaches your side from its **Outbound IP** (runtime detail page, [manage-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/manage-runtime)): add it to the allow list of any firewall, security group or database that filters by IP.
 - **Security Settings** ([create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime)):
-  - **Inbound Auth type** — `JWT` (same IdP as `AUTH_*`; lets the app call directly), `IAM Permissions` (server callers / BFF only), or `No authorization` (public — the agent still verifies tokens itself). Prefer JWT or IAM for production.
+  - **Inbound Auth type** — `IAM Permissions` (the Console default for a new Runtime, "secure by default"; server callers / BFF only), `JWT` (same IdP as `AUTH_*`), or `No authorization` (public — the agent still verifies tokens itself). Prefer IAM or JWT for production. A runtime created with `grn` / the API may not get the Console default: read the new version's inbound mode after creating it (a CI smoke test that calls `/health` without a token only works on a `No authorization` endpoint).
   - **IP Access Control** (`allowedCidrs`) — allowed source CIDRs for server-to-server callers with fixed egress (BFF, other agents).
-  - With Runtime JWT on `Authorization`, keep the agent's `AUTH_TOKEN_HEADER=Authorization` (same token); with Runtime IAM, the BFF sends the IAM token in `Authorization` and the user JWT in `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token` (`/agentbase-build-auth`).
+  - **The platform strips `Authorization` before the container in `IAM Permissions` and `JWT` mode** (and `x-agentbase-endpoint-inbound-header-authorization` too; only `No authorization` forwards them — [create-runtime](https://docs.greennode.ai/ai-stack/agent-base/agent-runtime/create-runtime), *Header Handling on Incoming Requests*). The agent can only read a token the caller also sends in another header: with Runtime IAM the BFF sends the IAM token in `Authorization` and the user JWT in `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`; with Runtime JWT the caller sends the same user JWT twice (`Authorization` for the platform, the custom header for the agent, `AUTH_TOKEN_HEADER=X-GreenNode-AgentBase-Custom-User-Token`). `AUTH_TOKEN_HEADER=Authorization` is right only behind `No authorization` — behind IAM/JWT every request would be a 401 (`/agentbase-build-auth`). Other headers (`X-Api-Key`, cookies, `traceparent`, custom) are forwarded in every mode.
 
 **Every update is a full replacement — re-send the network and security settings.** Before `runtime.sh update`, read the latest version without printing its env (it holds secrets; > 100 versions ⇒ also read `--page 2…` up to `totalPage`):
 
@@ -154,7 +154,7 @@ The scaffold ships `deploy/agent.yaml.tpl` (rendered by CI), `deploy/render_runt
 - **Credentials:**
   - CI uses a **dedicated service account**, never a person's.
   - Least privilege: the managed policies (`AgentBaseFullAccess`, `vcrFullAccess`, `AiPlatformFullAccess`) are broad; a custom IAM policy with only the actions CI needs can be built in the IAM Console (*Policies* → select actions — platform `/agentbase`, IAM setup). Validate it with one dev deploy. See `/agentbase-build` `references/iam-permissions.md`.
-  - Image pull/push uses a vCR robot account.
+  - Image pull/push uses a vCR robot account. The registry is `vcr.vngcloud.vn`; the image path is `vcr.vngcloud.vn/<repo backendName>/<image>:<tag>` (the repository's `backendName`, shown on the repository detail page / in the API response — not its display name) and the `docker login` username is the robot's full `backendName`, `<prefix>-<chosen name>` (e.g. `109072-deploy-bot`), not the name typed at creation ([supporting-services](https://docs.greennode.ai/ai-stack/agent-base/supporting-services)). A robot needs push + pull permissions and is created with a **Duration (days)**: note its expiry date and rotate it before then (what an expired robot does to a runtime's image pull is not documented).
 - **What the smoke test proves:** it calls `GET /ready` (retried for a minute): 200 = the replica can serve — and with `AUTH_MODE=jwt` can fetch the IdP's JWKS, so users can log in; 503 = it cannot (the body names `jwks`). `degraded` (HTTP 200, `failing: ["llm"]` or `backing_service`) reports a dependency that failed recently — it needs real traffic to appear, so a bad LLM key is not caught before the first request. A release from before `/ready` answers 404 and is checked on `/health`. It does not show which version serves: the deploy step checks the latest version's `imageUrl` and the tag's digest; Langfuse `release` (= tag) confirms what serves traffic.
 - **No automatic rollback** (a half-applied rollback is worse than a failed deploy): when a deploy fails, the last step prints the previous image and the exact dispatch (`image_tag=<previous tag>`, from that release's tag).
 - **`/health` vs `/ready`:** `/health` is the platform's liveness probe and stays 200 while the process runs — restarting a replica does not fix an IdP or LLM outage. Monitor `/ready` from outside (uptime check, alert on 503).
@@ -186,8 +186,8 @@ Runtimes are billed continuously (a `STOPPED` runtime keeps its config without c
 
 1. Runtime(s) incl. custom endpoints — `/agentbase-teardown` or `/agentbase-deploy`.
 2. Memory store(s), identity + Access Control providers, LLM API keys, vCR images — `/agentbase-teardown`.
-3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-<id>` service account).
-4. Policy Groups — `/agentbase-policy` (detach from the gateway first).
+3. MCP Gateways created for this agent — `/agentbase-gateway` (each gateway also leaves an `sa-gateway-<id>` service account; its own Access Control identity `gw-<gateway id>` — field `agentIdentityName` — is removed with the gateway, verified live 2026-11-09). A runtime likewise gets an identity `runtime-<runtime id>` right after creation.
+4. Policy Groups — `/agentbase-policy`. Deleting one is irreversible. The docs say it detaches the group from every gateway at once ([manage-policy-groups](https://docs.greennode.ai/ai-stack/agent-base/mcp-governance/policy-groups/manage-policy-groups)); **verified live (2026-11-09) the API does not**: `DELETE /policy/api/v1/policy-groups/{id}` returns 200 but the gateway keeps the dangling `policyGroupId` (unchanged after 60 s, state stays ACTIVE). **Unbind first**: `PATCH /gateway/api/v1/gateways/{name}` with `Content-Type: application/merge-patch+json` and `{"policyGroupId": null}` → 202, state UPDATING → ACTIVE, `policyGroupId` gone. With the dangling id the gateway denied every `tools/call` (still after 30 s; fail closed). A gateway with **no** group bound does not deny: verified live 2026-11-09 it let every valid IAM caller's `tools/call` through — so unbind-then-delete leaves the gateway open: delete the gateway too, or bind a replacement group at once. Binding an id that does not exist is rejected with 400 `policy_group_invalid`.
 5. MCP Connectors (catalog/custom) — Console.
 6. The **Orphaned** runtime service account (`auto-sa-<agent>`) and gateway service accounts — revoke in IAM.
 7. Inbound API keys: remove their hashes from `AUTH_API_KEY_SHA256` of the callers' environments, and the A2A keys peers hold for this agent.