Add integration test: deploy Keystone on kind (forge) + create a project via the admin account

[berendt]

Summary

Add a first integration test for openstack-project-manager that exercises
the real project-creation path against a live Keystone, not mocks. Keystone
is deployed locally on a kind cluster via
c5c3/forge (its ControlPlane Quick Start), and
the test then creates one project using the admin account and verifies it
through the OpenStack SDK.

This is deliberately a scaffold: one happy-path test. The goal is to stand
up the harness (kind + Keystone + clouds.yaml + a runnable assertion + CI
wiring) so that further integration tests can be added incrementally later.

Today the suite is unit-only (test/unit/, stdlib unittest, all OpenStack
clients mocked), run via the openstack-project-manager-tox Zuul job. Nothing
ever talks to a real Keystone, so regressions in the actual
openstack.connect() / keystone.projects.update() apply path can slip
through.

Reuse, don't reinvent

The kind/kubectl/Helm installation and the CI harness shape are copied from
the proven netbox-manager E2E work — do not author a novel installer:

• Reference: Add end-to-end (E2E) test: deploy NetBox on kind (Helm) + apply example/ + verify via API netbox-manager#263 — "Add end-to-end (E2E) test: deploy
  NetBox on kind (Helm) + apply example/ + verify via API".

What we reuse from that PR (almost verbatim, adapted for this repo):

1. Ansible pre-run playbook that installs pinned, checksum-verified
   kind/kubectl (+ Helm) and ensures Docker is running, using the
   ensure-pip / ensure-pipenv / ensure-docker roles.
2. The IPv6 accept_ra=2 sysctl fix in the pre-run playbook. This is the
   non-obvious, hard-won bit: the OSISM Zuul nodes are IPv6-only and learn
   their default route via SLAAC/RA. When kind makes Docker create a
   dual-stack network it enables net.ipv6.conf.all.forwarding, after which the
   kernel (default accept_ra=1) stops honouring RAs and the node drops off the
   network mid-run (RESULT_UNREACHABLE). Setting accept_ra=2 keeps RAs
   honoured while forwarding is on. Carry this over.
3. trap-based teardown in run.sh that always deletes the kind cluster on
   exit (success or failure), but only tears down a cluster this run actually
   created — a pre-existing debug cluster of the same name is reused and left in
   place. Plus a cluster diagnostics dump on failure before teardown.
4. Makefile targets integration / integration-up / integration-down
   (analogous to netbox's e2e / e2e-up / e2e-down).
5. Zuul job placed in periodic-daily (heavy, ~15 min) so the PR
   check gate stays fast — mirroring netbox's pre-e2e.yml / test-e2e.yml
   split and timeout: 2400.

Keystone via forge (only Keystone is needed)

Follow the forge ControlPlane Quick Start
(https://c5c3.github.io/forge/quick-start-controlplane.html) — only the
Keystone service is deployed:

# 1. toolchain (forge's own pinned installer → ~/.local/bin)
git clone https://github.com/c5c3/forge.git   # pin a ref!
cd forge
make install-test-deps           # pins kind v0.32.0, kubectl v1.36.1, jq, ...
export PATH="${HOME}/.local/bin:${PATH}"

# 2. infra + ControlPlane operator stack on a kind cluster (cluster name: "forge")
KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true make deploy-infra

Then apply a Keystone-only ControlPlane CR (namespace openstack) and
wait for it:

apiVersion: c5c3.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: openstack
spec:
  openStackRelease: "2025.2"
  services:
    keystone:
      replicas: 1
      publicEndpoint: https://keystone.127-0-0-1.nip.io:8443/v3
      gateway:
        parentRef:
          name: openstack-gw
        hostname: keystone.127-0-0-1.nip.io
        path: /

kubectl apply -f controlplane.yaml
kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=15m

forge also supports WITH_CONTROLPLANE_CR=true to auto-apply a bundled CR,
but that bundle may include more than Keystone. Since only Keystone is
needed, we apply our own minimal CR (above), exactly as the Quick Start does.

Keystone access (no port-forward — the shared Envoy Gateway exposes it):

• OS_AUTH_URL = https://keystone.127-0-0-1.nip.io:8443/v3 (nip.io → 127.0.0.1)
• OS_USERNAME = admin, OS_PROJECT_NAME = admin
• OS_USER_DOMAIN_NAME = Default, OS_PROJECT_DOMAIN_NAME = Default
• Admin password from the operator-projected Secret:

  kubectl get secret controlplane-keystone-admin-credentials \
    -n openstack -o jsonpath='{.data.password}' | base64 -d

• The endpoint uses a self-signed cert (cert-manager) → verify: false
  (a.k.a. --insecure).

How the test reaches the project-manager (clouds.yaml)

openstack-project-manager authenticates exclusively via clouds.yaml named
clouds; the create/manage commands default to --cloud admin. The harness
generates a clouds.yaml from the forge admin credentials:

clouds:
  admin:
    auth:
      auth_url: https://keystone.127-0-0-1.nip.io:8443/v3
      username: admin
      password: "<from controlplane-keystone-admin-credentials>"
      project_name: admin
      user_domain_name: Default
      project_domain_name: Default
    identity_api_version: 3
    verify: false     # self-signed cert behind the Envoy Gateway

Point the run at it via OS_CLIENT_CONFIG_FILE (or place it on the standard
search path).

The single test

Action — create one project as admin, using the real CLI path:

tox -e create -- --cloud admin --domain default --name opm-integration-test \
                 --quota-class basic --nocreate-user

(tox -e create runs python openstack_project_manager/create.py, see
tox.ini. --nocreate-user keeps the scaffold minimal — no per-user
application credential.)

Verify — assert via the OpenStack SDK in a standalone
test/integration/verify.py (mirroring netbox's verify.py), e.g.:

import openstack
cloud = openstack.connect(cloud="admin")
project = cloud.identity.find_project("opm-integration-test", domain_id=<default-domain-id>)
assert project is not None
# optional spot-checks: the quota-class property and the per-project group exist

The script exits non-zero on any failed assertion. run.sh then optionally
deletes the project to keep reruns idempotent.

⚠️ Must not be collected by the fast unit gate. tox -e test runs
python -m unittest discover ./test, which recurses. The verifier must be a
standalone script (not named test_*.py) — exactly the rationale
netbox-manager used for keeping verify.py out of pytest tests/unit.

Proposed deliverables (file-by-file)

Mirroring netbox-manager#263, adapted to this repo's layout (test/ singular,
no Helm needed, Keystone via forge instead of a chart):

File	Purpose
test/integration/deploy_keystone.sh	Clone forge @ pinned ref, make install-test-deps, KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true make deploy-infra, apply the Keystone-only ControlPlane CR, wait Ready, emit clouds.yaml. Safe to run standalone (make integration-up).
test/integration/run.sh	Orchestrate: deploy → generate clouds.yaml → tox -e create -- … → verify.py. trap-based teardown of a self-created cluster on every exit path + diagnostics dump on failure.
test/integration/verify.py	Standalone openstacksdk assertions (project exists, etc.). Not a unittest/pytest test.
test/integration/controlplane.yaml	The Keystone-only ControlPlane CR.
test/integration/README.md	Prerequisites (Docker, forge), make usage, env overrides, what is asserted.
Makefile (new)	integration (full cycle + teardown), integration-up (deploy + leave running), integration-down (kind delete cluster --name forge / forge make teardown-infra).
playbooks/pre-integration.yml	Reuse netbox#263 pre-e2e.yml: ensure-pip/ensure-pipenv/ensure-docker roles, the IPv6 accept_ra=2 sysctl fix, ensure Docker running. (Tool binaries via forge's make install-test-deps, invoked from deploy_keystone.sh.)
playbooks/test-integration.yml	Install deps into the venv, run make integration.
.zuul.yaml	New job openstack-project-manager-integration (nodeset: ubuntu-noble, pre-run: playbooks/pre-integration.yml, run: playbooks/test-integration.yml, timeout: 2400); add to periodic-daily only (keep check fast).

Open decisions (please confirm during implementation)

1. Toolchain install source. Recommended: reuse netbox#263's node-prep
   (Docker + IPv6 fix + pip) and let forge's own make install-test-deps
   install the pinned kind/kubectl/jq (authoritative for forge, avoids
   version drift; forge pins kind v0.32.0, the same version netbox#263 pins).
   Alternative: port netbox#263's pinned get_url download tasks and align
   versions with forge. Either way we are not writing a novel installer.
2. forge pinning. Pin a specific forge git ref/tag in deploy_keystone.sh
   for reproducibility (forge is a fast-moving prototyping repo).
3. Project-manager invocation flags. Confirm the minimal create flags work
   against a bare forge Keystone — create.py also creates a <domain>-admin
   group, a per-project group, assigns default roles, and sets quota-class
   properties on the project. Verify the default Keystone ships the expected
   roles (admin/member/reader) and Default domain; adjust flags
   (--nocreate-user, --quota-class basic) if any step is unsupported.
4. CI placement. Recommend periodic-daily only (heavy job). If a PR-gate
   signal is wanted, it can additionally be added to check.
5. Resources. forge's kind-config.yaml targets a single node ~7 GB RAM /
   2 vCPU; confirm the ubuntu-noble nodeset matches what netbox#263 uses.

Acceptance criteria

• make integration on a Docker-capable host: provisions kind + Keystone
  via forge, creates one project as admin, asserts it via the SDK, and tears
  the cluster down on every exit path (incl. failure).
• make integration-up / make integration-down work for local debugging.
• New openstack-project-manager-integration Zuul job runs in
  periodic-daily; the check gate is unchanged and stays fast.
• The verifier is not collected by tox -e test.
• test/integration/README.md documents prerequisites and usage.
• No real credentials are committed; the generated clouds.yaml is ephemeral.

References

• netbox-manager E2E harness (pattern to copy): Add end-to-end (E2E) test: deploy NetBox on kind (Helm) + apply example/ + verify via API netbox-manager#263
• forge ControlPlane Quick Start: https://c5c3.github.io/forge/quick-start-controlplane.html
• forge basic Quick Start (toolchain): https://c5c3.github.io/forge/quick-start.html
• forge repo: https://github.com/c5c3/forge (hack/install-test-deps.sh, hack/deploy-infra.sh, hack/kind-config.yaml)

[berendt]

Implementation Plan (issue #293)

Summary

Add a single happy-path integration test that deploys a Keystone-only forge ControlPlane on a kind cluster, runs the real tox -e create path against it as admin, and asserts the project via openstacksdk — mirroring the proven osism/netbox-manager#263 E2E harness, adapted to this repo's test/ (singular) layout and forge-instead-of-Helm deployment. The change is additive: nine new files plus two small supporting edits (tox.ini passenv, .gitignore), with the new Zuul job wired into periodic-daily only so the check gate stays fast. The one non-obvious correctness requirement uncovered during grounding — create.py crashes on a bare Keystone because load-balancer_member is not a default role — is handled inside the harness by pre-creating the roles create.py requires, keeping create.py itself untouched.

Ground-Truth Notes

• tox.ini — confirmed. [testenv:create] runs python openstack_project_manager/create.py {posargs}; [testenv:test] runs python -m unittest discover ./test {posargs}. No passenv is set, so tox -e create will NOT see OS_CLIENT_CONFIG_FILE unless we add it.
• openstack_project_manager/create.py — confirmed. Defaults: --cloud admin, --domain default, --name sandbox, --quota-class basic, --create-user default False, --domain-name-prefix default True, --assign-admin-user/--create-admin-user default True.
• create.py:16 DEFAULT_ROLES = ["member", "load-balancer_member"] and create.py:20 DEFAULT_ADMIN_ROLES = [...] — confirmed. Lines 236 and 423 evaluate CACHE_ROLES[role_name] in the caller's frame (the try/except in try_assign_role_to_group/try_assign_role only wraps the assignment call, not the dict lookup), so a missing role → uncaught KeyError → non-zero exit. A bare forge Keystone ships admin/member/reader (+ likely manager/service) but NOT load-balancer_member (Octavia provisions that). This is the key blocker for the proposed create command.
• create.py:191-192 domain-name-prefix — confirmed: with the default on, the created project is named f"{domain}-{name}" i.e. default-opm-integration-test, NOT opm-integration-test. The issue's verify snippet looks for opm-integration-test, contradicting its own create command. Reconciled below by passing --nodomain-name-prefix.
• create.py:251 os_client_config.make_client("identity", cloud=cloud_name) then keystone.projects.update(...) — confirmed. python-keystoneclient is present in Pipfile.lock (line 735), and the Pipfile package set equals requirements.txt, so pip install -r requirements.txt (what tox -e create does) pulls it transitively. The quotaclass/quotamultiplier/has_* keys are arbitrary project "extra" properties Keystone stores verbatim — they work on bare Keystone.
• test/unit/test_create.py:62 — confirmed the role set ["member", "load-balancer_member"] and (test_cli_1/test_cli_9) the default-prefix behavior and --nodomain-name-prefix removing it.
• .zuul.yaml — confirmed: check and periodic-daily pipelines exist with jobs flake8, openstack-project-manager-mypy, openstack-project-manager-tox, yamllint, python-black. No Makefile exists (confirmed by ls). No test/integration/ exists (only test/unit/).
• playbooks/tox.yml / playbooks/mypy.yml — confirmed: both apt-install libldap2-dev + libsasl2-dev (needed because requirements.txt pins python-ldap==3.4.7, which tox -e create will compile). The integration pre-run must do the same.
• requirements.txt — confirmed openstacksdk==4.10.0, os-client-config==2.3.0, python-ldap==3.4.7, python-neutronclient==12.0.0. .gitignore ignores clouds.yml/secure.yml/settings.toml/.tox (note .yml, not .yaml).
• forge (c5c3/forge) — grounded via repo + docs: hack/install-test-deps.sh installs to ${HOME}/.local/bin, pins kind v0.32.0, kubectl v1.36.1 (+ chainsaw, flux; no jq). hack/deploy-infra.sh: cluster name defaults to forge, KIND_HOST_PORT default 443, WITH_CONTROLPLANE deploys operators+CRDs, WITH_CONTROLPLANE_CR (default false) applies a bundled CR; uses hack/kind-config.yaml; make teardown-infra exists. The Quick Start's make install-test-deps/make deploy-infra path installs no Go toolchain, implying deploy-infra pulls prebuilt operator images (see Risks).
• netbox-manager#263 — grounded via PR diff: playbooks/pre-e2e.yml uses roles ensure-pip/ensure-pipenv/ensure-docker + a drop-in net.ipv6.conf.all.accept_ra = 2 / ...default.accept_ra = 2; tests/e2e/run.sh uses CREATED_CLUSTER flag + dump_diagnostics + trap cleanup EXIT deleting the cluster only if this run created it; Makefile has e2e/e2e-up/e2e-down with CLUSTER_NAME ?=; the Zuul job is nodeset: ubuntu-noble, pre-run/run playbooks, timeout: 2400. (netbox added it to both check and periodic-daily; this issue requires periodic-daily only.)

Work Breakdown Coverage

The issue's "Proposed deliverables (file-by-file)" table is the work breakdown; every row is covered, plus two required supporting edits and the five "Open decisions":

• test/integration/deploy_keystone.sh — covered by Change Set entry 1, Commit 1, README (Documentation Plan). Includes the role pre-provisioning step (Open decision 3).
• test/integration/run.sh — Change Set 2, Commit 2; trap teardown + diagnostics; runs create + verify.
• test/integration/verify.py — Change Set 3, Commit 2; Test Plan (the behavioral assertion) + AC 4 negative check.
• test/integration/controlplane.yaml — Change Set 4, Commit 1.
• test/integration/README.md — Change Set 5, Commit 6, Documentation Plan.
• Makefile (new) — Change Set 6, Commit 3 (integration/integration-up/integration-down).
• playbooks/pre-integration.yml — Change Set 7, Commit 4 (roles + accept_ra=2 + docker + ldap apt deps).
• playbooks/test-integration.yml — Change Set 8, Commit 4 (venv with tox+openstacksdk, run make integration).
• .zuul.yaml (new job, periodic-daily only) — Change Set 9, Commit 5 (AC 3).
• Supporting: tox.ini passenv (Change Set 10) and .gitignore clouds.yaml (Change Set 11) — required for AC 1 and AC 6; not in the issue's table, flagged in Risks.
• Open decision 1 (toolchain source) → forge make install-test-deps; Open decision 2 (forge pin) → FORGE_REF var; Open decision 3 (create flags vs bare Keystone) → role pre-provisioning + --nodomain-name-prefix; Open decision 4 (CI placement) → periodic-daily only; Open decision 5 (resources) → ubuntu-noble (Risks).

User Stories

• As an openstack-project-manager maintainer, I want a daily CI job that runs the real create.py path against a live Keystone, so that regressions in openstack.connect() / keystone.projects.update() that mocked unit tests cannot catch are surfaced. — derived.
• As a contributor debugging the create path locally, I want make integration-up / make integration / make integration-down, so that I can stand up Keystone on kind, run the test, and tear it down without hand-running forge. — derived.

Change Set

• test/integration/deploy_keystone.sh — create (executable, #!/usr/bin/env bash, set -euo pipefail): (a) compute repo ROOT from ${BASH_SOURCE[0]}; (b) FORGE_REF="${FORGE_REF:-<pin a commit SHA>}", FORGE_DIR="${FORGE_DIR:-${TMPDIR:-/tmp}/opm-forge}", clone https://github.com/c5c3/forge.git if absent and git -C "${FORGE_DIR}" checkout "${FORGE_REF}"; (c) make -C "${FORGE_DIR}" install-test-deps and export PATH="${HOME}/.local/bin:${PATH}"; (d) KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true make -C "${FORGE_DIR}" deploy-infra; (e) kubectl get ns openstack >/dev/null 2>&1 || kubectl create ns openstack; (f) kubectl apply -f "${ROOT}/test/integration/controlplane.yaml"; (g) kubectl wait controlplane/controlplane -n openstack --for=condition=Ready --timeout=15m; (h) read password via kubectl get secret controlplane-keystone-admin-credentials -n openstack -o jsonpath='{.data.password}' | base64 -d; (i) export OS_CLIENT_CONFIG_FILE="${ROOT}/test/integration/clouds.yaml", umask 077, write the clouds.yaml (admin cloud, verify: false) with the password substituted; (j) ensure required roles exist so create.py does not KeyError: python3 - <<'PY' using openstacksdk to find_role/create_role for member and load-balancer_member (idempotent). Safe to run standalone (make integration-up).
• test/integration/run.sh — create (executable, set -euo pipefail): compute ROOT/HERE; CLUSTER_NAME="${CLUSTER_NAME:-forge}", NAMESPACE=openstack, PROJECT_NAME=opm-integration-test; export PATH="${HOME}/.local/bin:${PATH}" and export OS_CLIENT_CONFIG_FILE="${ROOT}/test/integration/clouds.yaml". Detect ownership before deploying: CREATED_CLUSTER=0; kind get clusters 2>/dev/null | grep -qx "${CLUSTER_NAME}" || CREATED_CLUSTER=1. Define dump_diagnostics() (kubectl get nodes/pods -A/events/keystone logs, each || true) and cleanup() that on non-zero rc runs dump_diagnostics, then kind delete cluster --name "${CLUSTER_NAME}" only if CREATED_CLUSTER==1 else logs "leaving pre-existing cluster in place"; trap cleanup EXIT. Body: "${HERE}/deploy_keystone.sh"; cd "${ROOT}"; tox -e create -- --cloud admin --domain default --name "${PROJECT_NAME}" --nodomain-name-prefix --quota-class basic --nocreate-user; python "${HERE}/verify.py"; optional [[ "${KEEP_PROJECT:-0}" == "1" ]] || python3 -c best-effort delete_project (the EXIT trap destroys the whole cluster anyway, so reruns are idempotent by default).
• test/integration/verify.py — create: module docstring noting it is standalone (not collected by tox -e test); import sys, openstack; PROJECT_NAME="opm-integration-test", DOMAIN_NAME="default"; main() -> int does cloud = openstack.connect(cloud="admin"), domain = cloud.identity.find_domain(DOMAIN_NAME) + assert domain is not None, project = cloud.identity.find_project(PROJECT_NAME, domain_id=domain.id) + assert project is not None, group = cloud.identity.find_group(PROJECT_NAME, domain_id=domain.id) + assert group is not None, print OK, return 0; sys.exit(main()). Keep black/flake8-clean. Do not create test/integration/__init__.py (keeps the dir a non-package so unittest discover never recurses into it).
• test/integration/controlplane.yaml — create: the Keystone-only ControlPlane CR verbatim from the issue (apiVersion: c5c3.io/v1alpha1, kind: ControlPlane, metadata.name: controlplane, metadata.namespace: openstack, spec.openStackRelease: "2025.2", spec.services.keystone with replicas: 1, publicEndpoint: https://keystone.127-0-0-1.nip.io:8443/v3, gateway.parentRef.name: openstack-gw, gateway.hostname: keystone.127-0-0-1.nip.io, gateway.path: /). Start with ---, yamllint-clean.
• test/integration/README.md — create: single H1; prerequisites (Docker running, outbound network, openstacksdk importable in the active env for local runs), make integration / integration-up / integration-down usage, env overrides (FORGE_REF, CLUSTER_NAME, KIND_HOST_PORT, OS_CLIENT_CONFIG_FILE, KEEP_PROJECT), what is asserted (project + project group exist), and the note that the generated clouds.yaml is ephemeral/gitignored and no credentials are committed. Fenced blocks tagged bash/yaml.
• Makefile — create (new, top level): CLUSTER_NAME ?= forge + export CLUSTER_NAME; .PHONY: integration integration-up integration-down; integration: → test/integration/run.sh; integration-up: → test/integration/deploy_keystone.sh; integration-down: → kind delete cluster --name $(CLUSTER_NAME). (Recipe lines TAB-indented.)
• playbooks/pre-integration.yml — create: hosts: all; roles: [ensure-pip, ensure-pipenv, ensure-docker]; tasks: apt install libldap2-dev+libsasl2-dev (become: true, mirrors playbooks/tox.yml); ansible.builtin.copy a drop-in /etc/sysctl.d/99-opm-integration-accept-ra.conf containing net.ipv6.conf.all.accept_ra = 2 and net.ipv6.conf.default.accept_ra = 2 (become: true), followed by ansible.builtin.command: sysctl --system to apply it; ensure Docker is running. (kind/kubectl come from forge's make install-test-deps, invoked by deploy_keystone.sh — do not install them here.)
• playbooks/test-integration.yml — create: hosts: all; vars: {python_venv_dir: /tmp/venv}; task 1 ansible.builtin.pip install tox and openstacksdk into python_venv_dir (virtualenv_command: python3 -m venv); task 2 ansible.builtin.shell (chdir: "{{ zuul.project.src_dir }}", set -e -o pipefail -x) that exports PATH="{{ python_venv_dir }}/bin:${HOME}/.local/bin:${PATH}" and runs make integration.
• .zuul.yaml — modify: add job openstack-project-manager-integration (nodeset: ubuntu-noble, pre-run: playbooks/pre-integration.yml, run: playbooks/test-integration.yml, timeout: 2400; no parent); add - openstack-project-manager-integration to the periodic-daily.jobs list only. Leave check.jobs unchanged (AC 3).
• tox.ini — modify (supporting): add passenv = OS_CLIENT_CONFIG_FILE under [testenv] so tox -e create inherits the credentials path. (Inherited by [testenv:create].)
• .gitignore — modify (supporting): add test/integration/clouds.yaml so the generated credential file is never committed (AC 6).

Commit Sequence

1. Add forge Keystone deploy script and ControlPlane CR — test/integration/deploy_keystone.sh, test/integration/controlplane.yaml; stands up kind + Keystone-only ControlPlane and emits clouds.yaml, pre-creating the roles create.py needs.
2. Add integration run.sh orchestration and SDK verifier — test/integration/run.sh, test/integration/verify.py, tox.ini (passenv), .gitignore (clouds.yaml); create-then-verify with trap teardown.
3. Add Makefile integration targets — Makefile; integration/integration-up/integration-down.
4. Add Ansible pre/test playbooks for integration job — playbooks/pre-integration.yml, playbooks/test-integration.yml; node prep (roles, IPv6 accept_ra=2, ldap apt deps) + venv + make integration.
5. Add periodic-daily Zuul integration job — .zuul.yaml; new job in periodic-daily only, check untouched.
6. Document the integration test harness — test/integration/README.md.

Test Plan

• test/integration/verify.py — the behavioral test for both user stories: asserts the default domain resolves, the opm-integration-test project exists in it, and the per-project group exists; exits non-zero otherwise. (Given-When-Then: given a freshly deployed Keystone, when tox -e create runs as admin, then the project and its group are observable via the SDK.) Optional later extension (kept out of the scaffold for reliability): assert the quotaclass extra property equals basic.
• Regression guard for AC 4 — after adding the files, run tox -e test and confirm the discovered test count is unchanged (the existing test/unit suite still runs; verify.py is not collected because its name does not match test*.py and test/integration/ is not a package).
• No unit-test changes: create.py is intentionally left untouched, so test/unit/test_create.py is unaffected.

Documentation Plan

• test/integration/README.md — new; the sole doc deliverable (prerequisites, make usage, env overrides, what is asserted, ephemeral-credentials note), as described in the Change Set. Top-level README.md and CHANGELOG: none (the repo has no CHANGELOG; the top-level README is a single doc link and the issue does not ask to change it).

Verification Commands

• tox -e test — existing unit suite still passes; confirm count unchanged (AC 4).
• python -m unittest discover ./test -v — confirm no integration tests are listed.
• flake8 and black --check . — verify.py is lint/format-clean (run in check).
• yamllint test/integration/controlplane.yaml playbooks/pre-integration.yml playbooks/test-integration.yml .zuul.yaml — YAML passes (truthy enabled).
• make integration on a Docker-capable host — full cycle: provision → create → verify → teardown (AC 1); make integration-up then make integration-down for local debug (AC 2). (Pin FORGE_REF first.)
• git status --porcelain after make integration — confirm test/integration/clouds.yaml is untracked/ignored (AC 6).

Risks & Open Questions

• create.py needs load-balancer_member (Open decision 3, primary risk). On a bare forge Keystone the proposed create command raises an uncaught KeyError at create.py:236/:423. The plan resolves this in deploy_keystone.sh by pre-creating member + load-balancer_member (matching what a real OSISM/Octavia cloud has). Alternative (out of current scope): harden create.py to tolerate missing roles (CACHE_ROLES.get(...)), which would change production behavior and need its own unit test. Please confirm pre-provisioning is the desired approach.
• Project name reconciliation. The issue's create command (prefix default on) would produce default-opm-integration-test, but its verify snippet looks for opm-integration-test. The plan passes --nodomain-name-prefix so the name is exactly opm-integration-test (deterministic, matches the verify snippet). Flip both create-name and verify-name together if the prefixed name is preferred.
• Does forge make deploy-infra build operators from source? The Quick Start installs no Go toolchain, so the plan assumes deploy-infra pulls prebuilt operator images. If it builds from source, playbooks/pre-integration.yml must additionally install Go. Verify against the pinned FORGE_REF during implementation.
• forge ref pinning (Open decision 2). FORGE_REF must be set to a concrete known-good forge commit SHA/tag at implementation time; left as a clearly-marked placeholder in deploy_keystone.sh.
• nip.io DNS dependency. Resolution of keystone.127-0-0-1.nip.io → 127.0.0.1 requires outbound DNS on the CI node; if blocked, add a /etc/hosts entry (127.0.0.1 keystone.127-0-0-1.nip.io) in pre-integration.yml/run.sh as a fallback.
• tox.ini and .gitignore edits are outside the issue's deliverable table but are required for AC 1 (credentials reach tox -e create) and AC 6 (no committed creds). Both are minimal; surfaced here for visibility.
• Resources / nodeset (Open decision 5). ubuntu-noble mirrors netbox#263; confirm the OSISM ubuntu-noble flavor meets forge's ~7 GB RAM / 2 vCPU single-node target.
• accept_ra apply step. netbox#263's surfaced diff shows only the drop-in copy; the plan adds sysctl --system to apply it within the run. Match netbox#263's exact apply mechanism if it differs.
• Pre-existing dead-code style note (not addressed, not in scope): create.py uses bare except: in try_assign_role* (flagged by the Python Exception Handling pattern) — left untouched per surgical-change rules.

Status

STATUS: PLAN_READY
(Executable as written and covers every deliverable in the issue's file-by-file table plus all five Open decisions. The one behavioral blocker — missing load-balancer_member role — is resolved inside the harness without modifying create.py; the create.py-hardening alternative and the forge-build assumption are flagged for confirmation but do not block implementation.)

---

Implementation plan generated by planwerk-agent 84299e2 implement with Claude

[berendt]

Implementation Report (issue #293)

Work Breakdown Coverage

The issue's "Proposed deliverables (file-by-file)" table is the work breakdown; every row is delivered, plus the two supporting edits the plan flagged.

• test/integration/deploy_keystone.sh — done — Commit 474b422. Clones forge @ pinned commit 35ef73a (HEAD; forge has no tags), make install-test-deps, KIND_HOST_PORT=8443 WITH_CONTROLPLANE=true make deploy-infra, applies the Keystone-only CR, waits Ready, emits clouds.yaml, and pre-creates the member/load-balancer_member roles create.py requires.
• test/integration/run.sh — done — Commit 27c41c5. Orchestrates deploy → tox -e create → verify.py; trap-based teardown that only deletes a cluster this run created, with a diagnostics dump on failure.
• test/integration/verify.py — done — Commit 27c41c5. Standalone openstacksdk script (not test_*.py); asserts the domain, project, and per-project group exist; exits non-zero on any failed assertion.
• test/integration/controlplane.yaml — done — Commit 474b422. Keystone-only ControlPlane CR verbatim from the issue.
• test/integration/README.md — done — Commit 0ceb8e9. Prerequisites, make usage, env-override table, what is asserted, ephemeral-credentials note.
• Makefile (new) — done — Commit f93563e. integration / integration-up / integration-down.
• playbooks/pre-integration.yml — done — Commit 234aee2. ensure-pip/ensure-pipenv/ensure-docker roles, the IPv6 accept_ra=2 sysctl fix, libldap/libsasl build deps.
• playbooks/test-integration.yml — done — Commit 234aee2. Installs tox+openstacksdk into a venv, runs make integration.
• .zuul.yaml — done — Commit 0d182e0. New openstack-project-manager-integration job (ubuntu-noble, both playbooks, timeout: 2400) added to periodic-daily only.
• Supporting (plan, not in issue table): tox.ini passenv and .gitignore clouds.yaml — done — Commit 27c41c5.

Open decisions resolved: (1) toolchain via forge make install-test-deps; (2) FORGE_REF pinned; (3) roles pre-provisioned + --nodomain-name-prefix; (4) periodic-daily only; (5) ubuntu-noble.

Acceptance Criteria

• make integration provisions kind + Keystone, creates one project as admin, asserts via SDK, tears down on every exit path
  • Status: satisfied
  • Evidence: test/integration/run.sh:38-58 (trap cleanup EXIT, teardown + diagnostics, then deploy/create/verify); Makefile:9 → run.sh. Not executed end-to-end here (requires a Docker/kind host; not available in this session).
• make integration-up / make integration-down work for local debugging
  • Status: satisfied
  • Evidence: Makefile:13 (deploy_keystone.sh), Makefile:17 (kind delete cluster --name forge); both validated with make -n.
• New openstack-project-manager-integration job runs in periodic-daily; check unchanged and stays fast
  • Status: satisfied
  • Evidence: .zuul.yaml; asserted programmatically — integration job present in periodic-daily.jobs, absent from check.jobs.
• The verifier is not collected by tox -e test
  • Status: satisfied
  • Evidence: python -m unittest discover ./test -v lists only unit.* modules; test/integration/ has no __init__.py and verify.py is not named test_*.py. Full suite ran 129 tests, OK.
• test/integration/README.md documents prerequisites and usage
  • Status: satisfied
  • Evidence: test/integration/README.md (single H1, prerequisites, make usage, env-override table, assertions, credentials).
• No real credentials are committed; the generated clouds.yaml is ephemeral
  • Status: satisfied
  • Evidence: .gitignore:5 (git check-ignore confirms test/integration/clouds.yaml ignored); deploy_keystone.sh writes it at runtime under umask 077; run.sh rm -fs it on exit.

Commits

• 474b422 Add forge Keystone deploy script and ControlPlane CR
• 27c41c5 Add integration run.sh orchestration and SDK verifier
• f93563e Add Makefile integration targets
• 234aee2 Add Ansible pre/test playbooks for integration job
• 0d182e0 Add periodic-daily Zuul integration job
• 0ceb8e9 Document the integration test harness

Local verification

• python -m unittest discover ./test (Python 3.13 venv with requirements.txt) — pass (129 tests, OK; only unit.* collected)
• flake8 . — pass
• black --check . — pass (14 files unchanged)
• yamllint . — pass (zero warnings on new files; only pre-existing warnings on tox.yml/mypy.yml/add-to-project.yml)
• mypy test/integration/verify.py — pass (no issues)
• bash -n deploy_keystone.sh && bash -n run.sh — pass
• make -n integration|integration-up|integration-down — pass (recipes tab-indented, resolve correctly)
• git check-ignore test/integration/clouds.yaml — pass (ignored)
• tox -e test on the host directly — skipped: system Python is 3.14 and dynaconf==3.3.0 caps at <3.14; re-ran the identical suite under a 3.13 venv instead (pass above)
• make integration end-to-end — skipped: no Docker/kind-capable host in this session

Branch

• implement/issue-293-keystone-integration-test (committed, not pushed — the finalize step opens the PR)

Deviations from the issue

• forge pin is a commit SHA, not a tag. forge publishes no tags (git ls-remote --tags is empty), so FORGE_REF pins HEAD commit 35ef73ae48d6814270af1674370d60ae410efdc7 — satisfies "pin a specific ref" (Open decision 2).
• --nodomain-name-prefix added to the create command. create.py defaults --domain-name-prefix on, which would name the project default-opm-integration-test, contradicting the issue's own verify snippet looking for opm-integration-test. Passing the flag makes the name deterministic and matches the verifier (flagged in the plan).
• Roles pre-provisioned in deploy_keystone.sh. A bare forge Keystone lacks load-balancer_member, which create.py indexes directly (CACHE_ROLES[role_name] at create.py:236/:423) → uncaught KeyError. The harness pre-creates the roles instead of modifying create.py (keeps the production code untouched; Open decision 3).
• Two supporting edits outside the issue's deliverable table: tox.ini passenv = OS_CLIENT_CONFIG_FILE (so tox -e create sees the credentials, required for AC 1) and .gitignore test/integration/clouds.yaml (required for AC 6). Both minimal.

Status

STATUS: DONE_WITH_CONCERNS

Every work package and Acceptance Criterion is implemented, committed, and statically verified. The reservation a reviewer should see: the end-to-end make integration path (kind + forge Keystone) could not be exercised in this session — no Docker/kind-capable host — so the live deploy/create/verify flow is verified by construction and static checks, not by a real run. Two runtime assumptions therefore remain unconfirmed against a live cluster: the keystone-admin Secret name/key, and that forge's bare Keystone needs exactly the two pre-created roles. Both are isolated to the harness scripts and are what the new periodic-daily job will exercise.

---

Implementation report generated by planwerk-agent 84299e2 implement with Claude

[berendt]

Review Report

Resolved

• deploy_keystone.sh silently overwrites the user's real clouds.yaml — Root cause: CLOUDS_FILE honored OS_CLIENT_CONFIG_FILE, and make integration-up runs the script directly with the operator's environment, so the unconditional cat > would truncate a real config. Added an early fail-fast guard that refuses to write when CLOUDS_FILE already exists and is not the dedicated in-tree ${HERE}/clouds.yaml. This still permits the documented throwaway-path override (a non-existent target proceeds) but protects an existing real config. (folded into 68776e6 Add forge Keystone deploy script and ControlPlane CR)
• Admin password string-interpolated into a double-quoted YAML scalar — Root cause: a minted password containing " (breaks parsing) or \ (escape-processed to a wrong value) corrupted the heredoc-generated clouds.yaml. Replaced the heredoc with an inline python3 + yaml.safe_dump writer (matching the file's existing python-heredoc idiom and PyYAML dependency via openstacksdk), passing values through the environment. Verified round-trip of an adversarial password (", \, tab, ') reproduces the exact bytes. (folded into 68776e6 Add forge Keystone deploy script and ControlPlane CR)
• Cluster-ownership check treats CLUSTER_NAME as a regex — Root cause: grep -qx "${CLUSTER_NAME}" anchors the line but still interprets the pattern as a basic regex, so a name with metacharacters (e.g. forge.dev) can mis-detect ownership and leak or wrongly delete a cluster. Changed to grep -Fqx for fixed-string matching. Verified forge.dev no longer falsely matches forgeXdev while exact matches still work. (folded into b7bd7ec Add integration run.sh orchestration and SDK verifier)

Skipped

• (none — all three findings were real and resolved)

No regression tests were added: all three fixes live in integration test-harness shell scripts (test code), and none could plausibly be covered by a unit/integration test, which would require provisioning a kind cluster + forge + network. This is explicitly exempted by the regression-guard rule.

Diff summary

• Files: test/integration/deploy_keystone.sh, test/integration/run.sh
• Approx lines added/removed: +40/-14

Verification

• bash -n test/integration/deploy_keystone.sh → OK; bash -n test/integration/run.sh → OK
• YAML serializer round-trip with adversarial password (p"a\sw⇥x'y) → round-trip password matches: True; old heredoc would have emitted broken password: "p"a\sw…"
• Guard truth table → in-tree default always proceeds; existing operator config refuses; throwaway path proceeds when absent, refuses when present
• grep -Fqx 'forge.dev' → no match against forgeXdev (correct), matches exact forge.dev (correct); old grep -qx false-matched
• Full make integration / make integration-up not run: requires a kind cluster, the forge toolchain, and network — not available in this environment.

Status

STATUS: DONE

---

Review report generated by planwerk-agent 84299e2 implement with Claude