test: nvmet/nvmet-tcp integration suite on a Talos/QEMU cluster

[noctarius]

An end-to-end integration suite that boots a Talos cluster in QEMU and drives a
real NVMe-oF fabric on it, so atlas's detection is exercised against a kernel
rather than against a fixture.

It lives in test/integration/ as its own Go module, so its Kubernetes and QEMU
dependencies stay out of atlas-lib and out of the driver it exercises.

What is here

• fabric/ — the nvmet and nvmet-tcp core. target.go builds a subsystem
  and its TCP port through configfs; two targets sharing an NQN need disjoint
  CntlIDMin/CntlIDMax ranges, which is what makes the host merge them rather
  than reject the second controller as a duplicate cntlid. initiator.go
  connects by writing an options line to /dev/nvme-fabrics and driving
  delete_controller/rescan_controller, the same mechanism atlas's own
  FabricsConnector uses, so the fabric is established the way the code under
  test establishes one — and no nvme-cli is needed in the pod.
• cluster/ — talosctl's QEMU provisioner, plus each node's QEMU human
  monitor, so a host can be faulted from outside it.
• controlplane/ — a control-plane simulator generated from
  shared/openapi.json, so the compiler is what notices when the control plane
  grows a path or renames one. 9 of 98 endpoints are implemented; the rest answer
  501. The generated files are committed and make generate-check fails on drift.
• suites/ — TestSmoke, and TestDefect_ControllerNotContributing, which
  forces the defect that costs the most to reach by hand: two targets on two
  nodes sharing one NQN, one exporting the namespace and one exporting nothing,
  leaving a host with a single subsystem, two live controllers, and only one of
  them serving a path.

The cluster-booting tests are gated behind SB_INTEGRATION, so they skip
themselves everywhere else. make preflight checks the tools and privileges a
run needs and pre-warms sudo, since the QEMU provisioner needs root and the
prompt would otherwise land invisibly mid-run.

The second commit adds the nightly OpenAPI export that keeps
shared/openapi.json from going stale silently. It is separable from the suite;
say the word and it can move to its own pull request.

History

Force-pushed onto a clean base. The previous head had no merge base with main
— its first commit was a whole-repo snapshot, which is why rebasing it conflicted
on essentially every file. The framework was re-applied onto current main as
two commits instead. No review comments existed on the old head.

Verification

• Builds and vets clean against current atlas-lib; the only wiring change was
  go mod tidy, which bumped two indirect dependencies.
• go test ./... green for the four packages that do not need a cluster.
• make generate-check passes: shared/openapi.json is unchanged, so the
  committed simulator is current.
• gofmt clean, and the house-style quality gate passes.

Not yet verified: no Talos cluster has been booted since the rebase, so the
cluster-booting paths are unexercised on this base. That is what the
Integration: Tests workflow is for.

Known gaps

• cluster/kubectl.go:15 trips the terminology gate on Kubectl, the
  godoc-mandated leading identifier of the method it documents. Backticks would
  reach pkg.go.dev as literal characters, which is the same reason the gate
  already exempts a package clause. Resolving it properly means extending
  EXEMPT_PHRASES in check-terminology.py, which is shared tooling and left
  out of this change.
• The new module is not in repo_lint.yaml's matrix, which still covers three
  modules. integration_test.yaml runs go vet and the local Makefile has a
  lint target, but golangci-lint does not run in CI for it.

[Copilot]

🟡 Changes recommended

There are a couple of concrete workflow/house-style breakages in the new integration harness (sudo handling and terminology gating) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a new end-to-end NVMe-oF integration test harness under test/integration/ that boots a Talos Kubernetes cluster in QEMU, stands up a real nvmet/nvmet-tcp fabric inside it, and validates atlas-lib’s sysfs-based NVMe-oF inspection against kernel state (not fixtures). It also adds automation to keep the shared control-plane OpenAPI spec in sync.

Changes:

• Introduce a standalone Go module (test/integration) with a Talos/QEMU cluster provisioner, NVMe-oF fabric driver, and integration suites (smoke + “controller not contributing” defect reproduction).
• Add a generated control-plane simulator (cpsim) based on shared/openapi.json, plus stubs for unimplemented endpoints and unit tests for the simulator and monitor protocol parsing.
• Add CI workflows for running the integration suite and for nightly OpenAPI export + PR creation, plus a helper script to export/summarize spec changes.

File summaries

File	Description
test/integration/suites/smoke_test.go	Smoke suite to validate kernel modules/configfs access and QEMU monitor faulting.
test/integration/suites/defect_test.go	Reproduces and asserts atlas-lib’s “controller not contributing” defect from captured sysfs state.
test/integration/Makefile	Local build/vet/lint/test entrypoints and preflight checks for required tooling/privileges.
test/integration/go.mod	New standalone integration-test module with a replace to the repo’s atlas-lib/.
test/integration/go.sum	Dependency lockfile for the integration module.
test/integration/.gitignore	Prevents accidental commit of Talos-generated configs containing sensitive material.
test/integration/fabric/target.go	Configures nvmet subsystems/ports via configfs and backs namespaces with loop devices.
test/integration/fabric/initiator.go	Connects/disconnects/rescans controllers via /dev/nvme-fabrics and sysfs triggers.
test/integration/fabric/nodeshell.go	Privileged per-node pod “shell” for running commands against node kernel/configfs.
test/integration/fabric/sysfs.go	Dumps node NVMe sysfs to a text snapshot and reconstructs it locally for resolvers.
test/integration/fabric/sysfs_test.go	Unit tests validating sysfs reconstruction and replay against an existing fixture.
test/integration/cluster/talos.go	Talos/QEMU cluster lifecycle: create/destroy, config patching, kubeconfig fetch, diagnostics.
test/integration/cluster/talos_test.go	Tests for cluster-name length/path constraints (Unix socket sun_path limits).
test/integration/cluster/monitor.go	QEMU HMP monitor client for fault injection (freeze/thaw/link down/power off).
test/integration/cluster/monitor_test.go	Unit tests for monitor protocol framing/echo-stripping/netdev discovery.
test/integration/cluster/kubectl.go	Thin kubectl-based Kubernetes access layer (apply/exec/waits).
test/integration/controlplane/connect.go	Builds /connect responses matching control-plane conventions, including secrets-in-commandline behavior.
test/integration/controlplane/state.go	In-memory state model for clusters/nodes/pools/volumes used by the simulator.
test/integration/controlplane/server.go	HTTP server wrapper with auth and DTO conversion helpers.
test/integration/controlplane/server_test.go	Drives simulator via atlas-lib’s client to ensure strict decode + behavior correctness.
test/integration/controlplane/handlers.go	Implemented control-plane endpoints (health/ready + node/pool/volume/connect reads).
test/integration/controlplane/cpsim.gen.go	Generated OpenAPI models/router/server interface used to enforce spec drift detection.
test/integration/controlplane/unimplemented.gen.go	Generated 501 stubs for every endpoint the simulator doesn’t implement.
test/integration/controlplane/gen.go	go:generate entrypoint wiring for codegen + stub generation.
test/integration/controlplane/tools.go	Tool pin for oapi-codegen via a tools build tag.
test/integration/controlplane/oapi-codegen.yaml	oapi-codegen config (models + std-http-server + overlay).
test/integration/controlplane/overlay.yaml	Spec overlay patching response/parameter gaps in the upstream OpenAPI export.
test/integration/controlplane/gen/main.go	Generator producing unimplemented.gen.go and validating handler/interface alignment.
shared/export-openapi.py	Exports the v2 OpenAPI spec from sbcli’s FastAPI app and summarizes structural diffs.
.github/workflows/repo_openapi_sync.yaml	Nightly job exporting shared/openapi.json, regenerating consumers, and opening/updating a PR.
.github/workflows/integration_test.yaml	CI workflow to boot Talos in QEMU (/dev/kvm required) and run the integration module tests.

Review details

Files not reviewed (1)

• test/integration/controlplane/unimplemented.gen.go: Generated file

Suppressed comments (1)

test/integration/cluster/talos.go:183

• When cfg.Sudo is true, talosctl is executed under sudo via CombinedOutput(). If sudo needs an interactive password prompt, it will block with the prompt hidden inside go test output capture, making the run look hung. It would be safer to fail fast with a clear error if sudo cannot run non-interactively (or credentials are not cached).

	if *cfg.Sudo {
		if _, err := exec.LookPath("sudo"); err != nil {
			return nil, fmt.Errorf("sudo not found, and the QEMU provisioner needs root: %w", err)
		}
	}

• Files reviewed: 28/31 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.