feat(dev): add local model serving tool

[binaryaaron]

Summary

• Add a source-tree developer tool that compiles and manages local vLLM inference services from pinned TOML profiles.
• Support stock Hugging Face generation models and NVIDIA GLiNER or GLiNER2 through the pinned external vLLM Factory integration.
• Preserve pure compilation, digest-protected plans, task-aware capability probes, PID-reuse protection, and bounded process-group cleanup.
• Preserve vLLM API-key authentication on the Factory detector compatibility route.
• Ship seven profiles: NVIDIA GLiNER, GLiNER2, TinyLlama, GPT-OSS 20B and 120B, Qwen3 30B A3B Instruct, and NVIDIA Nemotron 3.5 Lightning.
• Add deployment guidance and a checked-in NVIDIA GPU container image that runs the same lifecycle inside the container boundary.

Scope and lifecycle

The tool owns one deployment domain: a managed local-process vLLM server.

• The v2 input schema has four sections: task, model, vllm, and local.
• Lifecycle commands are compile, launch, probe, status, and stop.
• Factory-backed detection and stock vLLM generation share one compiler and lifecycle.
• The Factory detector route enforces VLLM_API_KEY before request parsing or inference and returns 401 for missing, malformed, or incorrect credentials.
• The supplied container runs that unchanged local-process lifecycle. It does not add a second compiler path.
• Docker-specific compiler branches, native Transformers GLiNER serving, and Hugging Face cache discovery remain outside the design because no shipped profile requires them.
• Plan SHA-256 digests detect accidental storage or transport changes. They do not authenticate plan writers or prove semantic validity.

The tool remains under tools/ and is not part of the distributable Anonymizer package. It does not attach to existing endpoints or manage remote compute.

Validation

Validation on current HEAD 8562bdd:

• Focused inference-service and vLLM Factory tests: 58 passed and 3 expected skips.
• make format-check: passed.
• make typecheck: passed.
• make test: 1,341 passed and 3 expected skips.

Earlier full-branch validation on rebased HEAD 836a13f:

• make check: passed, including formatting, lint, ty, lock consistency, and SPDX checks.
• make docs-build: strict build passed.
• Range-diff matched all 17 product commits exactly after rebasing onto current main.
• Every rebased product commit has a DCO trailer and a valid repository ED25519 signature.

Runtime evidence from the patch-equivalent pre-rebase tree:

• Built tools/inference_service.Dockerfile as a real GPU image. The packaged CLI exposed all lifecycle commands and the vLLM Factory compatibility doctor passed.
• Co-hosted pinned NVIDIA GLiNER and GPT-OSS 20B services on one A100 80GB. Capability probes passed for dynamic labels, offsets, scores, and chat completions.
• Processed all 25 records in docs/data/NVIDIA_synthetic_biographies.csv with zero workflow failures.
• Published a sealed PR-versus-main pair to the W&B scorecard. The PR candidate completed in 458.09 seconds versus 558.92 seconds for the August 14 main baseline.
• Both managed services stopped through their receipts with cleanup_complete=true; final A100 memory use was 0 MiB.

Fresh NVSkills validation remains requested so generated skill artifacts and signatures bind to the current commit chain.

Fixes #252

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Post-rebase A100 dogfood completed on HEAD 1dd931ea8b7cf558c22b3cd2c2916d746492ca19 using the managed compiled plans and local endpoints for nvidia/gliner-pii plus pinned nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (33268dc8a6da85a56be2b12241453e4e1237bbe1).

• Full benchmark: 50 records across biographies redact+evaluate and legal hash+evaluate
• Both cases completed on their first attempt
• Model requests: 200 total, 200 successful, 0 failed
• Pipeline errors: 0 stage, 0 NDD workflow, 0 model workflow
• Replacement completeness: 0 missing final entities and 0 missing final values
• Managed shutdown: both services terminated with cleanup_complete: true
• Final A100 state: 0 MiB used

The analyzer reproduced the seven previously verified measurement false positives (1 biographies, 6 legal). No leak-metric changes are included in this hoster PR.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added pinned GPT-OSS 120B, GPT-OSS 20B, and Qwen3 30B A3B Instruct profiles on rebased HEAD 9d9536b990738e409e2ea335dd7d4b051dcd9c74. GPT-OSS 120B loaded successfully on the A100 through vLLM 0.27.1; the managed capability probe passed after accounting for reasoning-token output, and cleanup returned GPU use to 0 MiB. Fresh validation: 1,260 local tests passed, make check passed, strict docs build passed, and GitHub tests pass on Python 3.11, 3.12, and 3.13. NVSkills live evaluation is running.

[binaryaaron]

Medium hoster simplification is now on ccfedc8a4aeeb353d90bfc7692b4cc10a2c2c887.

• Removed Docker, native Transformers GLiNER, and cache-discovery branches.
• Compacted profiles to intent schema v2 with task, model, vllm, and local sections.
• Kept compile/launch/probe/inspect/cancel, plan digests, receipts, secrets, PID identity checks, bounded cleanup, LoRA, vLLM controls, and Factory-backed detection.
• Net hoster reduction: 988 production lines. Dedicated tests: 1,705 to 852 lines.
• Local validation: 1,226 tests passed, 35 focused inference-host tests passed, make check passed, and strict docs passed.
• A100 dogfood: local GPT-OSS 20B plus NVIDIA GLiNER passed managed probes; a two-record Anonymizer run found and redacted four PII entities with zero failed records; both receipt-owned services canceled cleanly; final GPU use was 0 MiB.

The implementation began as an Engineer Arc candidate. Parent review rejected its over-aggressive test reduction, restored lifecycle-focused behavioral coverage, removed a stray compiler branch, and reran all validation before push.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Added the local-model deployment docs and checked-in GPU container image in commit 0034b8f. Validation: full suite 1,226 passed; make check passed; strict docs build passed; Docker image build passed; packaged CLI and vLLM Factory compatibility doctor passed inside the image.

[binaryaaron]

/nvskills-ci

[binaryaaron]

Modern Python boundary refactor is now on HEAD ff40e2b.

• Closed FactoryPlugin typing now flows through intent, compiler, runtime, Factory integration, and adapter boundaries.
• Exhaustive match/assert_never dispatch covers owned task, environment, and plugin unions.
• Internal multi-field products are frozen slotted dataclasses; Factory metadata is one immutable exhaustive specification.
• CommandSpec owns redacted rendering, secret-source inventory, and strict execution-time resolution.
• HTTP client ownership is context-managed, compiler argument construction is more declarative, and obsolete delegation/test-loader seams are removed.
• CLI translation remains narrow while preserving exit 125 for known filesystem, transport, integrity, runtime, validation, TOML, and empty-source-revision failures.

A Terra implementation and independent-review cycle found and resolved plugin-spec duplication, secret ownership, positional chunk tuples, and stale delegation seams. Primary-agent review then caught the empty-source-revision exit regression and the remaining manual HTTP ownership flag before push.

Validation on the exact signed commit chain:

• focused inference-service tests: 36 passed, 3 skipped
• make check: passed, including ty
• make test: 1,227 passed, 3 skipped
• all four commits have DCO signoff and valid repository ED25519 signatures

No bundled skill files changed, so no new /nvskills-ci request is needed.

[binaryaaron]

Semantic cleanup is now on HEAD 7cc0c58037e64ae31f8f8ccfdfb5e5be3f817d9a.

• Reframed the draft input as LocalInferenceServiceSpec; profiles now use inference-service.local-spec/v2 in place, with no compatibility layer or schema-v3 migration.
• Renamed endpoint and compatibility concepts to EndpointAddress and CompatibilityAssessment; lifecycle commands are now status and stop.
• Removed redundant ServiceCompilation, CommandArgument, Generation.chat, RunPlan.runtime, intent_digest, declared_capabilities, and duplicate readiness coordinates.
• Preserved the pure spec-to-plan compiler, plan digest, secret-safe command resolution, task-aware readiness probing, process-group cleanup, and separate parent/child runtime boundaries.
• Tightened process identity: launch now fails closed and cleans up if a PID start marker cannot be recorded.
• Updated the deployment docs, README, devnote, profiles, and bundled Anonymizer skill to the new vocabulary.

A Terra reviewed-engineering Arc performed implementation, independent review, bounded fix-up, and confirmation. Review initially blocked on the PID-marker fallback and stale shipped instructions; both were fixed, and confirmation accepted the exact final tree with no findings.

Validation on that tree:

• focused inference-service tests: 24 passed
• make test: 1,229 passed, 3 skipped
• make check: passed, including formatting, lint, ty, lock, and SPDX checks
• make docs-build: passed
• final commit has DCO signoff and a valid repository ED25519 signature

[binaryaaron]

/nvskills-ci

[binaryaaron]

Follow-up council warnings addressed in abf8a7e:

• Bound the missing-start-marker cleanup path across TERM and KILL waits; an unreapable child now returns the normal diagnostic with cleanup_complete=false instead of hanging.
• Made plan validity explicit: SHA-256 detects accidental storage/transport changes only. The developer tool trusts plan writers and does not authenticate, sign, or recompile plans for semantic proof.
• Replaced stale reconnectable/proves/intent wording in CLI help, runtime/compiler docstrings, README, and deployment docs.

Validation:

• focused inference-service tests: 25 passed
• make check: passed
• make docs-build: passed
• commit has ED25519 signature and Signed-off-by trailer

[binaryaaron]

Post-fix A100 dogfood passed on HEAD abf8a7e.

• Managed services: pinned NVIDIA GLiNER on 127.0.0.1:8001 and pinned GPT-OSS 20B on 127.0.0.1:8000; GPT-OSS used the documented co-hosting cap of 0.75 GPU memory utilization.
• Launch receipts recorded exact PID/start markers. GLiNER passed dynamic-labels, offsets, and scores; GPT-OSS passed chat-completions.
• Required preview: 2 records, 14 entities across 8 labels, 0 failed records.
• Full Anonymizer Replace + Substitute run: 2/2 records transformed, 0 failed records, and no detected source value remained verbatim in substituted text.
• Local serving handled 24 chat-completion requests with no ERROR, Traceback, or Exception entries in either service log.
• Receipt-based stop: both services terminated with cleanup_complete=true; both status receipts report stopped; final A100 usage is 0 MiB.

Run artifacts are retained locally under /tmp/anonymizer-abf8a7e-full-run-U7pB7p.

[binaryaaron]

/nvskills-ci

[binaryaaron]

/nvskills-ci