[Commercialization Gap] Productionize tenancy, durability, observability, release, and support

[seonghobae]

Buyer problem

TEPP has strong repository-level contracts and partial PostgreSQL/API/security slices, but it is not yet an operable multi-tenant product. Buyers cannot install a supported release, authenticate users/services, run durable jobs, monitor SLOs, recover from failures, upgrade safely, or obtain a complete procurement/assurance evidence pack.

Product outcome

Turn the scientifically validated product vertical into a production service that can run standalone and as a modular CWL service. The service must preserve TEPP's purpose-bound PII utility without blanket masking and must remain honest about CSAP/SOC 2 readiness versus certification.

Required capabilities

Deployment and runtime

• versioned public API/CLI and generated clients;
• durable analysis queue, status lifecycle, cancellation, retry, dead-letter and idempotency;
• PostgreSQL reference store plus object storage for immutable source/model/export artifacts;
• Compose deployment first, with Kubernetes-ready probes/configuration and no hidden monolithic assumptions;
• production TLS and authenticated service boundaries;
• bounded resource/capacity controls and backpressure.

Identity, tenancy, and privacy

• Keyverse-compatible OIDC/OAuth/JWT and service identity adapter;
• tenant isolation through PostgreSQL RLS or equivalent mandatory enforcement;
• purpose-, role-, lifetime-, and field-bound access;
• protected re-identification/identity-mapping path without disabling legitimate PII-dependent work;
• retention, legal hold, export, deletion, backup, and restore controls with evidence.

Operability and assurance

• OpenTelemetry traces/metrics/logs without source-text or credential leakage;
• SLOs for submission, queue latency, run completion, artifact retrieval, error rate, and recovery;
• runbooks for provider outage, database/object-store outage, corrupted artifact, model regression, GPU loss, queue backlog, security incident, and rollback;
• capacity/load/soak/failover tests and measured hot-partition mitigation;
• immutable audit/provenance, SBOM, dependency/license evidence, signed build provenance, reproducible release artifacts, and vulnerability-response process;
• backup/restore, point-in-time recovery, migration upgrade/downgrade rehearsal, and disaster-recovery evidence.

Acceptance criteria

• A clean environment can install and run a versioned release from documented Compose commands without repository-internal credentials.
• Liveness, readiness, startup, and dependency health are separate and fail with content-redacting responses.
• Durable jobs survive worker/API restarts; duplicate submission and retry produce no duplicate chargeable/scientific artifacts.
• Cross-tenant reads/writes, forged tenant headers, invalid JWT audience/signature, stale grants, and purpose escalation fail closed.
• External consumers use versioned APIs/events only; no CWL product reads TEPP application tables directly.
• OpenTelemetry spans connect request → queue → compute → validation → artifact → result without copying source text, identity mapping, provider keys, or unrestricted model output.
• Capacity tests cover realistic corpus sizes and concurrent tenants; queue depth, CPU/GPU/RAM/VRAM, database/object-store, and backpressure limits are documented.
• Backup/restore and corrupted-manifest rehearsals prove restored artifacts are unusable until integrity, tenant, cutoff, and provenance checks pass.
• Every release has version/CHANGELOG consistency, signed tag/artifacts, SPDX or CycloneDX SBOM, build provenance, checksums, migration manifest, rollback steps, and exact protected-head evidence.
• Security/threat-model tests cover SSRF, request smuggling/framing, hostile Unicode/JSON/archive, prompt injection, dependency compromise, log/export leakage, privilege escalation, and denial-of-service bounds.
• CSAP/SOC 2/ISO/NIST mappings identify control owner, implementation, runtime evidence, test, known limitation, and deployment/customer responsibility; no certification or attestation is claimed without independent evidence.
• Support policy defines compatibility, upgrade window, deprecation, incident severity, response/communication, data portability, and vulnerability disclosure.
• A release candidate executes the end-to-end product issue with exact-head CI/security/scientific evidence and no unresolved release blocker.

Commercial proof

Prepare a buyer-facing but source-backed package containing architecture, data flow, threat model, deployment sizing, SLOs, security/privacy controls, validation evidence, limitations, pricing/meter hooks, migration/rollback, and an anonymized reproducible demo. Every claim must link to protected-main or released evidence.

Scope boundary

This issue does not ask the repository to self-certify CSAP/SOC 2, store payment-card data, silently mask all PII, or weaken scientific gates for availability. A failed scientific-integrity gate is an operational failure, not a warning to hide.

Authority

Use OPERABILITY, COMPLIANCE_READINESS, SECURITY, THREAT_MODEL, ADR 0011/0013/0014, and the latest authoritative standards recorded in the repository doctoring register.