TypeTreeFlow

TypeTreeFlow is an LPSN-first type-strain genome acquisition and audit workflow. The current 2.2.42 release is a bounded maintenance release over v2.2.41. It preserves the reviewed-selection contract while failing closed on unresumed reviewed selections, aligning public workflow-stage status semantics, and isolating reviewed-selection continuation logic behind focused compatibility tests. The v2.2.41 release consolidated the full post-v2.2.40 delta: reviewed selection and independent download authorization, approval-attempt lifecycle and interruption recovery, atomic no-download projection, actual-presence completion, package/diagnostics/AI handoff, installed-wheel and curator-packet path contracts, and bounded download-smoke quality and safety controls. Easy, synthetic difficult, and fixed-cache/no-download Clostridium loops cover success, explicit-gap, and pressure behavior. The difficult loop is synthetic; the Clostridium loop is not strict completion; bounded real evidence proves only an authorized download plus barrnap slice. After the v2.2.41 release, the human user/reviewer accepted all 5/5 bounded benchmark interpretations; an AI agent only faithfully transcribed those decisions under explicit authorization and did not make them. The product-convergence Definition of Done is therefore satisfied under the bounded evidence standard defined in docs/product_direction.md. Coverage, provider, archive, and external-registration planning remains review-only or separately guarded; the release does not claim whole-genus completion, broad live-provider validation, production readiness, or complete scientific acceptance. It preserves the v2.2.39 P3f guarded strict-gating audit-only workflow, the v2.2.38 P3e manual-review audit-only pipeline, the v2.2.37 report-only ordering fix, the v2.2.36 audit-only reconciler package inclusion, the v2.2.35 compact audit-only report section, the v2.2.34 audit-only strict_reconciliation workflow outputs, the v2.2.32 offline reconciler audit mapper/writers, the v2.2.31 strict evidence reconciler model, the v2.2.30 BacDive compact report/package wording, the v2.2.29 BacDive source-audit top-level summary fields, the bounded BacDive v2 HTTP client, and the public tokens-only workflow from v2.2.28. The audit outputs are evidence/reconciler_audit.tsv, evidence/reconciler_summary.json, and evidence/reconciler_diagnostics.tsv. The strict_reconciliation run-state stage is inferred when the output triplet exists, and stage summaries include record_count, strict_count, candidate_count, conflict_count, gap_count, manual_review_count, diagnostic_count, and audit_only=true. The report section displays those count fields and audit_only, may show compact reconciler diagnostic-code counts, omits itself when outputs are absent, tolerates malformed summary JSON without failing report generation, and handles zero-count summaries. Packages copy any existing reconciler files without requiring the complete triplet, omit missing outputs gracefully, leave --failed-handoff behavior unchanged, and add one audit-only artifact_scope.tsv row per copied reconciler file with strict_scientific_deliverable=false. Reconciler package inclusion means audit availability, not strict scientific delivery; reconciler strict_count and strict_usable=true row values do not change completion metrics. Existing BacDive artifact-scope rows are unchanged, and raw/cache/private/env/tmp/sequence payloads remain excluded. The hook runs after stable selection/plan output and refreshes after the final post-download manifest write. Optional missing or malformed BacDive and BioSample inputs produce diagnostics and warnings, not core workflow failure. These surfaces are audit-only. They do not change manifests, selection, downloads, providers, completion metrics, package tiering, evidence-policy behavior, or strict gating. The pure reconciler records and reconcile_type_strain_evidence() function combine LPSN, NCBI/BioSample, BacDive, curated/archive, and selected-genome linkage evidence into strict_lpsn_confirmed, curated_strict_confirmed, authoritative_type_material_candidate, ncbi_type_material_candidate, likely_type_material_candidate, representative_non_type, conflict_blocked, insufficient_linkage, and missing_public_genome tiers. Strict upgrade requires LPSN type-strain equivalence, selected genome linkage to that equivalence set, and no conflict. BacDive alone and NCBI/BioSample alone never become strict, and conflicts block strict upgrade. The reconciler is offline and pure: tests confirm it does not read environment variables or open sockets. Strict tiers continue to come only from the reconciler model. The audit mapper/writers and workflow hook do not affect selection, manifests, downloads, completion metrics, report/package membership, or evidence-policy behavior. The current BacDive workflow contract remains an injectable client with simulated tests for timeout, rate-limit, schema drift, no-result, and 5xx handling. Endpoint construction covers /v2/culturecollectionno/{culturecollectionno}, /v2/taxon/{genus}/{species_epithet}, and /v2/fetch/{bacdive_id}. Explicit terms and citation confirmation are required to construct the live client; the client does not read environment variables, API keys, or cookies. Tests do not call the live BacDive API. BacDive enrichment remains candidate-only and audit-only, and raw BacDive payloads are not included in packages. The v2.2.25 skeleton, v2.2.24 configuration plumbing, v2.2.23 offline adapter contract, v2.2.22 offline BacDive model, and v2.2.21 artifact scope readability semantics remain valid.

AI-First Route

README is the user entry point. Start with docs/index.md, then use:

• docs/product_direction.md: product direction, current completion principles, capability tiers, and AI/core/human responsibility boundaries.
• docs/guide.md: operator commands and recipes.
• docs/reference.md: CLI JSON stdout, output layout, schemas, statuses, stable contracts, and handoff contract.
• docs/policy.md: scientific, provider, external-genome, workspace, results, completion, and species-checklist boundaries.
• docs/development.md: maintenance, tests, release gate, packaging, and hygiene.
• docs/architecture.md: current system design.
• docs/release_notes_v2_2_x.md: release history.

Compatibility entries retained for package tooling: docs/provider_automation_policy.md and docs/release_verification.md.

What TypeTreeFlow Is And Is Not

TypeTreeFlow uses LPSN-derived species checklists and type-strain tokens as the scientific anchor, then builds auditable NCBI/external/local evidence tables. It can plan candidates, review selections, run guarded downloads, extract 16S, run downstream ANI/phylogeny, summarize gaps, and package results.

Evidence levels such as strict_confirmed, likely_type_material, and representative_only must stay distinct. TypeTreeFlow does not treat representative, likely_type_material, provider proposals, provider plans, reference genomes, local query rows, or external request rows as strict confirmed type strains. Strict type-strain wording requires evidence tying the genome record to the species type-strain equivalence set.

Completion coverage and strict type evidence gaps are separate review claims. Evidence policy is a derived-view setting: strict is the default, while candidate and exploratory broaden report/completion wording without promoting weak evidence to strict confirmed type-strain status or changing selection, downloads, manifests, combined 16S FASTA, phylogeny inputs, or package membership.

It does not automate provider login, scraping, purchase, terms acceptance, credential processing, ATCC Genome Portal downloads, or unguarded NCBI downloads.

Root user examples are intentionally absent after cleanup. Fixtures under tests/fixtures/ are internal test data. The synthetic ATCC-style pilot FASTA is not a real ATCC genome.

Installation

python -m pip install -e ".[dev]"
python typetreeflow.py --version
typetreeflow doctor

See docs/guide.md for the clean deployment route and docs/reference.md for doctor readiness contracts.

Credentials belong in local untracked files or command-line options. Do not commit lpsn.env, API keys, provider credentials, downloaded archives, run outputs, package artifacts, or release evidence.

Output Workspace

Use explicit run directories outside the repository source root:

typetreeflow verify-genus Fusobacterium \
  --outdir <workspace>/runs/fusobacterium_plan \
  --dry-run

<workspace>/runs/ is for generated run outputs. Repository-root results/ is forbidden. typetreeflow_out/ is a legacy old default path only.

Recommended v2.2.42 workflow

Create a plan-only review checkpoint. This offline example requires these reviewed local caches; --smoke-profile plan-only does not inject inputs or authorize downloads:

typetreeflow verify-genus Fusobacterium \
  --outdir <workspace>/runs/fusobacterium_selection \
  --lpsn-cache <local-lpsn.tsv> \
  --discovery-cache <local-discovery.tsv> \
  --smoke-profile plan-only

Review status and next action:

typetreeflow status --outdir <workspace>/runs/fusobacterium_selection
typetreeflow next-step --outdir <workspace>/runs/fusobacterium_selection

Review selection/user_selection.tsv from that checkpoint. Use --selection-tsv to submit that exact reviewed selection. Use --auto-accept-selection only for bounded exploratory smoke or deliberately accepted policy output; exploratory representative rows are not strict type-strain confirmations.

Use --evidence-policy strict unless a broader derived report view is explicitly intended. candidate and exploratory policy output remains evidence-first and does not claim strict completion. Review rrna/strict_16S.fasta, rrna/policy_16S.fasta, and report/artifact_scope.tsv when 16S artifact scope matters. AI consumers of a results package should read package-root artifact_scope.tsv first when that handoff copy is present.

After reviewing the generated file, first validate and project it locally in the same genus task. This exits successfully when validation and local projection succeed, but creates no approval and performs no download:

typetreeflow verify-genus Fusobacterium \
  --outdir <workspace>/runs/fusobacterium_selection \
  --resume \
  --selection-tsv <workspace>/runs/fusobacterium_selection/selection/user_selection.tsv

Only after that review, use a separate invocation to authorize guarded downloads:

typetreeflow verify-genus Fusobacterium \
  --outdir <workspace>/runs/fusobacterium_selection \
  --resume \
  --selection-tsv <workspace>/runs/fusobacterium_selection/selection/user_selection.tsv \
  --enable-downloads

Omitting --enable-downloads is the reviewed no-download projection above. --auto-accept-selection --enable-downloads remains the distinct opt-in for accepting software-generated selection without human review.

Optional guarded actions include --enable-biosample-entrez, --enable-entrez, --enable-ncbi-discovery, --enable-ncbi-taxonomy, --enable-fastani, --enable-phylo, --enable-expanded-discovery, and --discovery-cache. Expanded discovery and NCBI Taxonomy outputs are audit-only and do not create automatic 100% coverage.

BacDive enrichment is opt-in and candidate-only. With --enable-bacdive-enrichment, --bacdive-query-mode {tokens,species,both}, --bacdive-timeout-seconds, and --bacdive-max-queries, an injected/fake client or the public bounded live tokens path can write evidence/bacdive_enrichment.tsv, evidence/bacdive_diagnostics.tsv, and evidence/bacdive_source_audit.json, plus a bacdive_enrichment run-state stage. Without an injected client, public live mode is allowed only for --bacdive-query-mode tokens; species and both write bacdive_live_query_mode_not_allowed before any HTTP call. The live client does not require an API key, read environment variables, or use cookies. BacDive rows remain strict_confirmed=false with selected_genome_linkage=not_evaluated, and they do not change selection, manifests, completion metrics, downloads, or strict evidence policy results. The BacDive source audit reports top-level accessed_at_start, accessed_at_end, endpoint_count, lookup_call_count, fetch_call_count, last_http_status, stopped_reason, and docs_url, while retaining http_call_count, raw_payload_saved, raw_payload_policy, terms_url, citation_url, license_url, api_documentation_url, and field_information_url for backward-compatible consumers. Report summaries may include a BacDive Candidate Review section with compact Counts and Source audit reader summaries, and package-results --include reports includes evidence/bacdive_enrichment.tsv, evidence/bacdive_diagnostics.tsv, and evidence/bacdive_source_audit.json when they exist. Package artifact_scope.tsv rows for BacDive outputs remain scope=audit and strict_scientific_deliverable=false. Package README.md, README_failure.md, and handoff_index.md describe BacDive inclusion as audit availability, not strict scientific delivery. Raw BacDive payloads are not included.

BacDiveLiveClient supports explicit BacDive v2 HTTP review through an injectable transport. It constructs /v2/culturecollectionno/{culturecollectionno}, /v2/taxon/{genus}/{species_epithet}, and /v2/fetch/{bacdive_id} requests, and is constructed by the public workflow only behind --enable-bacdive-enrichment in tokens mode when no client is injected. It requires explicit terms and citation confirmation and does not read environment variables, API keys, or cookies.

Common Commands

typetreeflow --help
typetreeflow verify-genus --help
typetreeflow verify-release-genus --help
typetreeflow package-results --help

typetreeflow verify-genus Fusobacterium --outdir <workspace>/runs/fusobacterium --dry-run
typetreeflow verify-genus Fusobacterium --outdir <workspace>/runs/fusobacterium --resume
typetreeflow verify-genus Fusobacterium --outdir <workspace>/runs/fusobacterium --force

typetreeflow package-results \
  --outdir <workspace>/runs/fusobacterium \
  --delivery-dir <workspace>/deliveries/fusobacterium

Useful flags include --dry-run, --version, --resume, --force, --gtdb-metadata, --species-checklist, --prepare-selection, --selection-tsv, --selection-policy, --evidence-policy, --plan-provider-registration, --strains-per-species, --limit-selected, --query-genome, --query-16s, --email, --api-key, --skip-ani, and --skip-tree.

Release Verification

Use the maintained release workflow for release checks:

typetreeflow verify-release-genus Fusobacterium \
  --outdir <workspace>/runs/release/v2_2_x_release_verification \
  --email you@example.org

Release verification covers shared acquisition cache behavior, checkpoint and resume behavior, manifest.tsv, selection/user_selection.tsv, completion gap reports, package handoff, and audit-only expanded discovery: completion/gaps.tsv, completion/uncovered_species.tsv, completion/16s_gaps.tsv, completion/expanded_discovery_plan.tsv, completion/expanded_discovery_results.tsv, completion/expanded_discovery_history.tsv, completion/rejected_candidates.tsv, and completion/manual_supplement_hints.tsv.

The v2.2.41 release record covers reviewed selection with independent download authorization, approval-attempt recovery, atomic no-download projection, actual-presence completion, installed-wheel AI/package/diagnostic handoff, and bounded post-tag download-smoke safety and quality controls. Coverage, provider, archive-candidate, and external-registration routes remain review-only or separately guarded. The difficult loop is synthetic; Clostridium evidence is fixed-cache/no-download pressure rather than strict completion; the bounded real slice proves only download plus barrnap. AI pre-review support was not human curator acceptance at release time. A later governance record captures 5/5 bounded human-user acceptance, AI transcription only, and fulfillment of the product-convergence Definition of Done; docs/product_direction.md is authoritative for that current status. Release notes remain the historical record of what was true when v2.2.41 was published.

The v2.2.39 release record batches the P3f guarded strict-gating audit-only workflow. Release evidence is retained outside the repository at D:\Draft\TypeTreeFlow_release_evidence\p3f_1_strict_gating_evaluator_closure_20260724, and D:\Draft\TypeTreeFlow_release_evidence\p3f_integrated_offline_smoke_20260725. The expected P3f-2 path D:\Draft\TypeTreeFlow_release_evidence\p3f_2_strict_gating_report_package_closure_20260725 was unavailable; the verified closure is at C:\Users\14394\.codex\visualizations\2026\07\25\019f98da-6c33-7411-a253-a39b8eb07429\p3f_2_strict_gating_report_package_closure_20260725. This is bounded offline evidence and does not claim live-provider validation, downloads, external-tool execution, applied strict upgrades, or strict deliverable materialization. The standalone strict-gating evaluate CLI is a no-write dry run by default and may explicitly write only strict_gating_audit.tsv, strict_gating_summary.json, and strict_gating_diagnostics.tsv. It fails closed for synthetic evidence, unresolved conflicts, duplicate decisions, missing direct evidence, stale fingerprints, mismatches, weak-source-only support, malformed artifacts, and missing independent review. --strict-gating-dir passively adds the explicit Strict Gating Audit report section or copies validated audit artifacts under package strict_gating/ with audit-only artifact_scope.tsv rows. strict_gate_passed=true means only that evaluator guards passed; strict_deliverable_written=false and strict_upgrade_applied=false remain the contract. No manifest, selection, completion, reconciler tier, evidence-policy gating, provider/download, or report/package strict semantics change. The current source writes audit-only strict_reconciliation outputs evidence/reconciler_audit.tsv, evidence/reconciler_summary.json, and evidence/reconciler_diagnostics.tsv from local verify-genus artifacts; the strict_reconciliation run-state stage is inferred when the triplet exists. Stage summaries include record_count, strict_count, candidate_count, conflict_count, gap_count, manual_review_count, diagnostic_count, and audit_only=true. When local reconciler outputs exist, report/summary.md includes a compact Strict Reconciliation Audit section with record_count, strict_count, candidate_count, conflict_count, gap_count, manual_review_count, diagnostic_count, and audit_only; it may include compact reconciler diagnostic-code counts. Missing reconciler outputs omit the section, malformed summary JSON does not fail report generation, and zero-count summaries are handled. The hook runs after stable selection/plan output and refreshes after the final post-download manifest write. Optional missing or malformed BacDive and BioSample inputs produce diagnostics and warnings, not core workflow failure. package-results --include reports and --include all copy existing reconciler audit files to package evidence/, omit missing outputs gracefully, and copy partial triplets without failing package generation. Package artifact_scope.tsv receives one audit-only row per copied reconciler file. Reconciler package inclusion means audit availability, not strict scientific delivery; strict_count and strict_usable=true row values do not change completion metrics. Existing BacDive artifact-scope rows are unchanged, raw/cache/private/env/tmp/sequence payloads remain excluded, and --failed-handoff behavior remains unchanged. These surfaces are audit-only: they do not change manifests, selection, downloads, provider behavior, completion metrics, package tiering, FASTA content, default phylogeny inputs, live query scope, evidence-policy behavior, or strict deliverable semantics. Strict tiers continue to come only from the reconciler model. The still-valid v2.2.35 report section, v2.2.34 hook, v2.2.32 audit mapper/writers, v2.2.31 strict evidence reconciler model, v2.2.30 BacDive compact wording, v2.2.29 BacDive source-audit polish, v2.2.28 bounded public live tokens path, v2.2.27 BacDive live-client HTTP skeleton, v2.2.26 BacDive report/package handoff, v2.2.25 skeleton, v2.2.24 configuration plumbing, v2.2.23 offline BacDive adapter contract, v2.2.22 offline BacDive model, v2.2.21 artifact scope readability semantics, and v2.2.20 policy-aware artifacts and GTDB gating validations remain release verification evidence only; they do not claim full Clostridium strict completion or full-download validation.

External And Provider Workflows

Reviewed local external genomes enter through external_genomes.tsv. Provider planning writes review handoff tables only.

typetreeflow register-external-genomes \
  --external-genomes <reviewed_external_genomes.tsv> \
  --outdir <workspace>/runs/external_registration

typetreeflow plan-provider-registration \
  --provider-request provider_request.tsv \
  --outdir <workspace>/runs/provider_review

Provider planning does not automate ATCC, does not log in, does not scrape, does not download, does not install FASTA files, does not write manifests, and does not change completion metrics. It may produce provider/proposed_external_genomes.tsv for curator review.

For the internal Fusobacterium external pilot fixture, NCBI Assembly strict completion remains 16/17; external-inclusive strict completion is 17/17.

Downstream Analysis

Same-genome barrnap 16S, Entrez fallback, FastANI, and phylogeny are separately gated:

typetreeflow verify-genus Fusobacterium \
  --outdir <workspace>/runs/fusobacterium_downstream \
  --resume \
  --selection-tsv <workspace>/runs/fusobacterium_downstream/selection/user_selection.tsv \
  --email you@example.org \
  --enable-downloads \
  --enable-barrnap \
  --enable-entrez \
  --enable-fastani \
  --enable-phylo \
  --query-genome <query.fna> \
  --query-16s <query.16s.fasta>

Reports distinguish Same-genome barrnap 16S, Strict-usable 16S, Available 16S in candidate-inclusive outputs, Fallback warnings, and Strict blocking count. rrna/all_16S.fasta remains candidate-inclusive and is not a strict same-genome-only FASTA. Default phylogeny tree/alignment inputs remain compatibility outputs, not strict scientific deliverables. rrna/strict_16S.fasta is marked with strict_scientific_deliverable=true; strict-policy rrna/policy_16S.fasta may also carry that marker. report/artifact_scope.tsv records each artifact's machine-readable scope, including artifact_label, recommended_use, not_for, source_artifact, consumer_priority, and strict_scientific_deliverable.

Safety Model

Default maintenance uses documentation checks, dry runs, fake runners, local fixtures, and focused tests. Do not run live LPSN, NCBI, Entrez, provider lookups, datasets downloads, barrnap, fastANI, MAFFT, trimAl, or IQ-TREE unless explicitly asked.

Testing

Docs-only:

python scripts/check_docs_hygiene.py
pytest tests/test_docs_consistency.py tests/test_docs_hygiene_script.py -q --basetemp .pytest_tmp -p no:cacheprovider

Release/local packaging gates:

python scripts/check_workspace_hygiene.py
python scripts/check_release_consistency.py
python scripts/check_docs_hygiene.py
python -m pytest -p no:cacheprovider --basetemp .tmp_pytest_vX_Y_Z
python -m build

Release and packaging checks are local gates only; they do not tag, push, publish releases, upload assets, or run live downloads.

Citation

Use the repository citation metadata when citing TypeTreeFlow.

License

See LICENSE.

Known Limitations

TypeTreeFlow does not promise automatic 100% coverage for a genus. Gap reports, expanded discovery, taxonomy enrichment, and provider planning make missing evidence easier to review; they do not relax strict evidence thresholds.