From a9f29c4a20ef8564715861876e45976776adbbe8 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 09:09:17 -0700 Subject: [PATCH 01/15] docs(msa-search-nim): document task-specific database profiles for faster startup Add NIM_MODEL_PROFILE / NIM_MODEL_NAME guidance so users can download only the databases a task needs instead of the full ~1.4 TB set. Notably, paired/complex MSA search needs only the UniRef30 profile (~500 GB), which skips the ColabFold envdb. Updates SKILL.md, references/api.md, and references/parameters.md. --- nim-skills/msa-search-nim/SKILL.md | 75 ++++++++++++++++++- nim-skills/msa-search-nim/references/api.md | 6 +- .../msa-search-nim/references/parameters.md | 14 ++++ 3 files changed, 92 insertions(+), 3 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index 7960e8b..be71df5 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -41,8 +41,11 @@ for template search unless the hosted docs/service changes. ## Local Docker -Local setup requires a GPU and about 1.4 TB / 1660 GB of NVMe storage for -databases. For setup answers, include env preflight, `docker login`, +Local setup requires a GPU. The full database set is about 1.4 TB / 1660 GB of +NVMe storage, but you rarely need all of it — use a **task-specific profile** +(see "Faster Startup" below) to download only the databases your task requires, +which cuts both storage and startup time. Size the cache volume to the profile you +pick. For setup answers, include env preflight, `docker login`, `docker run`, readiness, and then no-auth local inference. Do not invent a cache default or drop the `NVIDIA_API_KEY` fallback. @@ -78,6 +81,74 @@ Readiness: until curl -sf http://localhost:8000/v1/health/ready; do sleep 10; done ``` +## Faster Startup: Task-Specific Database Profiles + +The full database download is ~1.4 TB and can take well over an hour on first launch. If you +only need some databases, select a **task-specific profile** so the NIM downloads just those. +This is the single biggest lever on local startup time. + +List the profiles your image actually ships (hashes change between releases — never hardcode +them): + +```bash +docker run --rm --entrypoint list-model-profiles nvcr.io/nim/colabfold/msa-search:2 +``` + +Then pass the chosen hash with `NIM_MODEL_PROFILE`: + +```bash +docker run --rm --name msa-search \ + --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_PROFILE= \ + -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 +``` + +Profiles available in this image (confirm hashes with `list-model-profiles`): + +| Profile tags | Databases | Best for | Storage | +|---|---|---|---| +| `databases:pdb70` | PDB70 | Quick testing / smoke check | ~100 MB | +| `databases:uniref30` | UniRef30 | **Paired MSA search for complexes** — UniRef30 is the only DB used for species-based pairing | ~500 GB | +| `databases:uniref30,pdb70,pdb` | UniRef30 + PDB70 + PDB structures | Structural template search | ~700 GB | +| `databases:all` (default) | UniRef30 + ColabFold envdb + PDB70 + PDB100 + PDB structures | Full sensitivity, all databases | ~1.2 TB | + +Verify the loaded profile after readiness: + +```bash +curl -s localhost:8000/v1/metadata | jq +``` + +Notes: + +- The request-level `databases` parameter only selects among databases **already + downloaded**; it does NOT change what is fetched at startup. Startup footprint is set by + `NIM_MODEL_PROFILE` alone. +- **Paired search needs UniRef30 only.** `colabfold_envdb_202108` has no taxonomy and cannot + be used for pairing, so `databases:uniref30` is the correct, smallest profile for + complex/paired workflows — it skips the envdb, the largest part of the full set. +- For maximum monomer sensitivity (UniRef30 + envdb merged) you still need `databases:all`; + there is no envdb-inclusive profile smaller than the full set. + +### Custom Or Individual Databases + +To use a single manually downloaded database (or your own MMSeqs2 DB), download it from NGC +and point the NIM at the mount with `NIM_MODEL_NAME` instead of a profile: + +```bash +ngc registry model download-version nim/colabfold/msa-search:uniref30_2302-m18v1 +# then mount the directory and set -e NIM_MODEL_NAME=/databases +``` + +`NIM_MODEL_NAME` **replaces** the profile databases entirely — the NIM uses only what is +under that directory (discovered by scanning for `**/*.idx`). Mount multiple databases under +one parent to combine them. NGC-downloaded databases are pre-indexed for GPU Server; custom +databases must be indexed with `mmseqs createindex` first. Individually downloadable NGC model +versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`, +`pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`. + ## Standard MSA Request Use exact case-sensitive database names and response keys. diff --git a/nim-skills/msa-search-nim/references/api.md b/nim-skills/msa-search-nim/references/api.md index 52febac..07bb755 100644 --- a/nim-skills/msa-search-nim/references/api.md +++ b/nim-skills/msa-search-nim/references/api.md @@ -29,6 +29,8 @@ | `e_value` | float | No | `0.0001` | 0.0–1.0 | Hit filtering threshold | | `iterations` | integer | No | `1` | 1–6 | Ignored for cascaded search | | `max_msa_sequences` | integer | No | `500` | 1–500 | Must equal `NIM_GLOBAL_MAX_MSA_DEPTH` when GPU Server enabled | +| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. | +| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. | | `output_alignment_formats` | list[string] | No | `["a3m"]` | — | `"a3m"`, `"fasta"` | ### Response Body @@ -153,7 +155,7 @@ docker run --rm --name msa-search \ | Flag | Value | Notes | |---|---|---| | `--gpus` | `all` | Multi-GPU supported; 48 GB GPUs need ≥2 units | -| Cache mount | `/opt/nim/.cache` | ~1.4 TB for full database download | +| Cache mount | `/opt/nim/.cache` | ~1.4 TB for `databases:all`; far less with a task-specific `NIM_MODEL_PROFILE` (~500 GB UniRef30, ~100 MB PDB70) | | Image | `nvcr.io/nim/colabfold/msa-search:2` | v2.3.0 as of 2025; `:2` is the major version tag | ### Key Environment Variables @@ -161,6 +163,8 @@ docker run --rm --name msa-search \ | Variable | Default | Notes | |---|---|---| | `NIM_GLOBAL_MAX_MSA_DEPTH` | `500` | Must match `max_msa_sequences` in requests (GPU Server mode) | +| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. | +| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. | --- diff --git a/nim-skills/msa-search-nim/references/parameters.md b/nim-skills/msa-search-nim/references/parameters.md index 6f1685f..dc02a04 100644 --- a/nim-skills/msa-search-nim/references/parameters.md +++ b/nim-skills/msa-search-nim/references/parameters.md @@ -29,3 +29,17 @@ container configuration. - Set `max_structures` to the requested count. - Include `max_msa_sequences=500` unless the local `NIM_GLOBAL_MAX_MSA_DEPTH` was changed. + +## Startup Database Selection (local Docker) + +`databases` in the request body selects among **already-downloaded** databases only. To +control what is downloaded at container startup (and thus storage and launch time), set the +`NIM_MODEL_PROFILE` environment variable to a profile hash from `list-model-profiles`: + +- `databases:pdb70` — ~100 MB, quick test. +- `databases:uniref30` — ~500 GB, paired/complex MSA search (UniRef30 is the only taxonomy + DB used for pairing). +- `databases:uniref30,pdb70,pdb` — ~700 GB, template search. +- `databases:all` — ~1.2 TB, full sensitivity (adds ColabFold envdb, PDB100). + +Hashes change between NIM releases — always read them from `list-model-profiles`. From ef22698ce8c017d6dd0f77689cda8cf0659e12f1 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 10:39:21 -0700 Subject: [PATCH 02/15] docs(msa-search-nim): add parallel-download recipe for large single-DB profiles The databases:uniref30 profile is ~490 GB dominated by two huge files; the NIM's built-in downloader is per-file-parallel and stalls on them at the NGC per-connection throttle (>80 min, measured). Document a range-parallel aria2c download (~13.5 min measured) fed to the NIM via NIM_MODEL_NAME, plus the persist-the-cache follow-up. --- nim-skills/msa-search-nim/SKILL.md | 62 ++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index be71df5..0d75ae5 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -149,6 +149,68 @@ databases must be indexed with `mmseqs createindex` first. Individually download versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`, `pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`. +## Even Faster: Parallel Download Of A Large Single-DB Profile + +Task-specific profiles cut *what* you download; a parallel downloader cuts *how long* that +download takes. This matters most for the `databases:uniref30` profile, which is ~490 GB +dominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB). + +The NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls +each file over roughly one connection. The NGC CDN throttles a single connection to ~20–25 +MB/s, so while the downloader is fetching one of the two giant files, most of its parallel +slots sit idle and throughput collapses to that single-flow rate. Measured on an H100 node, +the built-in path did not reach `/health/ready` in over 80 minutes. + +A range-parallel downloader splits **each file** into many byte-range segments (the NGC CDN +advertises `accept-ranges: bytes`), so a single 241 GB file is pulled over 16 connections at +once — ~15× the single-flow rate. Same node, `aria2c` fetched the full ~490 GB in **~13.5 +minutes**. + +Workflow (download once with aria2, then start the NIM against the files via `NIM_MODEL_NAME`): + +```bash +# 1) Get presigned file URLs for the individual database model version from NGC. +# (Requires NGC_API_KEY. The response arrays `urls` and `filepath` are positionally paired.) +curl -s -H "Authorization: Bearer $NGC_API_KEY" \ + 'https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/uniref30_2302-m18v1/files' \ + -o files.json + +# 2) Build an aria2 input file (URL + target filename per entry) and download in parallel. +python3 - <<'PY' +import json +d = json.load(open("files.json")) +lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), " dir=/data/fast-db", f" out={path}"] +open("aria.in", "w").write("\n".join(lines) + "\n") +PY +aria2c -i aria.in \ + --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \ + --min-split-size=1M --continue=true --file-allocation=none + +# 3) Start the NIM against the downloaded directory. NIM_MODEL_NAME makes the NIM discover +# databases by scanning for **/*.idx, bypassing the profile/blob cache entirely. +docker run -d --name msa-search --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_NAME=/databases \ + -v /data/fast-db:/databases \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 +``` + +Notes: + +- The presigned URLs expire (typically within a day) — build the aria2 input and start the + download promptly after fetching `files.json`. +- Keep the downloaded directory's internal layout intact (e.g. `uniref30_2302/…`); the + `filepath` values already encode it. The NIM needs the `.idx` file plus its companion files + and the small `.UNIREF30_READY` / `*.tar.gz.unpacked` markers. +- The bottleneck is the CDN's per-connection cap, not local disk or CPU — a fast NVMe volume + writes far faster than the network delivers. Raising `--split` / `--max-connection-per-server` + helps only up to the node's aggregate egress ceiling. +- Best of all: download the profile once, then **persist the cache volume** (or this + `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. + ## Standard MSA Request Use exact case-sensitive database names and response keys. From 43737c9f89d942405526443259abb332eca75a94 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 10:59:15 -0700 Subject: [PATCH 03/15] test(msa-search-nim): add from-scratch launch-speed regression test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Self-contained script that installs the skill, downloads the databases:uniref30 profile in parallel with aria2c, warm-starts the NIM via NIM_MODEL_NAME, and runs a real paired MSA search (C1GY11 + C1HCX1) as a functional check. Validated on an H100 node: 455 GB downloaded in 804 s, NIM ready 12 s after start, paired search HTTP 200 with 500 rows/chain (colabfold_paired) — total ~14 min cold. --- .../scripts/test_from_scratch.sh | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 nim-skills/msa-search-nim/scripts/test_from_scratch.sh diff --git a/nim-skills/msa-search-nim/scripts/test_from_scratch.sh b/nim-skills/msa-search-nim/scripts/test_from_scratch.sh new file mode 100644 index 0000000..79e8998 --- /dev/null +++ b/nim-skills/msa-search-nim/scripts/test_from_scratch.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# ============================================================================= +# End-to-end test of the UPDATED msa-search-nim skill (PR #20) from scratch. +# +# Validates the two things the PR adds: +# 1. Task-specific database profile (download only UniRef30, not the full 1.4 TB) +# 2. Parallel download + NIM_MODEL_NAME warm start (~13.5 min vs >80 min) +# ...then proves correctness with a real paired MSA search (C1GY11 + C1HCX1). +# +# PREREQUISITES on a fresh node: +# - NVIDIA GPU (A100/H100/L40S...), Docker + NVIDIA runtime, ~600 GB free NVMe +# - NGC_API_KEY with registry (nvcr.io) pull scope <-- export before running +# - Tools: docker, curl, python3, aria2c (script installs aria2c if missing) +# +# USAGE: +# export NGC_API_KEY=nvapi-... +# bash test_msa_skill_from_scratch.sh +# ============================================================================= +set -uo pipefail + +: "${NGC_API_KEY:?export NGC_API_KEY (with nvcr.io pull scope) first}" +DATA=/data # fast NVMe mount with ~600 GB free +IMG=nvcr.io/nim/colabfold/msa-search:2 +DBDIR=$DATA/fast-db # where aria2 writes the databases +PORT=8000 +MODEL_VERSION=uniref30_2302-m18v1 # the individual UniRef30 DB model version on NGC + +log(){ echo -e "\n=== $* ==="; } + +# ----------------------------------------------------------------------------- +log "STEP 0 Install the UPDATED skill from the PR branch" +# The catalog ships the OLD skill; test the PR version by checking it out. +# (Skip if you already have the branch installed in your agent skills dir.) +SKILL_SRC=/tmp/bionemo-agent-toolkit +rm -rf "$SKILL_SRC" +git clone --depth 1 --branch msa-search-nim-db-profiles \ + https://github.com/nil16/bionemo-agent-toolkit.git "$SKILL_SRC" +echo "Updated SKILL.md sections:" +grep -E '^## ' "$SKILL_SRC/nim-skills/msa-search-nim/SKILL.md" +# You should see "Faster Startup: Task-Specific Database Profiles" +# and "Even Faster: Parallel Download Of A Large Single-DB Profile" + +# ----------------------------------------------------------------------------- +log "STEP 1 docker login + confirm the task-specific profile exists" +echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin +docker pull "$IMG" +echo "Available profiles (hashes change between releases — read them here, never hardcode):" +docker run --rm --entrypoint list-model-profiles "$IMG" | grep -iE 'databases:' +# Expect a 'databases:uniref30' profile — that is the paired-search profile (~490 GB). + +# ----------------------------------------------------------------------------- +log "STEP 2 Parallel download of UniRef30 with aria2c (the fast path)" +command -v aria2c >/dev/null || sudo apt-get install -y aria2 +mkdir -p "$DBDIR" +# 2a) presigned file URLs for the individual DB version +curl -s -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${MODEL_VERSION}/files" \ + -o /tmp/files.json +# 2b) build aria2 input (urls[] and filepath[] are positionally paired) +python3 - <<'PY' +import json +d = json.load(open("/tmp/files.json")) +lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), " dir=/data/fast-db", f" out={path}"] +open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") +print("files to fetch:", len(d["urls"])) +PY +# 2c) download in parallel — 4 files at once, 16 range-connections each +T0=$(date +%s) +aria2c -i /tmp/aria.in \ + --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \ + --min-split-size=1M --continue=true --file-allocation=none \ + --summary-interval=30 --console-log-level=warn +echo "aria2 exit=$? download_seconds=$(( $(date +%s) - T0 ))" +echo "downloaded size:"; du -sh "$DBDIR" + +# ----------------------------------------------------------------------------- +log "STEP 3 Start the NIM against the downloaded DB via NIM_MODEL_NAME" +docker rm -f msa-search 2>/dev/null || true +S0=$(date +%s) +docker run -d --name msa-search --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_NAME=/databases \ + -v "$DBDIR:/databases" \ + -p ${PORT}:8000 \ + "$IMG" + +log "STEP 4 Wait for readiness (should be well under a minute — index already local)" +until [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:${PORT}/v1/health/ready)" = "200" ]; do + sleep 3 + # bail if container died + docker ps --filter name=msa-search --format '{{.Names}}' | grep -q msa-search || { docker logs msa-search | tail -30; exit 1; } +done +echo "READY after $(( $(date +%s) - S0 )) s from container start" +echo "Loaded database:"; curl -s http://localhost:${PORT}/biology/colabfold/msa-search/config/msa-database-configs + +# ----------------------------------------------------------------------------- +log "STEP 5 Functional check — real paired MSA search (C1GY11 + C1HCX1)" +A=$(curl -s https://rest.uniprot.org/uniprotkb/C1GY11.fasta | grep -v '^>' | tr -d '\n') +B=$(curl -s https://rest.uniprot.org/uniprotkb/C1HCX1.fasta | grep -v '^>' | tr -d '\n') +cat > /tmp/paired.json <')} query_start={a.splitlines()[1][:24]}") +print("search_type:", d["metrics"].get("search_type")) +print("PASS: paired search returned equal-depth species-paired alignments" + if all(abc[c]['Uniref30_2302']['a3m']['alignment'].count('>')>0 for c in 'AB') + else "FAIL") +PY + +# ----------------------------------------------------------------------------- +log "DONE. Expected results:" +cat <<'EOF' + - list-model-profiles shows databases:uniref30 (~490 GB) profile + - aria2 download of ~490 GB completes in ~13-15 min (vs >80 min built-in) + - NIM reaches /health/ready within ~1 min of container start (index is local) + - paired search: HTTP 200, chain A & B each ~500 rows, search_type colabfold_paired + - Next time: persist $DBDIR (or snapshot the volume) -> ~20 s warm start, no re-download +EOF From c58571f676e9a22dd2b0a980154bae7e924743a7 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 10:59:28 -0700 Subject: [PATCH 04/15] docs(msa-search-nim): reference the from-scratch test script from SKILL.md --- nim-skills/msa-search-nim/SKILL.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index 0d75ae5..bce5004 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -211,6 +211,10 @@ Notes: - Best of all: download the profile once, then **persist the cache volume** (or this `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. +> A self-contained end-to-end test of this fast-launch path (install → parallel +> download → warm start → paired MSA search functional check) is provided in +> `scripts/test_from_scratch.sh`. + ## Standard MSA Request Use exact case-sensitive database names and response keys. From 99d4a1695ae021b6326f0d249a8b7b67e994d80d Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:26:47 -0700 Subject: [PATCH 05/15] feat(msa-search-nim): add launch_fast.sh one-command fast-launch wrapper MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Single command that makes the fast path automatic: parallel aria2c download of the requested DB (skipped if already present) then NIM launch via NIM_MODEL_NAME. Idempotent — warm start (~7 s, validated) when the DB dir is populated, ~14 min cold otherwise. A plain docker run with NIM_MODEL_PROFILE cannot use aria2c (the NIM's built-in downloader is not pluggable), so this wrapper is the entry point. --- .../msa-search-nim/scripts/launch_fast.sh | 125 ++++++++++++++++++ 1 file changed, 125 insertions(+) create mode 100644 nim-skills/msa-search-nim/scripts/launch_fast.sh diff --git a/nim-skills/msa-search-nim/scripts/launch_fast.sh b/nim-skills/msa-search-nim/scripts/launch_fast.sh new file mode 100644 index 0000000..90d40dd --- /dev/null +++ b/nim-skills/msa-search-nim/scripts/launch_fast.sh @@ -0,0 +1,125 @@ +#!/usr/bin/env bash +# ============================================================================= +# launch_fast.sh — one-command fast launch of the MSA-Search NIM. +# +# Wraps the two-step fast path into a single command: +# 1. Download the requested database in parallel with aria2c (if not already present) +# 2. Start the NIM against those files via NIM_MODEL_NAME (skips the slow built-in download) +# +# Idempotent: if the database dir already holds the DB, it skips straight to launch +# (~20 s warm start, no re-download). +# +# WHY THIS EXISTS: a plain `docker run ... -e NIM_MODEL_PROFILE=...` uses the NIM's +# built-in per-file downloader, which stalls on large DBs at the NGC per-connection +# throttle (>80 min for UniRef30). aria2c range-parallel download + NIM_MODEL_NAME +# brings that to ~14 min cold / ~20 s warm. The NIM cannot call aria2c itself, so this +# wrapper is the entry point that makes the fast path automatic. +# +# PREREQUISITES: NVIDIA GPU + Docker + NVIDIA runtime; NGC_API_KEY with nvcr.io pull +# scope; ~600 GB free NVMe for UniRef30; docker, curl, python3 (installs +# aria2c if missing). +# +# USAGE: +# export NGC_API_KEY=nvapi-... +# ./launch_fast.sh # defaults: UniRef30 -> /data/fast-db, port 8000 +# ./launch_fast.sh -m colabfold_envdb_202108-m18v1 -d /data/envdb -p 8001 +# +# OPTIONS: +# -m NGC DB model version (default: uniref30_2302-m18v1) +# -d local DB directory (default: /data/fast-db) +# -p host port (default: 8000) +# -n container name (default: msa-search) +# -s aria2 split / conns (default: 16) +# -c aria2 concurrent files (default: 4) +# ============================================================================= +set -uo pipefail + +MODEL_VERSION=uniref30_2302-m18v1 +DBDIR=/data/fast-db +PORT=8000 +NAME=msa-search +SPLIT=16 +CONC=4 +IMG=nvcr.io/nim/colabfold/msa-search:2 + +while getopts "m:d:p:n:s:c:h" opt; do + case $opt in + m) MODEL_VERSION=$OPTARG ;; + d) DBDIR=$OPTARG ;; + p) PORT=$OPTARG ;; + n) NAME=$OPTARG ;; + s) SPLIT=$OPTARG ;; + c) CONC=$OPTARG ;; + h) grep -E '^#( |$)' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) echo "bad option; -h for help"; exit 2 ;; + esac +done + +: "${NGC_API_KEY:?export NGC_API_KEY (with nvcr.io pull scope) first}" +log(){ echo -e "\n=== $* ==="; } + +# The DB short-name is the model version without the -m##v# suffix (e.g. uniref30_2302). +DBNAME=$(echo "$MODEL_VERSION" | sed -E 's/-m[0-9]+v[0-9]+$//') + +# ----------------------------------------------------------------------------- +# STEP 1: download (skipped if the DB's .idx already exists locally) +# ----------------------------------------------------------------------------- +if ls "$DBDIR/$DBNAME/"*_db.idx >/dev/null 2>&1; then + log "DB already present in $DBDIR/$DBNAME — skipping download (warm start)" +else + log "Downloading $MODEL_VERSION into $DBDIR with aria2c (${CONC}x${SPLIT})" + command -v aria2c >/dev/null || sudo apt-get install -y aria2 + mkdir -p "$DBDIR" + + curl -fsS -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${MODEL_VERSION}/files" \ + -o /tmp/files.json || { echo "NGC files API call failed"; exit 1; } + + DBDIR="$DBDIR" python3 - <<'PY' +import json, os +d = json.load(open("/tmp/files.json")) +dbdir = os.environ["DBDIR"] +lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), f" dir={dbdir}", f" out={path}"] +open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") +print(f"files to fetch: {len(d['urls'])}") +PY + + T0=$(date +%s) + aria2c -i /tmp/aria.in \ + --max-concurrent-downloads="$CONC" \ + --max-connection-per-server="$SPLIT" \ + --split="$SPLIT" \ + --min-split-size=1M --continue=true --file-allocation=none \ + --summary-interval=30 --console-log-level=warn + RC=$? + [ $RC -eq 0 ] || { echo "aria2c failed (exit $RC)"; exit $RC; } + echo "download_seconds=$(( $(date +%s) - T0 )) size=$(du -sh "$DBDIR" | cut -f1)" + rm -f /tmp/files.json /tmp/aria.in +fi + +# ----------------------------------------------------------------------------- +# STEP 2: launch the NIM against the downloaded files +# ----------------------------------------------------------------------------- +log "Launching NIM ($NAME) on port $PORT against $DBDIR" +docker rm -f "$NAME" >/dev/null 2>&1 || true +S0=$(date +%s) +docker run -d --name "$NAME" --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_NAME=/databases \ + -v "$DBDIR:/databases" \ + -p "${PORT}:8000" \ + "$IMG" >/dev/null + +log "Waiting for readiness" +until [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:${PORT}/v1/health/ready)" = "200" ]; do + sleep 3 + docker ps --filter "name=$NAME" --format '{{.Names}}' | grep -q "$NAME" \ + || { echo "container exited:"; docker logs "$NAME" | tail -30; exit 1; } +done +echo "READY after $(( $(date +%s) - S0 )) s" +echo "Loaded DB: $(curl -s http://localhost:${PORT}/biology/colabfold/msa-search/config/msa-database-configs)" +echo +echo "Endpoint: http://localhost:${PORT}/biology/colabfold/msa-search/{predict,paired/predict}" +echo "Tip: persist $DBDIR (snapshot the volume) so future launches skip the download entirely." From e4add1d4ed3307ab6ed3e466acb2ffadf7a2784f Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:26:59 -0700 Subject: [PATCH 06/15] docs(msa-search-nim): point to launch_fast.sh wrapper as the fast-path entry point --- nim-skills/msa-search-nim/SKILL.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index bce5004..53c5cdb 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -211,9 +211,12 @@ Notes: - Best of all: download the profile once, then **persist the cache volume** (or this `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. -> A self-contained end-to-end test of this fast-launch path (install → parallel -> download → warm start → paired MSA search functional check) is provided in -> `scripts/test_from_scratch.sh`. +> **One-command wrapper:** `scripts/launch_fast.sh` does the whole fast path in a +> single command — parallel aria2c download (skipped if the DB is already present) +> then NIM launch via `NIM_MODEL_NAME`. A plain `docker run` with `NIM_MODEL_PROFILE` +> uses the NIM's slow built-in downloader; only this wrapper (or the manual steps +> above) uses aria2c. `scripts/test_from_scratch.sh` additionally runs a paired-MSA +> functional check. ## Standard MSA Request From 6aaa909518caf3ea7b5a58485f515080f35b13b0 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:28:32 -0700 Subject: [PATCH 07/15] revert(msa-search-nim): remove launch_fast.sh wrapper test_from_scratch.sh is self-contained and does not depend on it; dropping the redundant wrapper to keep the PR focused on the documented manual fast path. --- .../msa-search-nim/scripts/launch_fast.sh | 125 ------------------ 1 file changed, 125 deletions(-) delete mode 100644 nim-skills/msa-search-nim/scripts/launch_fast.sh diff --git a/nim-skills/msa-search-nim/scripts/launch_fast.sh b/nim-skills/msa-search-nim/scripts/launch_fast.sh deleted file mode 100644 index 90d40dd..0000000 --- a/nim-skills/msa-search-nim/scripts/launch_fast.sh +++ /dev/null @@ -1,125 +0,0 @@ -#!/usr/bin/env bash -# ============================================================================= -# launch_fast.sh — one-command fast launch of the MSA-Search NIM. -# -# Wraps the two-step fast path into a single command: -# 1. Download the requested database in parallel with aria2c (if not already present) -# 2. Start the NIM against those files via NIM_MODEL_NAME (skips the slow built-in download) -# -# Idempotent: if the database dir already holds the DB, it skips straight to launch -# (~20 s warm start, no re-download). -# -# WHY THIS EXISTS: a plain `docker run ... -e NIM_MODEL_PROFILE=...` uses the NIM's -# built-in per-file downloader, which stalls on large DBs at the NGC per-connection -# throttle (>80 min for UniRef30). aria2c range-parallel download + NIM_MODEL_NAME -# brings that to ~14 min cold / ~20 s warm. The NIM cannot call aria2c itself, so this -# wrapper is the entry point that makes the fast path automatic. -# -# PREREQUISITES: NVIDIA GPU + Docker + NVIDIA runtime; NGC_API_KEY with nvcr.io pull -# scope; ~600 GB free NVMe for UniRef30; docker, curl, python3 (installs -# aria2c if missing). -# -# USAGE: -# export NGC_API_KEY=nvapi-... -# ./launch_fast.sh # defaults: UniRef30 -> /data/fast-db, port 8000 -# ./launch_fast.sh -m colabfold_envdb_202108-m18v1 -d /data/envdb -p 8001 -# -# OPTIONS: -# -m NGC DB model version (default: uniref30_2302-m18v1) -# -d local DB directory (default: /data/fast-db) -# -p host port (default: 8000) -# -n container name (default: msa-search) -# -s aria2 split / conns (default: 16) -# -c aria2 concurrent files (default: 4) -# ============================================================================= -set -uo pipefail - -MODEL_VERSION=uniref30_2302-m18v1 -DBDIR=/data/fast-db -PORT=8000 -NAME=msa-search -SPLIT=16 -CONC=4 -IMG=nvcr.io/nim/colabfold/msa-search:2 - -while getopts "m:d:p:n:s:c:h" opt; do - case $opt in - m) MODEL_VERSION=$OPTARG ;; - d) DBDIR=$OPTARG ;; - p) PORT=$OPTARG ;; - n) NAME=$OPTARG ;; - s) SPLIT=$OPTARG ;; - c) CONC=$OPTARG ;; - h) grep -E '^#( |$)' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; - *) echo "bad option; -h for help"; exit 2 ;; - esac -done - -: "${NGC_API_KEY:?export NGC_API_KEY (with nvcr.io pull scope) first}" -log(){ echo -e "\n=== $* ==="; } - -# The DB short-name is the model version without the -m##v# suffix (e.g. uniref30_2302). -DBNAME=$(echo "$MODEL_VERSION" | sed -E 's/-m[0-9]+v[0-9]+$//') - -# ----------------------------------------------------------------------------- -# STEP 1: download (skipped if the DB's .idx already exists locally) -# ----------------------------------------------------------------------------- -if ls "$DBDIR/$DBNAME/"*_db.idx >/dev/null 2>&1; then - log "DB already present in $DBDIR/$DBNAME — skipping download (warm start)" -else - log "Downloading $MODEL_VERSION into $DBDIR with aria2c (${CONC}x${SPLIT})" - command -v aria2c >/dev/null || sudo apt-get install -y aria2 - mkdir -p "$DBDIR" - - curl -fsS -H "Authorization: Bearer $NGC_API_KEY" \ - "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${MODEL_VERSION}/files" \ - -o /tmp/files.json || { echo "NGC files API call failed"; exit 1; } - - DBDIR="$DBDIR" python3 - <<'PY' -import json, os -d = json.load(open("/tmp/files.json")) -dbdir = os.environ["DBDIR"] -lines = [] -for url, path in zip(d["urls"], d["filepath"]): - lines += [url.strip(), f" dir={dbdir}", f" out={path}"] -open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") -print(f"files to fetch: {len(d['urls'])}") -PY - - T0=$(date +%s) - aria2c -i /tmp/aria.in \ - --max-concurrent-downloads="$CONC" \ - --max-connection-per-server="$SPLIT" \ - --split="$SPLIT" \ - --min-split-size=1M --continue=true --file-allocation=none \ - --summary-interval=30 --console-log-level=warn - RC=$? - [ $RC -eq 0 ] || { echo "aria2c failed (exit $RC)"; exit $RC; } - echo "download_seconds=$(( $(date +%s) - T0 )) size=$(du -sh "$DBDIR" | cut -f1)" - rm -f /tmp/files.json /tmp/aria.in -fi - -# ----------------------------------------------------------------------------- -# STEP 2: launch the NIM against the downloaded files -# ----------------------------------------------------------------------------- -log "Launching NIM ($NAME) on port $PORT against $DBDIR" -docker rm -f "$NAME" >/dev/null 2>&1 || true -S0=$(date +%s) -docker run -d --name "$NAME" --runtime=nvidia --gpus all \ - -e NGC_API_KEY \ - -e NIM_MODEL_NAME=/databases \ - -v "$DBDIR:/databases" \ - -p "${PORT}:8000" \ - "$IMG" >/dev/null - -log "Waiting for readiness" -until [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:${PORT}/v1/health/ready)" = "200" ]; do - sleep 3 - docker ps --filter "name=$NAME" --format '{{.Names}}' | grep -q "$NAME" \ - || { echo "container exited:"; docker logs "$NAME" | tail -30; exit 1; } -done -echo "READY after $(( $(date +%s) - S0 )) s" -echo "Loaded DB: $(curl -s http://localhost:${PORT}/biology/colabfold/msa-search/config/msa-database-configs)" -echo -echo "Endpoint: http://localhost:${PORT}/biology/colabfold/msa-search/{predict,paired/predict}" -echo "Tip: persist $DBDIR (snapshot the volume) so future launches skip the download entirely." From ed55ba9d06a8a7b83338096e4d45060c82f3815a Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:28:33 -0700 Subject: [PATCH 08/15] docs(msa-search-nim): revert SKILL.md pointer to test_from_scratch.sh only --- nim-skills/msa-search-nim/SKILL.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index 53c5cdb..bce5004 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -211,12 +211,9 @@ Notes: - Best of all: download the profile once, then **persist the cache volume** (or this `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. -> **One-command wrapper:** `scripts/launch_fast.sh` does the whole fast path in a -> single command — parallel aria2c download (skipped if the DB is already present) -> then NIM launch via `NIM_MODEL_NAME`. A plain `docker run` with `NIM_MODEL_PROFILE` -> uses the NIM's slow built-in downloader; only this wrapper (or the manual steps -> above) uses aria2c. `scripts/test_from_scratch.sh` additionally runs a paired-MSA -> functional check. +> A self-contained end-to-end test of this fast-launch path (install → parallel +> download → warm start → paired MSA search functional check) is provided in +> `scripts/test_from_scratch.sh`. ## Standard MSA Request From d957169bd5bd49031c6bbb310f6a17c499ea9015 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:29:08 -0700 Subject: [PATCH 09/15] revert(msa-search-nim): remove test_from_scratch.sh Not required by the skill; the SKILL.md/reference docs are the deliverable. Keeping the PR to documentation only. --- .../scripts/test_from_scratch.sh | 131 ------------------ 1 file changed, 131 deletions(-) delete mode 100644 nim-skills/msa-search-nim/scripts/test_from_scratch.sh diff --git a/nim-skills/msa-search-nim/scripts/test_from_scratch.sh b/nim-skills/msa-search-nim/scripts/test_from_scratch.sh deleted file mode 100644 index 79e8998..0000000 --- a/nim-skills/msa-search-nim/scripts/test_from_scratch.sh +++ /dev/null @@ -1,131 +0,0 @@ -#!/usr/bin/env bash -# ============================================================================= -# End-to-end test of the UPDATED msa-search-nim skill (PR #20) from scratch. -# -# Validates the two things the PR adds: -# 1. Task-specific database profile (download only UniRef30, not the full 1.4 TB) -# 2. Parallel download + NIM_MODEL_NAME warm start (~13.5 min vs >80 min) -# ...then proves correctness with a real paired MSA search (C1GY11 + C1HCX1). -# -# PREREQUISITES on a fresh node: -# - NVIDIA GPU (A100/H100/L40S...), Docker + NVIDIA runtime, ~600 GB free NVMe -# - NGC_API_KEY with registry (nvcr.io) pull scope <-- export before running -# - Tools: docker, curl, python3, aria2c (script installs aria2c if missing) -# -# USAGE: -# export NGC_API_KEY=nvapi-... -# bash test_msa_skill_from_scratch.sh -# ============================================================================= -set -uo pipefail - -: "${NGC_API_KEY:?export NGC_API_KEY (with nvcr.io pull scope) first}" -DATA=/data # fast NVMe mount with ~600 GB free -IMG=nvcr.io/nim/colabfold/msa-search:2 -DBDIR=$DATA/fast-db # where aria2 writes the databases -PORT=8000 -MODEL_VERSION=uniref30_2302-m18v1 # the individual UniRef30 DB model version on NGC - -log(){ echo -e "\n=== $* ==="; } - -# ----------------------------------------------------------------------------- -log "STEP 0 Install the UPDATED skill from the PR branch" -# The catalog ships the OLD skill; test the PR version by checking it out. -# (Skip if you already have the branch installed in your agent skills dir.) -SKILL_SRC=/tmp/bionemo-agent-toolkit -rm -rf "$SKILL_SRC" -git clone --depth 1 --branch msa-search-nim-db-profiles \ - https://github.com/nil16/bionemo-agent-toolkit.git "$SKILL_SRC" -echo "Updated SKILL.md sections:" -grep -E '^## ' "$SKILL_SRC/nim-skills/msa-search-nim/SKILL.md" -# You should see "Faster Startup: Task-Specific Database Profiles" -# and "Even Faster: Parallel Download Of A Large Single-DB Profile" - -# ----------------------------------------------------------------------------- -log "STEP 1 docker login + confirm the task-specific profile exists" -echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin -docker pull "$IMG" -echo "Available profiles (hashes change between releases — read them here, never hardcode):" -docker run --rm --entrypoint list-model-profiles "$IMG" | grep -iE 'databases:' -# Expect a 'databases:uniref30' profile — that is the paired-search profile (~490 GB). - -# ----------------------------------------------------------------------------- -log "STEP 2 Parallel download of UniRef30 with aria2c (the fast path)" -command -v aria2c >/dev/null || sudo apt-get install -y aria2 -mkdir -p "$DBDIR" -# 2a) presigned file URLs for the individual DB version -curl -s -H "Authorization: Bearer $NGC_API_KEY" \ - "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${MODEL_VERSION}/files" \ - -o /tmp/files.json -# 2b) build aria2 input (urls[] and filepath[] are positionally paired) -python3 - <<'PY' -import json -d = json.load(open("/tmp/files.json")) -lines = [] -for url, path in zip(d["urls"], d["filepath"]): - lines += [url.strip(), " dir=/data/fast-db", f" out={path}"] -open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") -print("files to fetch:", len(d["urls"])) -PY -# 2c) download in parallel — 4 files at once, 16 range-connections each -T0=$(date +%s) -aria2c -i /tmp/aria.in \ - --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \ - --min-split-size=1M --continue=true --file-allocation=none \ - --summary-interval=30 --console-log-level=warn -echo "aria2 exit=$? download_seconds=$(( $(date +%s) - T0 ))" -echo "downloaded size:"; du -sh "$DBDIR" - -# ----------------------------------------------------------------------------- -log "STEP 3 Start the NIM against the downloaded DB via NIM_MODEL_NAME" -docker rm -f msa-search 2>/dev/null || true -S0=$(date +%s) -docker run -d --name msa-search --runtime=nvidia --gpus all \ - -e NGC_API_KEY \ - -e NIM_MODEL_NAME=/databases \ - -v "$DBDIR:/databases" \ - -p ${PORT}:8000 \ - "$IMG" - -log "STEP 4 Wait for readiness (should be well under a minute — index already local)" -until [ "$(curl -s -o /dev/null -w '%{http_code}' http://localhost:${PORT}/v1/health/ready)" = "200" ]; do - sleep 3 - # bail if container died - docker ps --filter name=msa-search --format '{{.Names}}' | grep -q msa-search || { docker logs msa-search | tail -30; exit 1; } -done -echo "READY after $(( $(date +%s) - S0 )) s from container start" -echo "Loaded database:"; curl -s http://localhost:${PORT}/biology/colabfold/msa-search/config/msa-database-configs - -# ----------------------------------------------------------------------------- -log "STEP 5 Functional check — real paired MSA search (C1GY11 + C1HCX1)" -A=$(curl -s https://rest.uniprot.org/uniprotkb/C1GY11.fasta | grep -v '^>' | tr -d '\n') -B=$(curl -s https://rest.uniprot.org/uniprotkb/C1HCX1.fasta | grep -v '^>' | tr -d '\n') -cat > /tmp/paired.json <')} query_start={a.splitlines()[1][:24]}") -print("search_type:", d["metrics"].get("search_type")) -print("PASS: paired search returned equal-depth species-paired alignments" - if all(abc[c]['Uniref30_2302']['a3m']['alignment'].count('>')>0 for c in 'AB') - else "FAIL") -PY - -# ----------------------------------------------------------------------------- -log "DONE. Expected results:" -cat <<'EOF' - - list-model-profiles shows databases:uniref30 (~490 GB) profile - - aria2 download of ~490 GB completes in ~13-15 min (vs >80 min built-in) - - NIM reaches /health/ready within ~1 min of container start (index is local) - - paired search: HTTP 200, chain A & B each ~500 rows, search_type colabfold_paired - - Next time: persist $DBDIR (or snapshot the volume) -> ~20 s warm start, no re-download -EOF From 98d80227194741b78c56b9a3d16623d6ef045ee9 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Tue, 21 Jul 2026 11:29:09 -0700 Subject: [PATCH 10/15] docs(msa-search-nim): remove reference to test script --- nim-skills/msa-search-nim/SKILL.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index bce5004..0d75ae5 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -211,10 +211,6 @@ Notes: - Best of all: download the profile once, then **persist the cache volume** (or this `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. -> A self-contained end-to-end test of this fast-launch path (install → parallel -> download → warm start → paired MSA search functional check) is provided in -> `scripts/test_from_scratch.sh`. - ## Standard MSA Request Use exact case-sensitive database names and response keys. From 8cc37d711dd3415c5ef96f9b8f44cd750e17e39f Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Wed, 22 Jul 2026 12:09:10 -0700 Subject: [PATCH 11/15] docs(msa-search-nim): make the parallel fast-download path the recommended default Restructures the deployment guidance so the aria2c + NIM_MODEL_NAME fast path is presented as the recommended default (banner at the top of Local Docker + section retitle), with the plain full-download docker run demoted to an explicit fallback for the full databases:all set or a pdb70 smoke test. The NIM's built-in downloader is not pluggable, so aria2c cannot auto-trigger at runtime; this makes it the default in guidance instead. --- nim-skills/msa-search-nim/SKILL.md | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index 0d75ae5..ea81205 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -41,9 +41,24 @@ for template search unless the hosted docs/service changes. ## Local Docker +> **Recommended deployment path.** For any real workflow, do NOT start with the plain +> full-database `docker run` below — it triggers the NIM's built-in downloader over the +> full ~1.4 TB set, which is slow (well over an hour, and on large single-DB profiles it +> can stall past 80 minutes; see the measurements under "Parallel Download"). Instead, +> default to the two-step fast path: +> +> 1. **Pick the smallest task-specific profile** for your task ("Faster Startup" below) — +> e.g. `databases:uniref30` for paired/complex work. +> 2. **Download it in parallel with aria2c and launch via `NIM_MODEL_NAME`** +> ("Recommended For Large Profiles: Parallel Download" below) — ~14 min instead of >80 min, +> measured on an H100 node. +> +> Use the plain `docker run` in this section only when you genuinely need the full +> `databases:all` set and are not time-constrained, or for a quick `databases:pdb70` smoke test. + Local setup requires a GPU. The full database set is about 1.4 TB / 1660 GB of NVMe storage, but you rarely need all of it — use a **task-specific profile** -(see "Faster Startup" below) to download only the databases your task requires, +(see "Faster Startup" below) downloaded in parallel to download only the databases your task requires, which cuts both storage and startup time. Size the cache volume to the profile you pick. For setup answers, include env preflight, `docker login`, `docker run`, readiness, and then no-auth local inference. Do not invent a cache @@ -149,10 +164,11 @@ databases must be indexed with `mmseqs createindex` first. Individually download versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`, `pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`. -## Even Faster: Parallel Download Of A Large Single-DB Profile +## Recommended For Large Profiles: Parallel Download (Fast Deployment) -Task-specific profiles cut *what* you download; a parallel downloader cuts *how long* that -download takes. This matters most for the `databases:uniref30` profile, which is ~490 GB +**This is the recommended default for any large single-DB profile** (e.g. +`databases:uniref30`). Task-specific profiles cut *what* you download; this parallel +downloader cuts *how long* that download takes. It matters most for the `databases:uniref30` profile, which is ~490 GB dominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB). The NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls From 549025e082dbbf420349348cec0fa7d42b0be7a4 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Wed, 22 Jul 2026 12:10:46 -0700 Subject: [PATCH 12/15] docs(msa-search-nim): add fast-deployment prompt example Adds a natural-language 'Fast Local Deployment (recommended)' prompt as the first example so the prompt-level default matches the SKILL.md deployment default: parallel UniRef30 download + NIM_MODEL_NAME launch. --- .../msa-search-nim/references/examples.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/nim-skills/msa-search-nim/references/examples.md b/nim-skills/msa-search-nim/references/examples.md index 5fef666..4f5c176 100644 --- a/nim-skills/msa-search-nim/references/examples.md +++ b/nim-skills/msa-search-nim/references/examples.md @@ -2,6 +2,23 @@ Use these compact patterns when the activated skill needs more examples. +## Fast Local Deployment (recommended) + +When deploying the MSA-Search NIM locally for a paired/complex workflow, prefer the fast +path: download only the UniRef30 database in parallel, then launch against it. A plain +`docker run` uses the NIM's built-in downloader (>80 min for UniRef30); the parallel path +is ~14 min. Example prompt: + +> Deploy the MSA-Search NIM locally using the fast path: download only the UniRef30 +> database (`databases:uniref30` profile) in parallel with aria2c, then start the NIM +> against those files with `NIM_MODEL_NAME`. Look up the current profile hash with +> `list-model-profiles` rather than hardcoding it. If the endpoint is already running with +> the database loaded, skip the redeploy. Then run a paired MSA search for chains A and B +> using `Uniref30_2302` only. + +See the "Recommended For Large Profiles: Parallel Download" section of `SKILL.md` for the +exact aria2c + `NIM_MODEL_NAME` commands. + ## Hosted Standard MSA ```python From fd4a05210d5cf522d32c36ce1098b9846e04b702 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Wed, 22 Jul 2026 12:12:58 -0700 Subject: [PATCH 13/15] docs(msa-search-nim): recommend parallel download for ANY database set, incl. databases:all Generalizes the aria2c recommendation beyond single-DB profiles: retitles the section, broadens the lead and banner, and adds a multi-database recipe (fetch each DB version into sibling dirs under one parent, launch once via NIM_MODEL_NAME). The CDN per-connection throttle applies to every database, so parallel download helps the full set proportionally. --- nim-skills/msa-search-nim/SKILL.md | 41 +++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index ea81205..0de49ad 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -49,12 +49,14 @@ for template search unless the hosted docs/service changes. > > 1. **Pick the smallest task-specific profile** for your task ("Faster Startup" below) — > e.g. `databases:uniref30` for paired/complex work. -> 2. **Download it in parallel with aria2c and launch via `NIM_MODEL_NAME`** -> ("Recommended For Large Profiles: Parallel Download" below) — ~14 min instead of >80 min, -> measured on an H100 node. +> 2. **Download the database(s) in parallel with aria2c and launch via `NIM_MODEL_NAME`** +> ("Parallel Download For Any Database Set" below) — ~14 min for UniRef30 instead of +> >80 min, measured on an H100 node. This applies whether you need one database or +> the full `databases:all` set. > -> Use the plain `docker run` in this section only when you genuinely need the full -> `databases:all` set and are not time-constrained, or for a quick `databases:pdb70` smoke test. +> Use the plain `docker run` in this section only for a quick `databases:pdb70` smoke +> test, or when you specifically want the NIM to manage its own blob cache. The parallel +> path below covers every case, including the full `databases:all` set. Local setup requires a GPU. The full database set is about 1.4 TB / 1660 GB of NVMe storage, but you rarely need all of it — use a **task-specific profile** @@ -164,11 +166,12 @@ databases must be indexed with `mmseqs createindex` first. Individually download versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`, `pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`. -## Recommended For Large Profiles: Parallel Download (Fast Deployment) +## Recommended: Parallel Download For Any Database Set (Fast Deployment) -**This is the recommended default for any large single-DB profile** (e.g. -`databases:uniref30`). Task-specific profiles cut *what* you download; this parallel -downloader cuts *how long* that download takes. It matters most for the `databases:uniref30` profile, which is ~490 GB +**This is the recommended way to download the databases at all — for any profile, +including the full `databases:all` set.** Task-specific profiles cut *what* you +download; this parallel downloader cuts *how long* that download takes. Use it whether +you need one database or all of them. The gain is largest for the `databases:uniref30` profile, which is ~490 GB dominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB). The NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls @@ -214,6 +217,26 @@ docker run -d --name msa-search --runtime=nvidia --gpus all \ nvcr.io/nim/colabfold/msa-search:2 ``` +For **all databases** (equivalent to `databases:all`), repeat step 1 for each individual DB +version and download them into sibling directories under one parent, then point +`NIM_MODEL_NAME` at that parent — the NIM discovers every DB by scanning `**/*.idx`: + +```bash +# fetch each DB's file list into /data/all-db// ... then one aria2c per list, e.g.: +for V in uniref30_2302-m18v1 colabfold_envdb_202108-m18v1 pdb70_220313-m18v1 \ + pdb100_230517-m18v1 pdb_20251028_zip-m18v1; do + curl -s -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/$V/files" \ + -o "files_$V.json" + # build an aria2 input from files_$V.json (dir=/data/all-db) and run aria2c on it +done +# then launch once against the parent: +# docker run -d ... -e NIM_MODEL_NAME=/databases -v /data/all-db:/databases ... +``` + +The per-connection CDN throttle is the same for every database, so parallel download helps +the full set proportionally — the more you download, the more absolute time it saves. + Notes: - The presigned URLs expire (typically within a day) — build the aria2 input and start the From 449425d887deeb403a710213a3d9f92dcd95e7a8 Mon Sep 17 00:00:00 2001 From: "Nilkanth (Neel) Patel" Date: Wed, 22 Jul 2026 12:53:05 -0700 Subject: [PATCH 14/15] docs(msa-search-nim): make aria2c the FIRST recipe + name it in the description Behavioral fix: agents copy the first executable code block and are primed by the frontmatter description. Previous edits only added prose banners, which lost to the plain docker run sitting first. Now the parallel-download + NIM_MODEL_NAME recipe is the first code block in Local Docker, the plain docker run is demoted to an explicit 'Fallback' subsection, and the description names the fast path as the recommended default. --- nim-skills/msa-search-nim/SKILL.md | 96 +++++++++++++++++++----------- 1 file changed, 61 insertions(+), 35 deletions(-) diff --git a/nim-skills/msa-search-nim/SKILL.md b/nim-skills/msa-search-nim/SKILL.md index 0de49ad..4cfd0a1 100644 --- a/nim-skills/msa-search-nim/SKILL.md +++ b/nim-skills/msa-search-nim/SKILL.md @@ -1,7 +1,7 @@ --- name: msa-search-nim description: > - Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. + Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. For local deployment, download the databases in parallel with aria2c and launch via NIM_MODEL_NAME (the recommended default fast path, ~14 min vs >80 min for the built-in downloader); a plain docker run uses the slow built-in downloader. license: Apache-2.0 AND CC-BY-4.0 compatibility: "requests>=2.28" allowed-tools: Bash, Read, Write, AskUserQuestion @@ -41,53 +41,55 @@ for template search unless the hosted docs/service changes. ## Local Docker -> **Recommended deployment path.** For any real workflow, do NOT start with the plain -> full-database `docker run` below — it triggers the NIM's built-in downloader over the -> full ~1.4 TB set, which is slow (well over an hour, and on large single-DB profiles it -> can stall past 80 minutes; see the measurements under "Parallel Download"). Instead, -> default to the two-step fast path: -> -> 1. **Pick the smallest task-specific profile** for your task ("Faster Startup" below) — -> e.g. `databases:uniref30` for paired/complex work. -> 2. **Download the database(s) in parallel with aria2c and launch via `NIM_MODEL_NAME`** -> ("Parallel Download For Any Database Set" below) — ~14 min for UniRef30 instead of -> >80 min, measured on an H100 node. This applies whether you need one database or -> the full `databases:all` set. -> -> Use the plain `docker run` in this section only for a quick `databases:pdb70` smoke -> test, or when you specifically want the NIM to manage its own blob cache. The parallel -> path below covers every case, including the full `databases:all` set. - -Local setup requires a GPU. The full database set is about 1.4 TB / 1660 GB of -NVMe storage, but you rarely need all of it — use a **task-specific profile** -(see "Faster Startup" below) downloaded in parallel to download only the databases your task requires, -which cuts both storage and startup time. Size the cache volume to the profile you -pick. For setup answers, include env preflight, `docker login`, -`docker run`, readiness, and then no-auth local inference. Do not invent a cache -default or drop the `NVIDIA_API_KEY` fallback. +**Default local deployment = parallel download + `NIM_MODEL_NAME`.** The first recipe below +is the one to use for real workflows. It downloads the database(s) with a range-parallel +downloader (aria2c) and starts the NIM against those files — ~14 min for UniRef30 vs >80 min +for the NIM's built-in downloader (measured, H100). Do **not** reach for the plain `docker run` +(the "Fallback" subsection) unless you only want a `databases:pdb70` smoke test or you +deliberately want the NIM to manage its own blob cache. + +Local setup requires a GPU. Size the NVMe volume to the profile you pick (UniRef30 ~490 GB; +full set ~1.4 TB). For setup answers, include env preflight, `docker login`, the parallel +download, `NIM_MODEL_NAME` launch, readiness, and then no-auth local inference. Do not invent a +cache default or drop the `NVIDIA_API_KEY` fallback. ```bash +# --- env preflight (do not drop the NVIDIA_API_KEY fallback) --- set -a [ -f .env ] && . ./.env set +a - if [ -z "${NGC_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then export NGC_API_KEY="$NVIDIA_API_KEY" fi : "${NGC_API_KEY:?Set NGC_API_KEY or NVIDIA_API_KEY}" -: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}" +: "${DB_DIR:=/data/fast-db}" # where the parallel download lands echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin -echo "MSA-Search local databases require about 1.4 TB (1660 GB) of NVMe storage." -mkdir -p "${LOCAL_NIM_CACHE}" -chmod 777 "${LOCAL_NIM_CACHE}" +# --- 1) pick the DB version(s) you need (paired/complex work = uniref30 only) --- +DB_VERSION=uniref30_2302-m18v1 +command -v aria2c >/dev/null || sudo apt-get install -y aria2 +mkdir -p "$DB_DIR" + +# --- 2) parallel download from NGC (see "Parallel Download" section for the all-DB loop) --- +curl -fsS -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${DB_VERSION}/files" \ + -o /tmp/files.json +DB_DIR="$DB_DIR" python3 - <<'PY' +import json, os +d = json.load(open("/tmp/files.json")); dbdir = os.environ["DB_DIR"]; lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), f" dir={dbdir}", f" out={path}"] +open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") +PY +aria2c -i /tmp/aria.in --max-concurrent-downloads=4 --max-connection-per-server=16 \ + --split=16 --min-split-size=1M --continue=true --file-allocation=none -docker run --rm --name msa-search \ - --runtime=nvidia \ - --gpus all \ +# --- 3) launch the NIM against the downloaded files (skips the slow built-in download) --- +docker run -d --name msa-search --runtime=nvidia --gpus all \ -e NGC_API_KEY \ - -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \ + -e NIM_MODEL_NAME=/databases \ + -v "${DB_DIR}:/databases" \ -p 8000:8000 \ nvcr.io/nim/colabfold/msa-search:2 ``` @@ -95,9 +97,33 @@ docker run --rm --name msa-search \ Readiness: ```bash -until curl -sf http://localhost:8000/v1/health/ready; do sleep 10; done +until curl -sf http://localhost:8000/v1/health/ready; do sleep 5; done +``` + +If the DB is already present in `$DB_DIR`, skip steps 1-2 — the launch alone is a ~20 s warm +start. See "Parallel Download For Any Database Set" for the multi-database (`databases:all`) +loop and the full rationale. + +### Fallback: Let The NIM Download Its Own Databases (slower) + +Use this only for a quick `databases:pdb70` smoke test, or when you specifically want the NIM +to manage its own blob cache. It uses the built-in downloader, which is slow on large profiles +(UniRef30 stalled past 80 min in testing). Pin the smallest profile with `NIM_MODEL_PROFILE` +(see "Faster Startup") so it does not fetch the full 1.4 TB. + +```bash +: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}" +mkdir -p "${LOCAL_NIM_CACHE}"; chmod 777 "${LOCAL_NIM_CACHE}" +docker run --rm --name msa-search \ + --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_PROFILE= \ + -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 ``` + ## Faster Startup: Task-Specific Database Profiles The full database download is ~1.4 TB and can take well over an hour on first launch. If you From 444e32b73f68d44449ad3a0915ab3671412e15b8 Mon Sep 17 00:00:00 2001 From: nil16 Date: Thu, 23 Jul 2026 09:04:00 -0700 Subject: [PATCH 15/15] chore(plugin-sync): regenerate msa-search-nim plugin payload Run scripts/plugin_sync.py --write so the generated plugin copy under plugins/bionemo-agent-toolkit/skills/msa-search-nim/ matches the updated source (SKILL.md fast-path restructure + references). Fixes the plugin-sync CI check. --- .../skills/msa-search-nim/SKILL.md | 226 ++++++++++++++++-- .../skills/msa-search-nim/references/api.md | 6 +- .../msa-search-nim/references/examples.md | 17 ++ .../msa-search-nim/references/parameters.md | 14 ++ 4 files changed, 248 insertions(+), 15 deletions(-) diff --git a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/SKILL.md b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/SKILL.md index 7960e8b..4cfd0a1 100644 --- a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/SKILL.md +++ b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/SKILL.md @@ -1,7 +1,7 @@ --- name: msa-search-nim description: > - Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. + Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. For local deployment, download the databases in parallel with aria2c and launch via NIM_MODEL_NAME (the recommended default fast path, ~14 min vs >80 min for the built-in downloader); a plain docker run uses the slow built-in downloader. license: Apache-2.0 AND CC-BY-4.0 compatibility: "requests>=2.28" allowed-tools: Bash, Read, Write, AskUserQuestion @@ -41,43 +41,241 @@ for template search unless the hosted docs/service changes. ## Local Docker -Local setup requires a GPU and about 1.4 TB / 1660 GB of NVMe storage for -databases. For setup answers, include env preflight, `docker login`, -`docker run`, readiness, and then no-auth local inference. Do not invent a cache -default or drop the `NVIDIA_API_KEY` fallback. +**Default local deployment = parallel download + `NIM_MODEL_NAME`.** The first recipe below +is the one to use for real workflows. It downloads the database(s) with a range-parallel +downloader (aria2c) and starts the NIM against those files — ~14 min for UniRef30 vs >80 min +for the NIM's built-in downloader (measured, H100). Do **not** reach for the plain `docker run` +(the "Fallback" subsection) unless you only want a `databases:pdb70` smoke test or you +deliberately want the NIM to manage its own blob cache. + +Local setup requires a GPU. Size the NVMe volume to the profile you pick (UniRef30 ~490 GB; +full set ~1.4 TB). For setup answers, include env preflight, `docker login`, the parallel +download, `NIM_MODEL_NAME` launch, readiness, and then no-auth local inference. Do not invent a +cache default or drop the `NVIDIA_API_KEY` fallback. ```bash +# --- env preflight (do not drop the NVIDIA_API_KEY fallback) --- set -a [ -f .env ] && . ./.env set +a - if [ -z "${NGC_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then export NGC_API_KEY="$NVIDIA_API_KEY" fi : "${NGC_API_KEY:?Set NGC_API_KEY or NVIDIA_API_KEY}" -: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}" +: "${DB_DIR:=/data/fast-db}" # where the parallel download lands echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin -echo "MSA-Search local databases require about 1.4 TB (1660 GB) of NVMe storage." -mkdir -p "${LOCAL_NIM_CACHE}" -chmod 777 "${LOCAL_NIM_CACHE}" +# --- 1) pick the DB version(s) you need (paired/complex work = uniref30 only) --- +DB_VERSION=uniref30_2302-m18v1 +command -v aria2c >/dev/null || sudo apt-get install -y aria2 +mkdir -p "$DB_DIR" + +# --- 2) parallel download from NGC (see "Parallel Download" section for the all-DB loop) --- +curl -fsS -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${DB_VERSION}/files" \ + -o /tmp/files.json +DB_DIR="$DB_DIR" python3 - <<'PY' +import json, os +d = json.load(open("/tmp/files.json")); dbdir = os.environ["DB_DIR"]; lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), f" dir={dbdir}", f" out={path}"] +open("/tmp/aria.in", "w").write("\n".join(lines) + "\n") +PY +aria2c -i /tmp/aria.in --max-concurrent-downloads=4 --max-connection-per-server=16 \ + --split=16 --min-split-size=1M --continue=true --file-allocation=none + +# --- 3) launch the NIM against the downloaded files (skips the slow built-in download) --- +docker run -d --name msa-search --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_NAME=/databases \ + -v "${DB_DIR}:/databases" \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 +``` + +Readiness: + +```bash +until curl -sf http://localhost:8000/v1/health/ready; do sleep 5; done +``` + +If the DB is already present in `$DB_DIR`, skip steps 1-2 — the launch alone is a ~20 s warm +start. See "Parallel Download For Any Database Set" for the multi-database (`databases:all`) +loop and the full rationale. + +### Fallback: Let The NIM Download Its Own Databases (slower) + +Use this only for a quick `databases:pdb70` smoke test, or when you specifically want the NIM +to manage its own blob cache. It uses the built-in downloader, which is slow on large profiles +(UniRef30 stalled past 80 min in testing). Pin the smallest profile with `NIM_MODEL_PROFILE` +(see "Faster Startup") so it does not fetch the full 1.4 TB. +```bash +: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}" +mkdir -p "${LOCAL_NIM_CACHE}"; chmod 777 "${LOCAL_NIM_CACHE}" docker run --rm --name msa-search \ - --runtime=nvidia \ - --gpus all \ + --runtime=nvidia --gpus all \ -e NGC_API_KEY \ + -e NIM_MODEL_PROFILE= \ -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \ -p 8000:8000 \ nvcr.io/nim/colabfold/msa-search:2 ``` -Readiness: + +## Faster Startup: Task-Specific Database Profiles + +The full database download is ~1.4 TB and can take well over an hour on first launch. If you +only need some databases, select a **task-specific profile** so the NIM downloads just those. +This is the single biggest lever on local startup time. + +List the profiles your image actually ships (hashes change between releases — never hardcode +them): + +```bash +docker run --rm --entrypoint list-model-profiles nvcr.io/nim/colabfold/msa-search:2 +``` + +Then pass the chosen hash with `NIM_MODEL_PROFILE`: + +```bash +docker run --rm --name msa-search \ + --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_PROFILE= \ + -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 +``` + +Profiles available in this image (confirm hashes with `list-model-profiles`): + +| Profile tags | Databases | Best for | Storage | +|---|---|---|---| +| `databases:pdb70` | PDB70 | Quick testing / smoke check | ~100 MB | +| `databases:uniref30` | UniRef30 | **Paired MSA search for complexes** — UniRef30 is the only DB used for species-based pairing | ~500 GB | +| `databases:uniref30,pdb70,pdb` | UniRef30 + PDB70 + PDB structures | Structural template search | ~700 GB | +| `databases:all` (default) | UniRef30 + ColabFold envdb + PDB70 + PDB100 + PDB structures | Full sensitivity, all databases | ~1.2 TB | + +Verify the loaded profile after readiness: + +```bash +curl -s localhost:8000/v1/metadata | jq +``` + +Notes: + +- The request-level `databases` parameter only selects among databases **already + downloaded**; it does NOT change what is fetched at startup. Startup footprint is set by + `NIM_MODEL_PROFILE` alone. +- **Paired search needs UniRef30 only.** `colabfold_envdb_202108` has no taxonomy and cannot + be used for pairing, so `databases:uniref30` is the correct, smallest profile for + complex/paired workflows — it skips the envdb, the largest part of the full set. +- For maximum monomer sensitivity (UniRef30 + envdb merged) you still need `databases:all`; + there is no envdb-inclusive profile smaller than the full set. + +### Custom Or Individual Databases + +To use a single manually downloaded database (or your own MMSeqs2 DB), download it from NGC +and point the NIM at the mount with `NIM_MODEL_NAME` instead of a profile: + +```bash +ngc registry model download-version nim/colabfold/msa-search:uniref30_2302-m18v1 +# then mount the directory and set -e NIM_MODEL_NAME=/databases +``` + +`NIM_MODEL_NAME` **replaces** the profile databases entirely — the NIM uses only what is +under that directory (discovered by scanning for `**/*.idx`). Mount multiple databases under +one parent to combine them. NGC-downloaded databases are pre-indexed for GPU Server; custom +databases must be indexed with `mmseqs createindex` first. Individually downloadable NGC model +versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`, +`pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`. + +## Recommended: Parallel Download For Any Database Set (Fast Deployment) + +**This is the recommended way to download the databases at all — for any profile, +including the full `databases:all` set.** Task-specific profiles cut *what* you +download; this parallel downloader cuts *how long* that download takes. Use it whether +you need one database or all of them. The gain is largest for the `databases:uniref30` profile, which is ~490 GB +dominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB). + +The NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls +each file over roughly one connection. The NGC CDN throttles a single connection to ~20–25 +MB/s, so while the downloader is fetching one of the two giant files, most of its parallel +slots sit idle and throughput collapses to that single-flow rate. Measured on an H100 node, +the built-in path did not reach `/health/ready` in over 80 minutes. + +A range-parallel downloader splits **each file** into many byte-range segments (the NGC CDN +advertises `accept-ranges: bytes`), so a single 241 GB file is pulled over 16 connections at +once — ~15× the single-flow rate. Same node, `aria2c` fetched the full ~490 GB in **~13.5 +minutes**. + +Workflow (download once with aria2, then start the NIM against the files via `NIM_MODEL_NAME`): ```bash -until curl -sf http://localhost:8000/v1/health/ready; do sleep 10; done +# 1) Get presigned file URLs for the individual database model version from NGC. +# (Requires NGC_API_KEY. The response arrays `urls` and `filepath` are positionally paired.) +curl -s -H "Authorization: Bearer $NGC_API_KEY" \ + 'https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/uniref30_2302-m18v1/files' \ + -o files.json + +# 2) Build an aria2 input file (URL + target filename per entry) and download in parallel. +python3 - <<'PY' +import json +d = json.load(open("files.json")) +lines = [] +for url, path in zip(d["urls"], d["filepath"]): + lines += [url.strip(), " dir=/data/fast-db", f" out={path}"] +open("aria.in", "w").write("\n".join(lines) + "\n") +PY +aria2c -i aria.in \ + --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \ + --min-split-size=1M --continue=true --file-allocation=none + +# 3) Start the NIM against the downloaded directory. NIM_MODEL_NAME makes the NIM discover +# databases by scanning for **/*.idx, bypassing the profile/blob cache entirely. +docker run -d --name msa-search --runtime=nvidia --gpus all \ + -e NGC_API_KEY \ + -e NIM_MODEL_NAME=/databases \ + -v /data/fast-db:/databases \ + -p 8000:8000 \ + nvcr.io/nim/colabfold/msa-search:2 ``` +For **all databases** (equivalent to `databases:all`), repeat step 1 for each individual DB +version and download them into sibling directories under one parent, then point +`NIM_MODEL_NAME` at that parent — the NIM discovers every DB by scanning `**/*.idx`: + +```bash +# fetch each DB's file list into /data/all-db// ... then one aria2c per list, e.g.: +for V in uniref30_2302-m18v1 colabfold_envdb_202108-m18v1 pdb70_220313-m18v1 \ + pdb100_230517-m18v1 pdb_20251028_zip-m18v1; do + curl -s -H "Authorization: Bearer $NGC_API_KEY" \ + "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/$V/files" \ + -o "files_$V.json" + # build an aria2 input from files_$V.json (dir=/data/all-db) and run aria2c on it +done +# then launch once against the parent: +# docker run -d ... -e NIM_MODEL_NAME=/databases -v /data/all-db:/databases ... +``` + +The per-connection CDN throttle is the same for every database, so parallel download helps +the full set proportionally — the more you download, the more absolute time it saves. + +Notes: + +- The presigned URLs expire (typically within a day) — build the aria2 input and start the + download promptly after fetching `files.json`. +- Keep the downloaded directory's internal layout intact (e.g. `uniref30_2302/…`); the + `filepath` values already encode it. The NIM needs the `.idx` file plus its companion files + and the small `.UNIREF30_READY` / `*.tar.gz.unpacked` markers. +- The bottleneck is the CDN's per-connection cap, not local disk or CPU — a fast NVMe volume + writes far faster than the network delivers. Raising `--split` / `--max-connection-per-server` + helps only up to the node's aggregate egress ceiling. +- Best of all: download the profile once, then **persist the cache volume** (or this + `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download. + ## Standard MSA Request Use exact case-sensitive database names and response keys. diff --git a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/api.md b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/api.md index 52febac..07bb755 100644 --- a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/api.md +++ b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/api.md @@ -29,6 +29,8 @@ | `e_value` | float | No | `0.0001` | 0.0–1.0 | Hit filtering threshold | | `iterations` | integer | No | `1` | 1–6 | Ignored for cascaded search | | `max_msa_sequences` | integer | No | `500` | 1–500 | Must equal `NIM_GLOBAL_MAX_MSA_DEPTH` when GPU Server enabled | +| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. | +| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. | | `output_alignment_formats` | list[string] | No | `["a3m"]` | — | `"a3m"`, `"fasta"` | ### Response Body @@ -153,7 +155,7 @@ docker run --rm --name msa-search \ | Flag | Value | Notes | |---|---|---| | `--gpus` | `all` | Multi-GPU supported; 48 GB GPUs need ≥2 units | -| Cache mount | `/opt/nim/.cache` | ~1.4 TB for full database download | +| Cache mount | `/opt/nim/.cache` | ~1.4 TB for `databases:all`; far less with a task-specific `NIM_MODEL_PROFILE` (~500 GB UniRef30, ~100 MB PDB70) | | Image | `nvcr.io/nim/colabfold/msa-search:2` | v2.3.0 as of 2025; `:2` is the major version tag | ### Key Environment Variables @@ -161,6 +163,8 @@ docker run --rm --name msa-search \ | Variable | Default | Notes | |---|---|---| | `NIM_GLOBAL_MAX_MSA_DEPTH` | `500` | Must match `max_msa_sequences` in requests (GPU Server mode) | +| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. | +| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. | --- diff --git a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/examples.md b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/examples.md index 5fef666..4f5c176 100644 --- a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/examples.md +++ b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/examples.md @@ -2,6 +2,23 @@ Use these compact patterns when the activated skill needs more examples. +## Fast Local Deployment (recommended) + +When deploying the MSA-Search NIM locally for a paired/complex workflow, prefer the fast +path: download only the UniRef30 database in parallel, then launch against it. A plain +`docker run` uses the NIM's built-in downloader (>80 min for UniRef30); the parallel path +is ~14 min. Example prompt: + +> Deploy the MSA-Search NIM locally using the fast path: download only the UniRef30 +> database (`databases:uniref30` profile) in parallel with aria2c, then start the NIM +> against those files with `NIM_MODEL_NAME`. Look up the current profile hash with +> `list-model-profiles` rather than hardcoding it. If the endpoint is already running with +> the database loaded, skip the redeploy. Then run a paired MSA search for chains A and B +> using `Uniref30_2302` only. + +See the "Recommended For Large Profiles: Parallel Download" section of `SKILL.md` for the +exact aria2c + `NIM_MODEL_NAME` commands. + ## Hosted Standard MSA ```python diff --git a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/parameters.md b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/parameters.md index 6f1685f..dc02a04 100644 --- a/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/parameters.md +++ b/plugins/bionemo-agent-toolkit/skills/msa-search-nim/references/parameters.md @@ -29,3 +29,17 @@ container configuration. - Set `max_structures` to the requested count. - Include `max_msa_sequences=500` unless the local `NIM_GLOBAL_MAX_MSA_DEPTH` was changed. + +## Startup Database Selection (local Docker) + +`databases` in the request body selects among **already-downloaded** databases only. To +control what is downloaded at container startup (and thus storage and launch time), set the +`NIM_MODEL_PROFILE` environment variable to a profile hash from `list-model-profiles`: + +- `databases:pdb70` — ~100 MB, quick test. +- `databases:uniref30` — ~500 GB, paired/complex MSA search (UniRef30 is the only taxonomy + DB used for pairing). +- `databases:uniref30,pdb70,pdb` — ~700 GB, template search. +- `databases:all` — ~1.2 TB, full sensitivity (adds ColabFold envdb, PDB100). + +Hashes change between NIM releases — always read them from `list-model-profiles`.