docs(msa-search-nim): document task-specific database profiles for faster startup

nim-skills/msa-search-nim/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: msa-search-nim
 description: >
-  Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment.
+  Generate multiple sequence alignments (MSAs) for protein sequences using the ColabFold MSA-Search NIM. Use for homolog search, UniRef30/ColabFold env searches, A3M or FASTA alignments, paired MSA search for complexes, PDB70 structural templates, hosted NVIDIA API calls, or local Docker deployment. For local deployment, download the databases in parallel with aria2c and launch via NIM_MODEL_NAME (the recommended default fast path, ~14 min vs >80 min for the built-in downloader); a plain docker run uses the slow built-in downloader.
 license: Apache-2.0 AND CC-BY-4.0
 compatibility: "requests>=2.28"
 allowed-tools: Bash, Read, Write, AskUserQuestion
@@ -41,43 +41,241 @@ for template search unless the hosted docs/service changes.
 
 ## Local Docker
 
-Local setup requires a GPU and about 1.4 TB / 1660 GB of NVMe storage for
-databases. For setup answers, include env preflight, `docker login`,
-`docker run`, readiness, and then no-auth local inference. Do not invent a cache
-default or drop the `NVIDIA_API_KEY` fallback.
+**Default local deployment = parallel download + `NIM_MODEL_NAME`.** The first recipe below
+is the one to use for real workflows. It downloads the database(s) with a range-parallel
+downloader (aria2c) and starts the NIM against those files — ~14 min for UniRef30 vs >80 min
+for the NIM's built-in downloader (measured, H100). Do **not** reach for the plain `docker run`
+(the "Fallback" subsection) unless you only want a `databases:pdb70` smoke test or you
+deliberately want the NIM to manage its own blob cache.
+
+Local setup requires a GPU. Size the NVMe volume to the profile you pick (UniRef30 ~490 GB;
+full set ~1.4 TB). For setup answers, include env preflight, `docker login`, the parallel
+download, `NIM_MODEL_NAME` launch, readiness, and then no-auth local inference. Do not invent a
+cache default or drop the `NVIDIA_API_KEY` fallback.
 
 ```bash
+# --- env preflight (do not drop the NVIDIA_API_KEY fallback) ---
 set -a
 [ -f .env ] && . ./.env
 set +a
-
 if [ -z "${NGC_API_KEY:-}" ] && [ -n "${NVIDIA_API_KEY:-}" ]; then
   export NGC_API_KEY="$NVIDIA_API_KEY"
 fi
 : "${NGC_API_KEY:?Set NGC_API_KEY or NVIDIA_API_KEY}"
-: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}"
+: "${DB_DIR:=/data/fast-db}"          # where the parallel download lands
 
 echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
 
-echo "MSA-Search local databases require about 1.4 TB (1660 GB) of NVMe storage."
-mkdir -p "${LOCAL_NIM_CACHE}"
-chmod 777 "${LOCAL_NIM_CACHE}"
+# --- 1) pick the DB version(s) you need (paired/complex work = uniref30 only) ---
+DB_VERSION=uniref30_2302-m18v1
+command -v aria2c >/dev/null || sudo apt-get install -y aria2
+mkdir -p "$DB_DIR"
+
+# --- 2) parallel download from NGC (see "Parallel Download" section for the all-DB loop) ---
+curl -fsS -H "Authorization: Bearer $NGC_API_KEY" \
+  "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/${DB_VERSION}/files" \
+  -o /tmp/files.json
+DB_DIR="$DB_DIR" python3 - <<'PY'
+import json, os
+d = json.load(open("/tmp/files.json")); dbdir = os.environ["DB_DIR"]; lines = []
+for url, path in zip(d["urls"], d["filepath"]):
+    lines += [url.strip(), f"  dir={dbdir}", f"  out={path}"]
+open("/tmp/aria.in", "w").write("\n".join(lines) + "\n")
+PY
+aria2c -i /tmp/aria.in --max-concurrent-downloads=4 --max-connection-per-server=16 \
+  --split=16 --min-split-size=1M --continue=true --file-allocation=none
+
+# --- 3) launch the NIM against the downloaded files (skips the slow built-in download) ---
+docker run -d --name msa-search --runtime=nvidia --gpus all \
+  -e NGC_API_KEY \
+  -e NIM_MODEL_NAME=/databases \
+  -v "${DB_DIR}:/databases" \
+  -p 8000:8000 \
+  nvcr.io/nim/colabfold/msa-search:2
+```
+
+Readiness:
+
+```bash
+until curl -sf http://localhost:8000/v1/health/ready; do sleep 5; done
+```
+
+If the DB is already present in `$DB_DIR`, skip steps 1-2 — the launch alone is a ~20 s warm
+start. See "Parallel Download For Any Database Set" for the multi-database (`databases:all`)
+loop and the full rationale.
+
+### Fallback: Let The NIM Download Its Own Databases (slower)
+
+Use this only for a quick `databases:pdb70` smoke test, or when you specifically want the NIM
+to manage its own blob cache. It uses the built-in downloader, which is slow on large profiles
+(UniRef30 stalled past 80 min in testing). Pin the smallest profile with `NIM_MODEL_PROFILE`
+(see "Faster Startup") so it does not fetch the full 1.4 TB.
 
+```bash
+: "${LOCAL_NIM_CACHE:?Set LOCAL_NIM_CACHE}"
+mkdir -p "${LOCAL_NIM_CACHE}"; chmod 777 "${LOCAL_NIM_CACHE}"
 docker run --rm --name msa-search \
-  --runtime=nvidia \
-  --gpus all \
+  --runtime=nvidia --gpus all \
   -e NGC_API_KEY \
+  -e NIM_MODEL_PROFILE=<hash-from-list-model-profiles> \
   -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \
   -p 8000:8000 \
   nvcr.io/nim/colabfold/msa-search:2
 ```
 
-Readiness:
+
+## Faster Startup: Task-Specific Database Profiles
+
+The full database download is ~1.4 TB and can take well over an hour on first launch. If you
+only need some databases, select a **task-specific profile** so the NIM downloads just those.
+This is the single biggest lever on local startup time.
+
+List the profiles your image actually ships (hashes change between releases — never hardcode
+them):
+
+```bash
+docker run --rm --entrypoint list-model-profiles nvcr.io/nim/colabfold/msa-search:2
+```
+
+Then pass the chosen hash with `NIM_MODEL_PROFILE`:
+
+```bash
+docker run --rm --name msa-search \
+  --runtime=nvidia --gpus all \
+  -e NGC_API_KEY \
+  -e NIM_MODEL_PROFILE=<hash-from-list-model-profiles> \
+  -v "${LOCAL_NIM_CACHE}:/opt/nim/.cache" \
+  -p 8000:8000 \
+  nvcr.io/nim/colabfold/msa-search:2
+```
+
+Profiles available in this image (confirm hashes with `list-model-profiles`):
+
+| Profile tags | Databases | Best for | Storage |
+|---|---|---|---|
+| `databases:pdb70` | PDB70 | Quick testing / smoke check | ~100 MB |
+| `databases:uniref30` | UniRef30 | **Paired MSA search for complexes** — UniRef30 is the only DB used for species-based pairing | ~500 GB |
+| `databases:uniref30,pdb70,pdb` | UniRef30 + PDB70 + PDB structures | Structural template search | ~700 GB |
+| `databases:all` (default) | UniRef30 + ColabFold envdb + PDB70 + PDB100 + PDB structures | Full sensitivity, all databases | ~1.2 TB |
+
+Verify the loaded profile after readiness:
+
+```bash
+curl -s localhost:8000/v1/metadata | jq
+```
+
+Notes:
+
+- The request-level `databases` parameter only selects among databases **already
+  downloaded**; it does NOT change what is fetched at startup. Startup footprint is set by
+  `NIM_MODEL_PROFILE` alone.
+- **Paired search needs UniRef30 only.** `colabfold_envdb_202108` has no taxonomy and cannot
+  be used for pairing, so `databases:uniref30` is the correct, smallest profile for
+  complex/paired workflows — it skips the envdb, the largest part of the full set.
+- For maximum monomer sensitivity (UniRef30 + envdb merged) you still need `databases:all`;
+  there is no envdb-inclusive profile smaller than the full set.
+
+### Custom Or Individual Databases
+
+To use a single manually downloaded database (or your own MMSeqs2 DB), download it from NGC
+and point the NIM at the mount with `NIM_MODEL_NAME` instead of a profile:
+
+```bash
+ngc registry model download-version nim/colabfold/msa-search:uniref30_2302-m18v1
+# then mount the directory and set -e NIM_MODEL_NAME=/databases
+```
+
+`NIM_MODEL_NAME` **replaces** the profile databases entirely — the NIM uses only what is
+under that directory (discovered by scanning for `**/*.idx`). Mount multiple databases under
+one parent to combine them. NGC-downloaded databases are pre-indexed for GPU Server; custom
+databases must be indexed with `mmseqs createindex` first. Individually downloadable NGC model
+versions: `uniref30_2302-m18v1`, `colabfold_envdb_202108-m18v1`, `pdb70_220313-m18v1`,
+`pdb100_230517-m18v1`, `pdb_20251028_zip-m18v1`.
+
+## Recommended: Parallel Download For Any Database Set (Fast Deployment)
+
+**This is the recommended way to download the databases at all — for any profile,
+including the full `databases:all` set.** Task-specific profiles cut *what* you
+download; this parallel downloader cuts *how long* that download takes. Use it whether
+you need one database or all of them. The gain is largest for the `databases:uniref30` profile, which is ~490 GB
+dominated by two very large files (a ~241 GB GPU index and a ~134 GB sequence DB).
+
+The NIM's built-in downloader parallelizes **across files** (`max_parallel_files=10`) but pulls
+each file over roughly one connection. The NGC CDN throttles a single connection to ~20–25
+MB/s, so while the downloader is fetching one of the two giant files, most of its parallel
+slots sit idle and throughput collapses to that single-flow rate. Measured on an H100 node,
+the built-in path did not reach `/health/ready` in over 80 minutes.
+
+A range-parallel downloader splits **each file** into many byte-range segments (the NGC CDN
+advertises `accept-ranges: bytes`), so a single 241 GB file is pulled over 16 connections at
+once — ~15× the single-flow rate. Same node, `aria2c` fetched the full ~490 GB in **~13.5
+minutes**.
+
+Workflow (download once with aria2, then start the NIM against the files via `NIM_MODEL_NAME`):
 
 ```bash
-until curl -sf http://localhost:8000/v1/health/ready; do sleep 10; done
+# 1) Get presigned file URLs for the individual database model version from NGC.
+#    (Requires NGC_API_KEY. The response arrays `urls` and `filepath` are positionally paired.)
+curl -s -H "Authorization: Bearer $NGC_API_KEY" \
+  'https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/uniref30_2302-m18v1/files' \
+  -o files.json
+
+# 2) Build an aria2 input file (URL + target filename per entry) and download in parallel.
+python3 - <<'PY'
+import json
+d = json.load(open("files.json"))
+lines = []
+for url, path in zip(d["urls"], d["filepath"]):
+    lines += [url.strip(), "  dir=/data/fast-db", f"  out={path}"]
+open("aria.in", "w").write("\n".join(lines) + "\n")
+PY
+aria2c -i aria.in \
+  --max-concurrent-downloads=4 --max-connection-per-server=16 --split=16 \
+  --min-split-size=1M --continue=true --file-allocation=none
+
+# 3) Start the NIM against the downloaded directory. NIM_MODEL_NAME makes the NIM discover
+#    databases by scanning for **/*.idx, bypassing the profile/blob cache entirely.
+docker run -d --name msa-search --runtime=nvidia --gpus all \
+  -e NGC_API_KEY \
+  -e NIM_MODEL_NAME=/databases \
+  -v /data/fast-db:/databases \
+  -p 8000:8000 \
+  nvcr.io/nim/colabfold/msa-search:2
 ```
 
+For **all databases** (equivalent to `databases:all`), repeat step 1 for each individual DB
+version and download them into sibling directories under one parent, then point
+`NIM_MODEL_NAME` at that parent — the NIM discovers every DB by scanning `**/*.idx`:
+
+```bash
+# fetch each DB's file list into /data/all-db/<db>/ ... then one aria2c per list, e.g.:
+for V in uniref30_2302-m18v1 colabfold_envdb_202108-m18v1 pdb70_220313-m18v1 \
+         pdb100_230517-m18v1 pdb_20251028_zip-m18v1; do
+  curl -s -H "Authorization: Bearer $NGC_API_KEY" \
+    "https://api.ngc.nvidia.com/v2/org/nim/team/colabfold/models/msa-search/$V/files" \
+    -o "files_$V.json"
+  # build an aria2 input from files_$V.json (dir=/data/all-db) and run aria2c on it
+done
+# then launch once against the parent:
+#   docker run -d ... -e NIM_MODEL_NAME=/databases -v /data/all-db:/databases ...
+```
+
+The per-connection CDN throttle is the same for every database, so parallel download helps
+the full set proportionally — the more you download, the more absolute time it saves.
+
+Notes:
+
+- The presigned URLs expire (typically within a day) — build the aria2 input and start the
+  download promptly after fetching `files.json`.
+- Keep the downloaded directory's internal layout intact (e.g. `uniref30_2302/…`); the
+  `filepath` values already encode it. The NIM needs the `.idx` file plus its companion files
+  and the small `.UNIREF30_READY` / `*.tar.gz.unpacked` markers.
+- The bottleneck is the CDN's per-connection cap, not local disk or CPU — a fast NVMe volume
+  writes far faster than the network delivers. Raising `--split` / `--max-connection-per-server`
+  helps only up to the node's aggregate egress ceiling.
+- Best of all: download the profile once, then **persist the cache volume** (or this
+  `fast-db` directory) and mount it on future nodes for a ~20 s warm start with no re-download.
+
 ## Standard MSA Request
 
 Use exact case-sensitive database names and response keys.

nim-skills/msa-search-nim/references/api.md

@@ -29,6 +29,8 @@
 | `e_value` | float | No | `0.0001` | 0.0–1.0 | Hit filtering threshold |
 | `iterations` | integer | No | `1` | 1–6 | Ignored for cascaded search |
 | `max_msa_sequences` | integer | No | `500` | 1–500 | Must equal `NIM_GLOBAL_MAX_MSA_DEPTH` when GPU Server enabled |
+| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. |
+| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. |
 | `output_alignment_formats` | list[string] | No | `["a3m"]` | — | `"a3m"`, `"fasta"` |
 
 ### Response Body
@@ -153,14 +155,16 @@ docker run --rm --name msa-search \
 | Flag | Value | Notes |
 |---|---|---|
 | `--gpus` | `all` | Multi-GPU supported; 48 GB GPUs need ≥2 units |
-| Cache mount | `/opt/nim/.cache` | ~1.4 TB for full database download |
+| Cache mount | `/opt/nim/.cache` | ~1.4 TB for `databases:all`; far less with a task-specific `NIM_MODEL_PROFILE` (~500 GB UniRef30, ~100 MB PDB70) |
 | Image | `nvcr.io/nim/colabfold/msa-search:2` | v2.3.0 as of 2025; `:2` is the major version tag |
 
 ### Key Environment Variables
 
 | Variable | Default | Notes |
 |---|---|---|
 | `NIM_GLOBAL_MAX_MSA_DEPTH` | `500` | Must match `max_msa_sequences` in requests (GPU Server mode) |
+| `NIM_MODEL_PROFILE` | (auto: `databases:all`) | Set to a hash from `list-model-profiles` to download only that profile's databases — cuts storage & startup time. `databases:uniref30` (~500 GB) is the paired-search profile; `databases:pdb70` (~100 MB) is a smoke test. |
+| `NIM_MODEL_NAME` | (unset) | Path to a mounted directory of custom/manually-downloaded MMSeqs2 databases. Completely replaces profile databases; NIM uses only DBs found under this path. |
 
 ---
 

nim-skills/msa-search-nim/references/examples.md

@@ -2,6 +2,23 @@
 
 Use these compact patterns when the activated skill needs more examples.
 
+## Fast Local Deployment (recommended)
+
+When deploying the MSA-Search NIM locally for a paired/complex workflow, prefer the fast
+path: download only the UniRef30 database in parallel, then launch against it. A plain
+`docker run` uses the NIM's built-in downloader (>80 min for UniRef30); the parallel path
+is ~14 min. Example prompt:
+
+> Deploy the MSA-Search NIM locally using the fast path: download only the UniRef30
+> database (`databases:uniref30` profile) in parallel with aria2c, then start the NIM
+> against those files with `NIM_MODEL_NAME`. Look up the current profile hash with
+> `list-model-profiles` rather than hardcoding it. If the endpoint is already running with
+> the database loaded, skip the redeploy. Then run a paired MSA search for chains A and B
+> using `Uniref30_2302` only.
+
+See the "Recommended For Large Profiles: Parallel Download" section of `SKILL.md` for the
+exact aria2c + `NIM_MODEL_NAME` commands.
+
 ## Hosted Standard MSA
 
 ```python

nim-skills/msa-search-nim/references/parameters.md

@@ -29,3 +29,17 @@ container configuration.
 - Set `max_structures` to the requested count.
 - Include `max_msa_sequences=500` unless the local `NIM_GLOBAL_MAX_MSA_DEPTH`
   was changed.
+
+## Startup Database Selection (local Docker)
+
+`databases` in the request body selects among **already-downloaded** databases only. To
+control what is downloaded at container startup (and thus storage and launch time), set the
+`NIM_MODEL_PROFILE` environment variable to a profile hash from `list-model-profiles`:
+
+- `databases:pdb70` — ~100 MB, quick test.
+- `databases:uniref30` — ~500 GB, paired/complex MSA search (UniRef30 is the only taxonomy
+  DB used for pairing).
+- `databases:uniref30,pdb70,pdb` — ~700 GB, template search.
+- `databases:all` — ~1.2 TB, full sensitivity (adds ColabFold envdb, PDB100).
+
+Hashes change between NIM releases — always read them from `list-model-profiles`.