Add IMGT, OGRDB, and airrc-imgt germline reference sources

.github/workflows/check-apis.yml

@@ -0,0 +1,42 @@
+name: check-apis
+
+# A weekly canary against the live IMGT and OGRDB APIs. It runs the same code a
+# user's download would, using the endpoint constants the sources define, so a
+# red run here is the early warning that an upstream API changed shape. It never
+# blocks a merge; it only watches. workflow_dispatch allows an on-demand check.
+on:
+  schedule:
+    - cron: "0 6 * * 1"   # Mondays, 06:00 UTC
+  workflow_dispatch:
+
+concurrency:
+  group: check-apis
+  cancel-in-progress: true
+
+jobs:
+  probe:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install package
+        run: |
+          python -m pip install --upgrade pip
+          pip install .
+      - name: Probe the reference APIs
+        env:
+          SOURCERER_LIVE: '1'
+        run: python -m unittest tests.test_live -v
+      # On failure the run goes red and GitHub notifies the watchers. Opening an
+      # issue automatically is intentionally left off; enable the step below if a
+      # tracked issue is wanted instead of (or as well as) the email.
+      #
+      # - name: Open an issue on failure
+      #   if: failure()
+      #   uses: JasonEtco/create-an-issue@v2
+      #   env:
+      #     GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      #   with:
+      #     filename: .github/api-breakage-issue.md

NEWS.rst

@@ -26,4 +26,24 @@ General:
 + Added a download provenance record (what was fetched, from where, when, and
   its hash) written alongside every download.
 
+Germline references:
+
++ Added germline reference sources IMGT_ (``sourcerer imgt``) and OGRDB_
+  (``sourcerer ogrdb``, also reachable as ``sourcerer airrc``), and an
+  ``airrc-imgt`` blend that takes immunoglobulin V, D and J from OGRDB's AIRR-C
+  sets and the T-cell receptor and remaining constants from IMGT.
++ Added ``sourcerer <source> download <species>``, which writes the germline
+  ``reference_base`` in the `nf-core/airrflow`_ layout, and ``--igblast`` to also
+  build the IgBLAST databases (``makeblastdb`` plus the NCBI internal_data and
+  optional_file trees).
++ Added ``sourcerer reference build``, to validate a germline reference folder
+  -- in the ``reference_base`` layout or a flat folder of FASTAs -- and build
+  its IgBLAST databases, with ``--check`` to validate without building.
++ nf-core/airrflow can fetch germlines through ``sourcerer`` for its ``imgt``
+  and ``airrc-imgt`` database types, or consume a ``sourcerer``-built reference
+  passed to ``--reference_fasta`` / ``--reference_igblast``.
+
 .. _OAS: https://opig.stats.ox.ac.uk/webapps/oas/
+.. _IMGT: https://www.imgt.org/genedb/
+.. _OGRDB: https://ogrdb.airr-community.org/
+.. _nf-core/airrflow: https://nf-co.re/airrflow

README.rst

@@ -2,11 +2,21 @@ sourcerer
 ================================================================================
 
 ``sourcerer`` downloads data from online immune repertoire databases and formats
-it for use with the Immcantation_ framework. Each external source is a module;
-the first is OAS_ (Observed Antibody Space).
+it for use with the Immcantation_ framework and `nf-core/airrflow`_. Each
+external source is a module, and sources come in two kinds:
+
+- *dataset* sources such as OAS_ (Observed Antibody Space) download sequencing
+  data and write an airrflow samplesheet;
+- *germline reference* sources -- IMGT_, OGRDB_, and an ``airrc-imgt`` blend of
+  the two -- download germline sets and build the ``reference_base`` and IgBLAST
+  databases airrflow consumes. ``sourcerer reference`` can also validate and
+  build those databases from a reference folder you already have.
 
 .. _Immcantation: https://immcantation.readthedocs.io
+.. _nf-core/airrflow: https://nf-co.re/airrflow
 .. _OAS: https://opig.stats.ox.ac.uk/webapps/oas/
+.. _IMGT: https://www.imgt.org/genedb/
+.. _OGRDB: https://ogrdb.airr-community.org/
 
 Why
 --------------------------------------------------------------------------------
@@ -25,6 +35,8 @@ a reviewable diff and a failing test, not as a silently wrong download.
 Usage
 --------------------------------------------------------------------------------
 
+Datasets (OAS), producing an airrflow samplesheet:
+
 .. code-block:: bash
 
     sourcerer --version
@@ -38,6 +50,25 @@ Usage
         --outdir airrflow_out -c ../airrflow.config \
         --clonal_threshold 0.2 -resume
 
+Germline references, producing the ``reference_base`` and IgBLAST databases:
+
+.. code-block:: bash
+
+    # IMGT germline for a species, and (with --igblast) the IgBLAST databases
+    sourcerer imgt download human --outdir ref --igblast
+
+    # the AIRR-C sets blended with IMGT (immunoglobulin from OGRDB, TR and the
+    # remaining constants from IMGT) -- the airrflow airrc-imgt reference
+    sourcerer airrc-imgt download human --outdir ref --igblast
+
+    # validate a germline folder someone provided and build its databases;
+    # --check validates only, without makeblastdb
+    sourcerer reference build ref/reference_base --out igblast_base --check
+
+nf-core/airrflow uses the result either way: point ``--reference_fasta`` and
+``--reference_igblast`` at ``ref/reference_base`` and ``igblast_base`` with
+``--fetch_germlines none``.
+
 
 License
 --------------------------------------------------------------------------------

docs/api.rst

@@ -13,9 +13,14 @@ API
    modules/Catalog
    modules/Convert
    modules/Airrflow
+   modules/Reference
    modules/Provenance
    modules/Gzip
    modules/Exceptions
    modules/Sources
    modules/SourcesBase
+   modules/SourcesGermline
    modules/SourcesOas
+   modules/SourcesImgt
+   modules/SourcesOgrdb
+   modules/SourcesAirrcImgt

docs/info.rst

@@ -37,6 +37,32 @@ asks that both of the following be cited:
   *Protein Sci*. 2022;31(1):141-146.
   doi:`10.1002/pro.4205 <https://doi.org/10.1002/pro.4205>`__
 
+IMGT
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Data from `IMGT <https://www.imgt.org/genedb/>`__ is governed by the `IMGT terms
+of use <https://www.imgt.org/about/termsofuse.php>`__: it is free for academic
+research on condition that IMGT is cited. IMGT asks that the following be cited:
+
+- Lefranc MP, Giudicelli V, Duroux P, et al. IMGT, the international
+  ImMunoGeneTics information system 25 years on. *Nucleic Acids Res*.
+  2015;43(Database issue):D413-D422.
+  doi:`10.1093/nar/gku1056 <https://doi.org/10.1093/nar/gku1056>`__
+
+The ``airrc-imgt`` blend uses IMGT data too, so its downloads carry this
+obligation as well as the OGRDB one below.
+
+OGRDB
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Data from `OGRDB <https://ogrdb.airr-community.org/>`__ is distributed under a
+`CC BY 4.0 <https://creativecommons.org/licenses/by/4.0/>`__ license. In
+exchange, OGRDB asks that the following be cited:
+
+- Lees WD, Busse CE, Corcoran M, et al. OGRDB: a reference database of inferred
+  immune receptor genes. *Nucleic Acids Res*. 2020;48(D1):D964-D970.
+  doi:`10.1093/nar/gkz822 <https://doi.org/10.1093/nar/gkz822>`__
+
 
 License
 --------------------------------------------------------------------------------

docs/modules/Reference.rst

@@ -0,0 +1,7 @@
+sourcerer.Reference
+-------------------
+
+.. automodule:: sourcerer.Reference
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/modules/SourcesAirrcImgt.rst

@@ -0,0 +1,7 @@
+sourcerer.Sources.AirrcImgt
+---------------------------
+
+.. automodule:: sourcerer.Sources.AirrcImgt
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/modules/SourcesGermline.rst

@@ -0,0 +1,7 @@
+sourcerer.Sources.Germline
+--------------------------
+
+.. automodule:: sourcerer.Sources.Germline
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/modules/SourcesImgt.rst

@@ -0,0 +1,7 @@
+sourcerer.Sources.Imgt
+----------------------
+
+.. automodule:: sourcerer.Sources.Imgt
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/modules/SourcesOgrdb.rst

@@ -0,0 +1,7 @@
+sourcerer.Sources.Ogrdb
+-----------------------
+
+.. automodule:: sourcerer.Sources.Ogrdb
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/usage/airrc-imgt.rst

@@ -0,0 +1,15 @@
+.. _UsageAirrcImgt:
+
+sourcerer airrc-imgt
+================================================================================
+
+The AIRR-C germline sets blended with IMGT: immunoglobulin V, D and J from
+OGRDB, and everything OGRDB does not cover -- all of the T-cell receptor, and
+the immunoglobulin constants without a published set -- from IMGT. Offers
+``human`` and ``mouse`` collections. ``download`` writes an airrflow
+``reference_base`` mixing ``airrc_`` and ``imgt_`` files; ``--igblast``
+additionally builds the IgBLAST databases.
+
+.. autoprogram:: sourcerer.Cli:getArgParser()
+   :prog: sourcerer
+   :start_command: airrc-imgt

docs/usage/imgt.rst

@@ -0,0 +1,14 @@
+.. _UsageImgt:
+
+sourcerer imgt
+================================================================================
+
+`IMGT/GENE-DB <https://www.imgt.org/genedb/>`__: germline V, D, J and C
+reference sequences. Offers ``human`` and ``mouse`` collections, each narrowed
+with the ``--locus`` and ``--segment`` filters below. ``download`` writes an
+airrflow ``reference_base``; ``--igblast`` additionally builds the IgBLAST
+databases, which needs ``makeblastdb`` on the path.
+
+.. autoprogram:: sourcerer.Cli:getArgParser()
+   :prog: sourcerer
+   :start_command: imgt

docs/usage/index.rst

@@ -4,17 +4,26 @@ Commandline Usage
 ================================================================================
 
 ``sourcerer`` is a single command with a subcommand tree: one subcommand per
-external source (``oas`` today), a ``schema`` subcommand to inspect and
-re-harvest the stored snapshot each source is built from, and a ``sources``
-subcommand that lists what is registered.
+external source (``oas``, ``imgt``, ``ogrdb`` and ``airrc-imgt`` today), a
+``schema`` subcommand
+to inspect and re-harvest the stored snapshot each source is built from, a
+``reference`` subcommand to validate a germline reference folder and build its
+IgBLAST databases, and a ``sources`` subcommand that lists what is registered.
+
+Sources come in two kinds. A repertoire source such as ``oas`` downloads
+sequencing data and writes an airrflow samplesheet; a germline reference source
+such as ``imgt`` and ``ogrdb`` downloads germline sets and writes an airrflow
+``reference_base``, optionally building the IgBLAST databases with ``--igblast``.
+The ``reference`` subcommand does that same build for a reference folder supplied
+by hand, in either the ``reference_base`` layout or a flat folder of FASTAs.
 
 Every source subcommand exposes the same two actions, ``search`` and
-``download``, each taking a collection (for example ``paired`` or
-``unpaired``) as a further subcommand. The filter flags under a collection —
-``--species``, ``--disease``, and so on — are not hardcoded: they are
-generated at parser-construction time from the checked-in schema snapshot
-described in :ref:`API`, which is also why they appear below exactly as they
-would in ``--help`` on the machine building these docs.
+``download``, each taking a collection (for example ``paired`` and ``unpaired``
+for OAS, or a species for the germline sources) as a further subcommand. The
+filter flags under a collection — ``--species``, ``--locus``, and so on — are
+not hardcoded: they are generated at parser-construction time from the checked-in
+schema snapshot described in :ref:`API`, which is also why they appear below
+exactly as they would in ``--help`` on the machine building these docs.
 
 Commands are documented one page per top level subcommand, mirroring how the
 commandline itself groups them: :doc:`sources` and :doc:`schema` apply
@@ -27,4 +36,8 @@ one page here alongside it.
 
    sources
    schema
+   reference
    oas
+   imgt
+   ogrdb
+   airrc-imgt

docs/usage/ogrdb.rst

@@ -0,0 +1,19 @@
+.. _UsageOgrdb:
+
+sourcerer ogrdb
+================================================================================
+
+`OGRDB <https://ogrdb.airr-community.org/>`__: AIRR Community curated
+immunoglobulin germline sets. Offers ``human`` and ``mouse`` collections,
+narrowed with the ``--locus`` filter below. ``download`` writes an airrflow
+``reference_base``; ``--igblast`` additionally builds the IgBLAST databases,
+which needs ``makeblastdb`` on the path.
+
+OGRDB is the AIRR Community database, so ``ogrdb`` also answers to the alias
+``airrc`` (``sourcerer airrc download ...``). For a reference that additionally
+fills in the T-cell receptor and the remaining constants from IMGT, use the
+``airrc-imgt`` source instead.
+
+.. autoprogram:: sourcerer.Cli:getArgParser()
+   :prog: sourcerer
+   :start_command: ogrdb

docs/usage/reference.rst

@@ -0,0 +1,16 @@
+.. _UsageReference:
+
+sourcerer reference
+================================================================================
+
+Validate a folder of germline FASTAs and build the IgBLAST databases from it,
+for a reference someone supplies rather than one sourcerer downloaded. Files are
+recognised by name in any directory layout --
+``[<prefix>_][aa_]<species>_<CHAIN>.fasta``, for example ``human_IGHV.fasta`` or
+``imgt_human_IGHV.fasta`` -- so a nested ``reference_base`` and a flat folder both
+work. ``--check`` validates and reports what would build without building
+anything, and needs no ``makeblastdb``.
+
+.. autoprogram:: sourcerer.Cli:getArgParser()
+   :prog: sourcerer
+   :start_command: reference