Modified AWS directory

Author: hadiparsianNIH

AWS/Submodule_00_background.ipynb

@@ -109,7 +109,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "display_quiz(\"Transcriptome-Assembly-Refinement-and-Applications/quiz-material/00-pc1.json\")"
+    "display_quiz(\"../quiz-material/00-pc1.json\")"
    ]
   },
   {
@@ -281,7 +281,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "display_quiz(\"Transcriptome-Assembly-Refinement-and-Applications/quiz-material/00-cp1.json\", shuffle_questions = True)"
+    "display_quiz(\"../quiz-material/00-cp1.json\", shuffle_questions = True)"
    ]
   },
   {
@@ -407,7 +407,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "display_quiz(\"Transcriptome-Assembly-Refinement-and-Applications/quiz-material/00-cp2.json\", shuffle_questions = True)"
+    "display_quiz(\"../quiz-material/00-cp2.json\", shuffle_questions = True)"
    ]
   },
   {

AWS/Submodule_01_prog_setup.ipynb

@@ -30,12 +30,12 @@
     "\n",
     "2. **Set up the necessary software:** Students will install and configure essential tools including:\n",
     "    * Java (a prerequisite for Nextflow).\n",
-    "    * Mambaforge (a package manager for bioinformatics tools).\n",
+    "    * Miniforge (a package manager for bioinformatics tools).\n",
     "    *  `sra-tools`, `perl-dbd-sqlite`, and `perl-dbi` (specific bioinformatics packages).\n",
     "    * Nextflow (a workflow management system).\n",
-    "    *  `aws s3` (for interacting with AWS S3 Storage).\n",
+    "    *  `gsutil` (for interacting with Google Cloud Storage).\n",
     "\n",
-    "3. **Download and organize necessary data:** Students will download the TransPi transcriptome assembly software and its associated resources (databases, scripts, configuration files) from an S3 bucket.  This includes understanding the directory structure and file organization.\n",
+    "3. **Download and organize necessary data:** Students will download the TransPi transcriptome assembly software and its associated resources (databases, scripts, configuration files) from a Google Cloud Storage bucket.  This includes understanding the directory structure and file organization.\n",
     "\n",
     "4. **Manage file permissions:** Students will use the `chmod` command to set executable permissions for the necessary files and directories within the TransPi software.\n",
     "\n",
@@ -53,7 +53,7 @@
     "* **Shell Access:**  The ability to execute shell commands from within the Jupyter Notebook environment (using `!` and `%`).\n",
     "* **Java Development Kit (JDK):**  Required for Nextflow.\n",
     "* **Miniforge** A package manager for installing bioinformatics tools.\n",
-    "* **`aws s3`:** The AWS command-line tool. This is crucial for downloading data from an S3 storage bucket."
+    "* **`gsutil`:** The Google Cloud Storage command-line tool. This is crucial for downloading data from Google Cloud Storage."
    ]
   },
   {
@@ -104,7 +104,7 @@
     "## Time to begin!\n",
     "\n",
     "**Step 1:** To start, make sure that you are in the right starting place with a `cd`.\n",
-    "> `pwd` prints our current local working directory. Make sure the output from the command is: `/home/ec2-user/SageMaker`"
+    "> `pwd` prints our current local working directory. Make sure the output from the command is: `/home/jupyter`"
    ]
   },
   {
@@ -114,7 +114,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%cd /home/ec2-user/SageMaker"
+    "%cd /home/jupyter"
    ]
   },
   {
@@ -147,12 +147,50 @@
     "! java -version"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "7b3ffb16-3395-4c01-9774-ee568e815490",
+   "metadata": {},
+   "source": [
+    "**Step 3:** Install Miniforge (a package manager), which is needed to support the information held within the TransPi databases."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ac5b204a-f0db-4ceb-bf37-57eca6d77974",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh\n",
+    "! bash Miniforge3-$(uname)-$(uname -m).sh -b -p $HOME/miniforge"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "c5584e2e",
+   "metadata": {},
+   "source": [
+    "Next, add it to the path."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "ad030cd1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "os.environ[\"PATH\"] += os.pathsep + os.environ[\"HOME\"]+\"/miniforge/bin\""
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "7b930ad7",
    "metadata": {},
    "source": [
-    "**Step 3:** Using Mamba and bioconda, install the tools that will be used in this tutorial."
+    "Next, using Miniforge and bioconda, install the tools that will be used in this tutorial."
    ]
   },
   {
@@ -201,7 +239,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "! aws s3 cp  --recursive s3://nigms-sandbox/nosi-inbremaine-storage/TransPi ./TransPi"
+    "! gsutil -m cp -r gs://nigms-sandbox/nosi-inbremaine-storage/TransPi ./"
    ]
   },
   {
@@ -211,10 +249,10 @@
    "source": [
     "<div class=\"alert alert-block alert-success\">\n",
     "    <i class=\"fa fa-hand-paper-o\" aria-hidden=\"true\"></i>\n",
-    "    <b>Note: </b>  aws\n",
+    "    <b>Note: </b>  gsutil\n",
     "</div>\n",
     "\n",
-    ">`aws s3` is a tool allows you to interact with S3 Storage through the command line."
+    ">`gsutil` is a tool allows you to interact with Google Cloud Storage through the command line."
    ]
   },
   {
@@ -239,7 +277,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "! aws s3 cp --recursive s3://nigms-sandbox/nosi-inbremaine-storage/resources ./resources"
+    "! gsutil -m cp -r gs://nigms-sandbox/nosi-inbremaine-storage/resources ./"
    ]
   },
   {
@@ -264,7 +302,8 @@
     ">    - They can also be stacked so `../../` will take you two layers up.\n",
     ">\n",
     ">- If you were to type `!ls ./nextWeek/` it would return the contents of the `nextWeek` directory which is one layer down from the current directory, so it would return `manyThings.txt`.\n",
-    ">"
+    ">\n",
+    ">**This means that in the second line of the code cell above, the file `TransPi.nf` will be copied from the Google Cloud Storage bucket to the current directory.**"
    ]
   },
   {
@@ -338,7 +377,7 @@
    "outputs": [],
    "source": [
     "from jupyterquiz import display_quiz\n",
-    "display_quiz(\"Transcriptome-Assembly-Refinement-and-Applications/quiz-material/01-cp1.json\", shuffle_questions = True)"
+    "display_quiz(\"../quiz-material/01-cp1.json\", shuffle_questions = True)"
    ]
   },
   {

AWS/Submodule_02_basic_assembly.ipynb

@@ -180,8 +180,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!NXF_VER=22.10.1 ./nextflow run ./TransPi/TransPi.nf \\\n",
-    "-profile docker --k 17,25,43 --maxReadLen 50 --all "
+    "!nextflow run main.nf --input test_samplesheet.csv -profile test,docker -c conf/test_params.config --run_mode full "
    ]
   },
   {
@@ -345,7 +344,11 @@
    ]
   }
  ],
- "metadata": {},
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
  "nbformat": 4,
  "nbformat_minor": 5
 }

AWS/Submodule_03_annotation_only.ipynb

@@ -108,7 +108,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!pwd"
+    "! pwd"
    ]
   },
   {
@@ -141,7 +141,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!grep -c \">\" ./resources/trans/Oncorhynchus_mykiss_GGBN01.1.fa"
+    "! grep -c \">\" ./resources/trans/Oncorhynchus_mykiss_GGBN01.1.fa"
    ]
   },
   {
@@ -176,7 +176,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!perl -i.allloc -pe 's/basicRun/onlyAnnRun/g' ./TransPi/nextflow.config"
+    "! perl -i.allloc -pe 's/basicRun/onlyAnnRun/g' ./TransPi/nextflow.config"
    ]
   },
   {
@@ -229,7 +229,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!NXF_VER=22.10.1 ./nextflow run ./TransPi/TransPi.nf \\\n",
+    "! NXF_VER=22.10.1 ./nextflow run ./TransPi/TransPi.nf \\\n",
     "-profile docker --onlyAnn "
    ]
   },
@@ -248,7 +248,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!ls -l ./onlyAnnRun/output"
+    "! ls -l ./onlyAnnRun/output"
    ]
   },
   {
@@ -266,7 +266,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!cat ./onlyAnnRun/output/RUN_INFO.txt"
+    "! cat ./onlyAnnRun/output/RUN_INFO.txt"
    ]
   },
   {
@@ -297,7 +297,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!docker images"
+    "! docker images"
    ]
   },
   {
@@ -321,7 +321,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!docker run -it --rm --volume /home:/home quay.io/biocontainers/busco:5.4.3--pyhdfd78af_0 busco --help"
+    "! docker run -it --rm --volume /home:/home quay.io/biocontainers/busco:5.4.3--pyhdfd78af_0 busco --help"
    ]
   },
   {
@@ -365,7 +365,7 @@
    "source": [
     "numthreads=!lscpu | grep '^CPU(s)'| awk '{print $2-1}'\n",
     "THREADS = int(numthreads[0])\n",
-    "!echo $THREADS"
+    "! echo $THREADS"
    ]
   },
   {
@@ -375,7 +375,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!docker run -it --rm --volume /home:/home quay.io/biocontainers/busco:5.4.3--pyhdfd78af_0 busco \\\n",
+    "! docker run -it --rm --volume /home:/home quay.io/biocontainers/busco:5.4.3--pyhdfd78af_0 busco \\\n",
     "-i /home/jupyter/resources/trans/Oncorhynchus_mykiss_GGBN01.1.fa \\\n",
     "-l vertebrata_odb10 -o GGBN01_busco_vertebrata \\\n",
     "--out_path /home/jupyter/buscoOutput -m tran -c $THREADS"
@@ -425,7 +425,7 @@
    "outputs": [],
    "source": [
     "from jupytercards import display_flashcards\n",
-    "display_flashcards('Transcriptome-Assembly-Refinement-and-Applications/quiz-material/03-cp1-1.json')"
+    "display_flashcards('../quiz-material/03-cp1-1.json')"
    ]
   },
   {

AWS/Submodule_04_google_batch_assembly.ipynb

@@ -193,12 +193,31 @@
     "myBucketName"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "73e9985d",
+   "metadata": {},
+   "source": [
+    "**Step 4b:** Replace names in files with personal bucket and project names. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "860238b5",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "! sed -i \"s/<YOUR-PROJECT-ID>/$myProject/g\" nextflow.config\n",
+    "! sed -i \"s/<YOUR-BUCKET-NAME>/$myBucketName/g\" conf/test_params.config"
+   ]
+  },
   {
    "cell_type": "markdown",
    "id": "5676e5d2-f556-463e-93c2-5add30f1fff8",
    "metadata": {},
    "source": [
-    "**Step 4b:** Create a new GCS bucket. *If you are using an existing bucket, you can skip this step.*\n",
+    "**Step 4c:** Create a new GCS bucket. *If you are using an existing bucket, you can skip this step.*\n",
     "> To do this, we can use a new `gsutil` command: `mb` which stands for make bucket."
    ]
   },
@@ -397,8 +416,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "!NXF_VER=22.10.1 ./nextflow run ./TransPi/TransPi.nf \\\n",
-    "-profile gcb --k 17,25,43 --maxReadLen 50 --all -resume"
+    "!nextflow run main.nf --input test_samplesheet.csv -profile test,docker,gbatch -c conf/test_params.config --run_mode full "
    ]
   },
   {
@@ -598,7 +616,11 @@
    ]
   }
  ],
- "metadata": {},
+ "metadata": {
+  "language_info": {
+   "name": "python"
+  }
+ },
  "nbformat": 4,
  "nbformat_minor": 5
 }

AWS/denovotranscript/CHANGELOG.md

@@ -0,0 +1,46 @@
+# nf-core/denovotranscript: Changelog
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## v1.2.0 - [2025-01-30]
+
+- [PR #25](https://github.com/nf-core/denovotranscript/pull/25) - Bump version to 1.2.0dev
+- [PR #27](https://github.com/nf-core/denovotranscript/pull/27) - Update spades resources and usage instructions
+- [PR #28](https://github.com/nf-core/denovotranscript/pull/28) - Template update for nf-core/tools v3.1.0
+- [PR #29](https://github.com/nf-core/denovotranscript/pull/29) - Template update for nf-core/tools v3.1.1
+- [PR #30](https://github.com/nf-core/denovotranscript/pull/30) - Fix ch_transcripts and ch_pubids for AWS
+- [PR #32](https://github.com/nf-core/denovotranscript/pull/32) - Fix transrate dependencies issue
+- [PR #33](https://github.com/nf-core/denovotranscript/pull/33) - Bump version to 1.2.0
+- [PR #36](https://github.com/nf-core/denovotranscript/pull/36) - Template update for nf-core/tools v3.2.0
+
+## v1.1.0 - [2024-11-28]
+
+Enhancements & fixes:
+
+- [PR #14](https://github.com/nf-core/denovotranscript/pull/14) - Template update for nf-core/tools v3.0.2
+- [PR #15](https://github.com/nf-core/denovotranscript/pull/15) - Fix logic for skip_assembly parameter
+- [PR #16](https://github.com/nf-core/denovotranscript/pull/16) - Increase resources for Trinity
+- [PR #19](https://github.com/nf-core/denovotranscript/pull/21) - Fix Salmon quantification when running in skip_assembly mode
+- [PR #24](https://github.com/nf-core/denovotranscript/pull/24) - Update Trinity module and fix dependency issue
+
+### Software dependencies
+
+| Dependency | Old version | New version |
+| ---------- | ----------- | ----------- |
+| `Trinity`  | 2.15.1      | 2.15.2      |
+
+## v1.0.0 - [2024-08-15]
+
+Initial release of nf-core/denovotranscript, created with the [nf-core](https://nf-co.re/) template.
+
+This pipeline was created using code from [avani-bhojwani/TransFuse](https://github.com/avani-bhojwani/TransFuse) and [timslittle/nf-core-transcriptcorral](https://github.com/timslittle/nf-core-transcriptcorral/). It also incorporates modules and subworkflows developed by the nf-core community.
+
+The main features of this pipeline are:
+
+- Pre-processing of paired-end RNA-seq reads with FastQC, fastp, and SortMeRNA
+- _De novo_ transcriptome assembly with Trinity and rnaSPAdes
+- Redundancy reduction with EvidentialGene
+- Quality assessment of the assembly with BUSCO, rnaQUAST, and TransRate
+- Transcript-level quantification with Salmon
+- Reporting of the results with MultiQC