diff --git a/CHANGELOG.md b/CHANGELOG.md index 01faa82..cfe31ae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,7 @@ Initial release of nf-core/datasync, created with the [nf-core](https://nf-co.re - [[#46](https://github.com/nf-core/datasync/pull/46)] - Import Rclone module from nf-core([@antoniasaracco](https://github.com/antoniasaracco), review by [@delfiterradas](https://github.com/delfiterradas)). - [[#41](https://github.com/nf-core/datasync/pull/41)] - Generate MultiQC Report with comparechecksum tables and input samplesheet ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)). - [[#49](https://github.com/nf-core/datasync/pull/49)] - Install `RCLONE_CHECK` and `RCLONE_CHECKSUM` modules from nf-core ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)). +- [[#59](https://github.com/nf-core/datasync/pull/59)] - Create pipeline documentation ([@antoniasaracco](https://github.com/antoniasaracco), review by [@atrigila](https://github.com/atrigila) and [@delfiterradas](https://github.com/delfiterradas)). ### `Fixed` diff --git a/README.md b/README.md index 1c76aa4..ce077c1 100644 --- a/README.md +++ b/README.md @@ -21,52 +21,65 @@ ## Introduction -[WIP] WORK IN PROGRESS AND NOT YET STABLE - DO NOT USE FOR PRODUCTIVE SETTINGS YET +> [!WARNING] +> nf-core/datasync is under active development and is not yet recommended for production data transfers. Validate its behaviour with representative data before relying on it. + +**nf-core/datasync** is a Nextflow pipeline for copying files and directories between storage locations and documenting their integrity. For every row in an input samplesheet, the pipeline: + +1. validates the source against a supplied MD5 and/or SHA-256 checksum manifest using [`rclone checksum`](https://rclone.org/commands/rclone_checksum/); +2. copies the source to the requested destination with [`rclone copy`](https://rclone.org/); +3. compares the copied data with the source using [`rclone check`](https://rclone.org/commands/rclone_check/); and +4. produces detailed `rclone` status files and a consolidated MultiQC report. + +Sources and destinations may be local paths, HTTP(S) URLs, or rclone-supported remote storage such as Amazon S3, S3-compatible object storage, or Azure Blob Storage. -**nf-core/datasync** is a system operation pipeline that provides several workflows for handling system operation / automation tasks that are commonly helpful for various tasks in large data processing / analysis facilities. This includes: +The current tested use case for this pipeline is transfer between S3 buckets. -- Data Synchronization & Checksum generation - - Configurable: Can provide YAML file which files to include or exclude from sync - - Checksum backend: Can configure which backend to use for checksum generation (e.g. sha256sum, md5, ...) - - Configurable whether to include (sub-) folders in the sync or not (search for checkpoint files, e.g. has to have DEMUX_DONE that signals a demultiplexing run was finished & successfully copied) -- Data Integrity validation - - Provided with a directory to check, can validate that file(s) found are matching checksums from Synchronization subworkflow -- Data Archival & Deletion - - Can check source and target location for existence of file(s) and decide based on user configurable rules whether files can be considered archived - - Timestamp older than X days - - Checksums match Integrity validation report - - Create empty files to make it obvious that archival was performed - - Optionally: Delete files or create list of files to be deleted for manual deletion process +Pass an `rclone` configuration with `--rclone_config` whenever a source or destination needs a configured remote, endpoint, or credentials. A single configuration file can contain separate named remotes for multiple providers; for non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/). -The pipeline can be configured by users to execute any of the aforementioned subworkflows and then produces a report using MultiQC custom content that also serves as a report of _what_ was done by the pipeline for documentation purposes. +![nf-core/datasync metro map](docs/images/datasync-metromap.png) -## Usage +## Quick start > [!NOTE] -> If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/get_started/environment_setup/overview) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/get_started/run-your-first-pipeline) with `-profile test` before running the workflow on actual data. +> If you are new to Nextflow and nf-core, see the [nf-core environment setup guide](https://nf-co.re/docs/get_started/environment_setup/overview). Nextflow 25.10.4 or later is required. -Now, you can run the pipeline using: +To explore the pipeline outputs before preparing your own data, run the bundled `test` profile with a container profile: ```bash nextflow run nf-core/datasync \ - -profile \ - --input samplesheet.csv \ - --outdir - --sync - --sync_backend 'sha256' - --sync_done true #Creates SYNC_DONE file when done in each folder + -profile test,docker \ + --outdir results ``` -> [!WARNING] -> Please provide pipeline parameters via the CLI or Nextflow `-params-file` option. Custom config files including those provided by the `-c` Nextflow option can be used to provide any configuration _**except for parameters**_; see [docs](https://nf-co.re/docs/running/run-pipelines#using-parameter-files). +The `test` profile supplies a small samplesheet and `rclone` configuration automatically. It also enables `--rclone_dry_run`, so no files are actually transferred. This makes it useful for exploring the `rclone/` output folders and `multiqc/multiqc_report.html`; remember that post-copy comparison reports describe whatever is already present at the destination because the dry run does not write transfer data. + +To run the pipeline on your own data, create a samplesheet containing one transfer per row: + +```csv +sample,input,output_path,checksum_md5,checksum_sha +run_001,/data/run_001,s3://archive/runs,/data/manifests/run_001_md5.tsv +reference,https://example.org/reference.fa,/data/references,,/data/manifests/reference_sha256.tsv +``` + +Then launch the pipeline using: + +```bash +nextflow run nf-core/datasync \ + -r \ + -profile docker \ + --input samplesheet.csv \ + --outdir results \ + --rclone_config /path/to/rclone.conf +``` + +`--rclone_config` is optional only when every source and destination is accessible without a configured rclone remote. See the [`rclone` configuration section](docs/usage.md#configuring-rclone-remotes) for the tested S3-to-S3 use case and guidance on adapting rclone configuration files for other providers. To preview copy operations without transferring data, add `--rclone_dry_run`; note that subsequent comparison reports will then describe the unchanged destination. -For more details and further functionality, please refer to the [usage documentation](https://nf-co.re/datasync/usage) and the [parameter documentation](https://nf-co.re/datasync/parameters). +See the [usage documentation](docs/usage.md) for samplesheet rules, destination semantics, remote configuration, and reproducible execution. The complete generated parameter reference is available on the [nf-core pipeline page](https://nf-co.re/datasync/parameters). ## Pipeline output -To see the results of an example test run with a full size dataset refer to the [results](https://nf-co.re/datasync/results) tab on the nf-core website pipeline page. -For more details about the output files and reports, please refer to the -[output documentation](https://nf-co.re/datasync/output). +Results are written below `--outdir`. See the [output documentation](docs/output.md) for file names and status-code interpretation. ## Credits diff --git a/docs/images/datasync-metromap.png b/docs/images/datasync-metromap.png new file mode 100644 index 0000000..108bbc0 Binary files /dev/null and b/docs/images/datasync-metromap.png differ diff --git a/docs/images/datasync-multiqc-checksum-md5.png b/docs/images/datasync-multiqc-checksum-md5.png new file mode 100644 index 0000000..42968b7 Binary files /dev/null and b/docs/images/datasync-multiqc-checksum-md5.png differ diff --git a/docs/images/datasync-multiqc-post-transfer-check.png b/docs/images/datasync-multiqc-post-transfer-check.png new file mode 100644 index 0000000..22014f3 Binary files /dev/null and b/docs/images/datasync-multiqc-post-transfer-check.png differ diff --git a/docs/output.md b/docs/output.md index 1b7eb8b..548f2e0 100644 --- a/docs/output.md +++ b/docs/output.md @@ -2,60 +2,136 @@ ## Introduction -This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline. +This document describes the reports produced by nf-core/datasync. Paths below are relative to the directory supplied with `--outdir`. + +> [!IMPORTANT] +> The copied payload is written to each samplesheet row's `output_path`. It is not placed in `--outdir` unless `output_path` explicitly points there. + +## Output overview + +```text +/ +├── rclone/ +│ ├── copy/ +│ │ └── -rclone-copy.log +│ ├── checksum/ +│ │ └── / +│ │ ├── .combined.txt +│ │ ├── .match.txt +│ │ ├── .differ.txt +│ │ ├── .missing_on_dst.txt +│ │ ├── .missing_on_src.txt +│ │ └── .error.txt +│ └── check/ +│ └── / +│ ├── .combined.txt +│ ├── .match.txt +│ ├── .differ.txt +│ ├── .missing_on_dst.txt +│ ├── .missing_on_src.txt +│ └── .error.txt +├── multiqc/ +│ ├── multiqc_report.html +│ └── multiqc_data/ +└── pipeline_info/ + ├── nf_core_datasync_software_mqc_versions.yml + └── execution_* / pipeline_dag_* +``` + +The `rclone/` directory is split by module stage. Copy logs are published to `rclone/copy/`, pre-copy checksum validation reports are published to `rclone/checksum//`, and post-copy source-to-destination comparison reports are published to `rclone/check//`. The `` directory name is taken from the `sample` value in the samplesheet row. + +## `rclone` directory -The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory. - - +
+Output files -## Pipeline overview +- `rclone/copy/` + - `-rclone-copy.log`: informational log from the copy operation. +- `rclone/checksum//` + - `.combined.txt`: combined pre-copy checksum-validation status, one path per line. + - `.match.txt`: paths whose content matched the supplied checksum manifest (`=`). + - `.differ.txt`: paths present in the source and manifest but with different content (`*`). + - `.missing_on_dst.txt`: paths present in the checksum manifest but absent from the checked source (`-`). + - `.missing_on_src.txt`: paths present in the checked source but absent from the checksum manifest (`+`). + - `.error.txt`: paths that could not be read or hashed (`!`). +- `rclone/check//` + - `.combined.txt`: combined post-copy source-to-destination comparison status, one path per line. + - `.match.txt`: paths whose content matched between source and destination (`=`). + - `.differ.txt`: paths present on both sides but with different content (`*`). + - `.missing_on_dst.txt`: paths found in the source but absent from the destination (`-`). + - `.missing_on_src.txt`: paths found at the destination but absent from the source (`+`). + - `.error.txt`: paths that could not be read or hashed (`!`). -The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps: +
-- [FastQC](#fastqc) - Raw read QC -- [MultiQC](#multiqc) - Aggregate report describing results and QC from the whole pipeline -- [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution +Two integrity stages create reports: -### FastQC +1. **Pre-copy checksum validation** uses each supplied MD5 and/or SHA-256 manifest to check the source and publishes reports under `rclone/checksum//`. +2. **Post-copy validation** compares the source with the destination after the copy task finishes and publishes reports under `rclone/check//`. -
-Output files +Both stages use the same `.*.txt` naming convention and publish to `rclone/`. When a row supplies a checksum manifest, similarly named pre-copy and post-copy files may target the same published path; use the consolidated MultiQC sections for the stage-specific summary and retain the Nextflow work directory if both raw report sets must be audited independently. -- `fastqc/` - - `*_fastqc.html`: FastQC report containing quality metrics. - - `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images. +The combined files use `rclone`'s one-character status prefixes: -
+| Prefix | Meaning | Action | +| ------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- | +| `=` | File matches | No action required. | +| `-` | Missing from destination | Investigate an incomplete source checksum set or transfer. | +| `+` | Missing from source | Review unexpected destination content. The post-copy check uses `--one-way`, so destination-only files are tolerated there. | +| `*` | Content differs | Re-copy or investigate source/destination mutation. | +| `!` | Read/hash error | Inspect permissions, credentials, connectivity, and the copy log. | -[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. For further reading and documentation see the [FastQC help pages](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/). +Empty category files mean that `rclone` reported no entries in that category. The commands are designed to preserve these reports rather than terminate the whole workflow on comparison differences. Always inspect the reports; workflow success alone is not an integrity guarantee. -### MultiQC +## MultiQC
Output files - `multiqc/` - - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser. - - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline. - - `multiqc_plots/`: directory containing static images from the report in various formats. + - `multiqc_report.html`: standalone report viewable in a browser. + - `multiqc_data/`: machine-readable data, logs, source inventory, software versions, and parsed rclone tables.
-[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory. +The MultiQC report consolidates: + +- checksum validation status for MD5 and/or SHA-256 manifests generated from `rclone checksum`; +- post-copy source-to-destination validation status generated from `rclone check`; +- the validated samplesheet and workflow parameter summary; and +- pipeline and tool versions. -Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see . +### `rclone checksum` section (source integrity checks) -### Pipeline information +The MD5 and SHA-256 input-validation sections show the results from [`rclone checksum`](https://rclone.org/commands/rclone_checksum/). Use these sections to confirm that each source file matched the checksum manifest supplied in `checksum_md5` and/or `checksum_sha` before copying. + +When a samplesheet `input` points to S3, `rclone checksum` can normally validate MD5 manifests from S3 object hashes, but S3 does not provide SHA-256 object hashes for rclone to read remotely. SHA-256 validation for S3 inputs therefore requires `rclone checksum --download`, which downloads object data and calculates the hash locally during validation. Configure the checksum process to pass `--download` when SHA-256 validation is required for S3 inputs; see the [`rclone checksum` --download documentation](https://rclone.org/commands/rclone_checksum/). + +![nf-core/multiqc checksum md5](images/datasync-multiqc-checksum-md5.png) + +### `rclone check` section (post-transfer checks) + +The source-destination validation section shows the results from `rclone check` after the copy step. Use this section to confirm that copied files at `output_path` match the corresponding source files. + +![nf-core/multiqc checksum md5](images/datasync-multiqc-post-transfer-check.png) + +Open `multiqc_report.html` after every run and investigate any non-matching, missing, or error entries. In the `rclone` result sections, entries are prioritised so rows needing attention are shown before successful matches: errors, mismatches, and missing-file statuses appear ahead of matching files. Data under `multiqc_data/` can be retained for automated auditing or downstream reporting; exact filenames may vary with the MultiQC version and the checksum types present in the samplesheet. + +## Pipeline information
Output files - `pipeline_info/` - - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`. - - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline. - - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`. - - Parameters used by the pipeline run: `params.json`. + - `nf_core_datasync_software_mqc_versions.yml`: versions of the pipeline and tools collected for MultiQC. + - `execution_timeline_.html`: chronological task execution view. + - `execution_report_.html`: task runtime and resource report. + - `execution_trace_.txt`: tabular task-level execution trace. + - `pipeline_dag_.html`: workflow dependency graph. + - Completion reports generated when `--email` or `--email_on_fail` is configured may also be present.
-[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage. +These files provide operational provenance and help diagnose performance or failures. Archive them with the MultiQC and rclone reports. The Nextflow `work/` directory and `.nextflow.log` remain in the launch directory rather than `--outdir`; keep them until transfer verification is complete if detailed troubleshooting or `-resume` may be needed. + +The supplied rclone configuration is an input credential file and is not intentionally copied to `--outdir`. Nevertheless, execution logs may contain remote names and object paths. Review logs before sharing them, and manage `rclone.conf` separately as a secret. diff --git a/docs/usage.md b/docs/usage.md index 1c85a3e..2e7c6fb 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -2,65 +2,157 @@ ## :warning: Please read this documentation on the nf-core website: [https://nf-co.re/datasync/usage](https://nf-co.re/datasync/usage) -> _Documentation of pipeline parameters is generated automatically from the pipeline schema and can no longer be found in markdown files._ +> Pipeline parameter documentation is generated automatically from [`nextflow_schema.json`](../nextflow_schema.json). This page explains how to prepare a transfer and operate the pipeline. -## Introduction +## Prerequisites - +Install Nextflow 25.10.4 or later and use a supported software profile. Docker or Singularity/Apptainer is recommended for reproducibility. Ensure that the account running Nextflow can read each source and checksum manifest and can write to every destination. + +For cloud or other authenticated rclone remotes, create an [rclone configuration](https://rclone.org/docs/) and pass it with `--rclone_config`. The configuration applies to remote **sources and destinations**. The pipeline has currently been tested for transfers between S3 buckets. Other rclone-supported layouts, such as Azure Blob Storage to S3 or transfers between S3-compatible providers, should be configured and validated against the upstream `rclone` documentation for each provider before use. ## Samplesheet input -You will need to create a samplesheet with information about the samples you would like to analyse before running the pipeline. Use this parameter to specify its location. It has to be a comma-separated file with 3 columns, and a header row as shown in the examples below. +Supply a comma-separated samplesheet with `--input`: ```bash ---input '[path to samplesheet file]' +--input /path/to/samplesheet.csv ``` -### Multiple runs of the same sample +Each row describes an independent transfer. The header names are fixed; columns may be in any order. + +| Column | Required | Description | +| -------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sample` | Yes | Unique identifier used in task labels and output report names. It must not contain whitespace. Use a unique value for each row to prevent published report files from colliding. | +| `input` | Yes | Source file or directory. This can be a local path, HTTP(S) URL, object-storage URL such as `s3://bucket/prefix`, or configured remote such as `source_s3:bucket/prefix` or `source_azure:container/prefix`. Whitespace is not allowed. | +| `output_path` | Yes | Destination directory understood by rclone, such as `/archive/runs`, `s3://bucket/prefix`, or a configured `remote:path`. Whitespace is not allowed. | +| `checksum_md5` | One checksum column | Path or URL to an MD5 checksum manifest used to validate `input` before copying. The manifest format is described below. Leave empty when using SHA-256 only. | +| `checksum_sha` | One checksum column | Path or URL to a SHA-256 checksum manifest used to validate `input` before copying. The manifest format is described below. Leave empty when using MD5 only. | + +At least one checksum manifest is required on every row. If both are supplied, both validations run. Checksum files must use the format accepted by [`rclone checksum`](https://rclone.org/commands/rclone_checksum/): one checksum record per line with the hash value followed by two spaces and then the file path. Paths must be relative to the source root from the `input` column, not absolute paths. + +For a directory input, the source root is the directory named in the samplesheet. For example, if the samplesheet `input` is `/data/run_001` and one file in that directory is `/data/run_001/reads/sample_R1.fastq.gz`, the checksum manifest path must be `reads/sample_R1.fastq.gz`. Do not write `/data/run_001/reads/sample_R1.fastq.gz` in the manifest. For a single-file input, use the input file name as the manifest path. -The `sample` identifiers have to be the same when you have re-sequenced the same sample more than once e.g. to increase sequencing depth. The pipeline will concatenate the raw reads before performing any downstream analysis. Below is an example for the same sample sequenced across 3 lanes: +Checksum manifests may use a `.tsv` or `.csv` filename extension, but their contents are not tab-separated or comma-separated tables and must not include a header. Each record is plain text with the hash and path separated by exactly two spaces. The required fields are: + +| Field | Required | Description | +| ----- | -------- | ----------------------------------------------------------------------------------------------- | +| Hash | Yes | MD5 hash for `checksum_md5` files or SHA-256 hash for `checksum_sha` files. | +| Path | Yes | Relative path to the file being validated, resolved from the corresponding `input` source root. | + +Example samplesheet: ```csv title="samplesheet.csv" -sample,fastq_1,fastq_2 -CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz -CONTROL_REP1,AEG588A1_S1_L003_R1_001.fastq.gz,AEG588A1_S1_L003_R2_001.fastq.gz -CONTROL_REP1,AEG588A1_S1_L004_R1_001.fastq.gz,AEG588A1_S1_L004_R2_001.fastq.gz +sample,input,output_path,checksum_md5,checksum_sha +run_001,/data/run_001,s3://archive/runs,/data/checksums/run_001_md5.tsv, +reference,https://example.org/reference.fa,/data/references,,/data/checksums/reference_sha256.tsv +run_002,/data/run_002,archive:runs,/data/checksums/run_002_md5.tsv,/data/checksums/run_002_sha256.tsv ``` -### Full samplesheet +For the `run_001` directory example, `/data/checksums/run_001_md5.tsv` could contain: -The pipeline will auto-detect whether a sample is single- or paired-end using the information provided in the samplesheet. The samplesheet can have as many columns as you desire, however, there is a strict requirement for the first 3 columns to match those defined in the table below. +```text title="run_001_md5.tsv" +d41d8cd98f00b204e9800998ecf8427e reads/sample_R1.fastq.gz +0cc175b9c0f1b6a831c399e269772661 reads/sample_R2.fastq.gz +900150983cd24fb0d6963f7d28e17f72 reports/qc_summary.txt +``` -A final samplesheet file consisting of both single- and paired-end data may look something like the one below. This is for 6 samples, where `TREATMENT_REP3` has been sequenced twice. +For a SHA-256 manifest, the same relative paths are used with SHA-256 hashes: -```csv title="samplesheet.csv" -sample,fastq_1,fastq_2 -CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz -CONTROL_REP2,AEG588A2_S2_L002_R1_001.fastq.gz,AEG588A2_S2_L002_R2_001.fastq.gz -CONTROL_REP3,AEG588A3_S3_L002_R1_001.fastq.gz,AEG588A3_S3_L002_R2_001.fastq.gz -TREATMENT_REP1,AEG588A4_S4_L003_R1_001.fastq.gz, -TREATMENT_REP2,AEG588A5_S5_L003_R1_001.fastq.gz, -TREATMENT_REP3,AEG588A6_S6_L003_R1_001.fastq.gz, -TREATMENT_REP3,AEG588A6_S6_L004_R1_001.fastq.gz, +```text title="run_001_sha256.tsv" +e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 reads/sample_R1.fastq.gz +ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb reads/sample_R2.fastq.gz +ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad reports/qc_summary.txt +``` + +An [example samplesheet](../assets/samplesheet.csv) is included in the repository. + +## Configuring `rclone` remotes + +The file supplied with `--rclone_config` uses `rclone`'s INI-style format. + +Each `[name]` section defines a remote, and samplesheet paths refer to it as `name:path`. The remote name is an arbitrary local label; it does not need to match the provider or bucket name. + +> [!NOTE] +> The pipeline's documented and tested configuration pattern is S3-to-S3 transfer. + +One file may contain several sections, so other cloud-to-cloud transfers can define both providers in the same file, but provider-specific options should be taken from the relevant `rclone` documentation. This is an example using S3: + +```text +source_s3:incoming/run_001 +archive_azure:research-archive/run_001 +``` + +Create the file interactively where possible: + +```bash +rclone config --config /secure/rclone.conf +rclone listremotes --config /secure/rclone.conf +``` + +Then provide that exact file to the pipeline: + +```bash +nextflow run nf-core/datasync \ + -profile docker \ + --input samplesheet.csv \ + --outdir results \ + --rclone_config /secure/rclone.conf +``` + +### S3 and S3-compatible storage + +The main use case tested for nf-core/datasync is transferring files between S3 buckets. An S3 remote specifies the provider, region, and credentials. For example: + +```ini title="rclone.conf" +[s3] +type = s3 +provider = AWS +access_key_id = YOUR_ACCESS_KEY_ID +secret_access_key = YOUR_SECRET_ACCESS_KEY +region = eu-central-1 ``` -| Column | Description | -| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `sample` | Custom sample name. This entry will be identical for multiple sequencing libraries/runs from the same sample. Spaces in sample names are automatically converted to underscores (`_`). | -| `fastq_1` | Full path to FastQ file for Illumina short reads 1. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | -| `fastq_2` | Full path to FastQ file for Illumina short reads 2. File has to be gzipped and have the extension ".fastq.gz" or ".fq.gz". | +The corresponding input values could be `source_s3:incoming/run_001` and `institutional_s3:project/run_002`. Provider-specific settings vary: consult the [`rclone` S3 documentation](https://rclone.org/s3/) and your storage provider's endpoint, region, addressing-style, and credential documentation rather than copying example values unchanged. -An [example samplesheet](../assets/samplesheet.csv) has been provided with the pipeline. +The pipeline also accepts an `s3://bucket/path` source or destination. In that form, ensure credentials and provider settings are available to both Nextflow and `rclone` in the execution environment. A named remote such as `source_s3:bucket/path` makes the selected configuration section explicit and is preferable when a config file contains multiple S3 providers. + +## Destination layout + +The pipeline preserves the source basename: + +- for a file source, `rclone` copies the file into `output_path`, and validation expects `output_path/`; +- for a directory source, the pipeline appends the source directory name, so `/data/run_001` with `output_path=/archive/runs` is copied and checked at `/archive/runs/run_001`. + +A trailing slash on `output_path` is removed before these paths are constructed. Ensure that a destination does not already contain unrelated files: post-copy validation uses `rclone check --one-way`, which checks that source content exists and matches at the destination while tolerating destination-only files. You can override this behavior by providing your own config file with external arguments for `rclone check`. ## Running the pipeline -The typical command for running the pipeline is as follows: +A typical local-to-cloud run is: + +```bash +nextflow run nf-core/datasync \ + -r \ + -profile docker \ + --input /data/samplesheet.csv \ + --outdir /data/datasync-results \ + --rclone_config /secure/rclone.conf +``` + +`--outdir` stores logs, integrity reports, MultiQC, and execution metadata. It does **not** override the transfer destinations in the samplesheet. + +To inspect the proposed copy without writing destination data: ```bash -nextflow run nf-core/datasync --input ./samplesheet.csv --outdir ./results --genome GRCh37 -profile docker +nextflow run nf-core/datasync \ + -r \ + -profile docker \ + --input /data/samplesheet.csv \ + --outdir /data/datasync-dry-run \ + --rclone_config /secure/rclone.conf \ + --rclone_dry_run ``` -This will launch the pipeline with the `docker` configuration profile. See below for more information about profiles. +The checksum and post-copy check stages still run during a dry run. Consequently, post-copy results reflect whatever was already present at the destination rather than a simulated final state. Note that the pipeline will create the following files in your working directory: @@ -95,14 +187,86 @@ genome: 'GRCh37' You can also generate such `YAML`/`JSON` files via [nf-core/launch](https://nf-co.re/launch). -### Updating the pipeline +### Parameter files -When you run the above command, Nextflow automatically pulls the pipeline code from GitHub and stores it as a cached version. When running the pipeline after this, it will always use the cached version if available - even if the pipeline has been updated since. To make sure that you're running the latest version of the pipeline, make sure that you regularly update the cached version of the pipeline: +Frequently reused settings can be stored in YAML or JSON and loaded with `-params-file`: + +```yaml title="params.yaml" +input: /data/samplesheet.csv +outdir: /data/datasync-results +rclone_config: /secure/rclone.conf +multiqc_title: July archive transfer +``` ```bash -nextflow pull nf-core/datasync +nextflow run nf-core/datasync -r -profile docker -params-file params.yaml +``` + +Do not use `-c` for pipeline parameters. Use it only for Nextflow executor, resources, and other infrastructure configuration. + +### Common integrity outcomes + +The workflow is designed to collect `rclone` reports even when `rclone` detects differences. The table below summarises common edge cases and how to interpret them in the published reports and MultiQC. + +| Situation | Where it is detected | Report status | Pipeline behaviour and action | +| ------------------------------------------------------------------ | ---------------------------------------------------------------- | --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| File exists and checksum/content matches | `rclone checksum` before copy and `rclone check` after copy | `=` / `Match` | Expected result; no action needed. | +| File is listed in the checksum manifest but absent from the source | Pre-copy `rclone checksum` | `-` / missing from checked source | The report is retained for review. Fix the manifest or restore the missing source file before relying on the transfer. | +| Source file exists but is absent from the checksum manifest | Pre-copy `rclone checksum` | `+` / missing from manifest | Review whether the manifest is incomplete or whether the extra source file should be excluded from the transfer. | +| Source file hash differs from the supplied manifest | Pre-copy `rclone checksum` | `*` / mismatch | Investigate source mutation, stale manifests, or incorrect checksum files before accepting the copy. | +| Source file cannot be read or hashed | Pre-copy `rclone checksum` | `!` / error | Inspect credentials, permissions, connectivity, and source path spelling. | +| Destination is missing a copied file | Post-copy `rclone check` | `-` / missing from destination | Treat as an incomplete transfer unless the file was intentionally excluded; re-run or inspect the rclone copy log. | +| Destination file exists but content differs from source | Post-copy `rclone check` | `*` / mismatch | Re-copy or investigate concurrent source/destination changes. | +| Destination contains files absent from the source | Post-copy `rclone check` | `+` / missing from source | The post-copy check uses `--one-way`, so destination-only files are tolerated, but should still be reviewed for unexpected stale or unrelated data. | +| Dry-run execution | Copy step uses `--dry-run`; checksum and check reports still run | Depends on existing destination | No transfer data is written. Post-copy reports describe whatever was already present at the destination. | + +### Including or excluding files + +Filter files by passing additional `rclone` filter flags to the relevant rclone module through a Nextflow configuration file. `rclone` supports flags such as `--include`, `--exclude`, `--filter`, `--files-from`, and related rule files; see the [`rclone` filtering documentation](https://rclone.org/filtering/) for rule syntax and ordering. + +For example, to copy and check only FASTQ files while excluding temporary files, create a small infrastructure config: + +```groovy title="rclone_filters.config" +process { + withName: 'RCLONE_COPY' { + ext.args = { + [ + '--log-level INFO', + '--stats 30s', + '--stats-one-line', + '--stats-log-level INFO', + '--s3-chunk-size 64M', + '--no-check-certificate', + params.rclone_dry_run ? '--dry-run' : '', + '--include "*.fastq.gz"', + '--include "*.fq.gz"', + '--exclude "*.tmp"', + '--exclude "*"' + ].findAll { it }.join(' ') + } + } + + withName: 'RCLONE_CHECK' { + ext.args = { + [ + '--no-check-certificate', + '--one-way', + '--include "*.fastq.gz"', + '--include "*.fq.gz"', + '--exclude "*.tmp"', + '--exclude "*"' + ].join(' ') + } + } +} ``` +Run it with `-c rclone_filters.config` in addition to your normal profile and parameters. Because `ext.args` overrides module defaults, include the default `rclone` flags you still need when adding filters. Keep checksum manifests consistent with the same filtering rules: if a file is intentionally excluded from copy/check, remove it from the checksum manifest or generate a manifest for only the included files. + +## Understanding completion and integrity + +For each row, the pipeline first validates supplied checksum manifests, performs the copy, and then compares source and destination. `rclone` comparison commands write status reports even when differences are found, allowing all results to be collected in MultiQC. Therefore, a successful Nextflow run means the workflow completed; it does **not by itself** prove every object matched. Review `multiqc/multiqc_report.html` and the reports under `rclone/`, especially lines marked `-`, `+`, `*`, or `!` (see [output documentation](output.md)). + ### Reproducibility It is a good idea to specify the pipeline version when running the pipeline on your data. This ensures that a specific version of the pipeline code and software are used when you run your pipeline. If you keep using the same tag, you'll be running the same version of the pipeline, even if there have been changes to the code since. @@ -171,8 +335,19 @@ Specify the path to a specific config file (this is a core Nextflow command). Se ### Resource requests -Whilst the default requirements set within the pipeline will hopefully work for most people and with most input data, you may find that you want to customise the compute resources that the pipeline requests. Each step in the pipeline has a default set of requirements for number of CPUs, memory and time. For most of the pipeline steps, if the job exits with any of the error codes specified [here](https://github.com/nf-core/rnaseq/blob/4c27ef5610c87db00c3c5a3eed10b1d161abf575/conf/base.config#L18) it will automatically be resubmitted with higher resources request (2 x original, then 3 x original). If it still fails after the third attempt then the pipeline execution is stopped. +The `rclone` processes use the `process_low` label. Configure executors and override CPU, memory, or time in a Nextflow config, for example: +```groovy title="resources.config" +process { + withLabel: process_low { + cpus = 8 + memory = '16 GB' + time = '12h' + } +} +``` + +Run with `-c resources.config`. `rclone` derives its checker count from allocated CPUs, and the copy step uses roughly half that count (minimum one) for parallel transfers. To change the resource requests, please see the [max resources](https://nf-co.re/docs/running/configuration/nextflow-for-your-system#set-max-resources) and [customise process resources](https://nf-co.re/docs/running/configuration/nextflow-for-your-system#customize-process-resources) section of the nf-core website. ### Custom Containers @@ -194,21 +369,3 @@ In most cases, you will only need to create a custom config as a one-off but if See the main [Nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more information about creating your own configuration files. If you have any questions or issues please send us a message on [Slack](https://nf-co.re/join/slack) on the [`#configs` channel](https://nfcore.slack.com/channels/configs). - -## Running in the background - -Nextflow handles job submissions and supervises the running jobs. The Nextflow process must run until the pipeline is finished. - -The Nextflow `-bg` flag launches Nextflow in the background, detached from your terminal so that the workflow does not stop if you log out of your session. The logs are saved to a file. - -Alternatively, you can use `screen` / `tmux` or similar tool to create a detached session which you can log back into at a later time. -Some HPC setups also allow you to run nextflow within a cluster job submitted your job scheduler (from where it submits more jobs). - -## Nextflow memory requirements - -In some cases, the Nextflow Java virtual machines can start to request a large amount of memory. -We recommend adding the following line to your environment to limit this (typically in `~/.bashrc` or `~./bash_profile`): - -```bash -NXF_OPTS='-Xms1g -Xmx4g' -```