Create documentation

CHANGELOG.md

@@ -17,6 +17,7 @@ Initial release of nf-core/datasync, created with the [nf-core](https://nf-co.re
 - [[#46](https://github.com/nf-core/datasync/pull/46)] - Import Rclone module from nf-core([@antoniasaracco](https://github.com/antoniasaracco), review by [@delfiterradas](https://github.com/delfiterradas)).
 - [[#41](https://github.com/nf-core/datasync/pull/41)] - Generate MultiQC Report with comparechecksum tables and input samplesheet ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila)).
 - [[#49](https://github.com/nf-core/datasync/pull/49)] - Install `RCLONE_CHECK` and `RCLONE_CHECKSUM` modules from nf-core ([@delfiterradas](https://github.com/delfiterradas), review by [@atrigila](https://github.com/atrigila) and [@antoniasaracco](https://github.com/antoniasaracco)).
+- [[#59](https://github.com/nf-core/datasync/pull/59)] - Create pipeline documentation ([@antoniasaracco](https://github.com/antoniasaracco), review by [@atrigila](https://github.com/atrigila) and [@delfiterradas](https://github.com/delfiterradas)).
 
 ### `Fixed`
 

README.md

@@ -21,52 +21,65 @@
 
 ## Introduction
 
-[WIP] WORK IN PROGRESS AND NOT YET STABLE - DO NOT USE FOR PRODUCTIVE SETTINGS YET
+> [!WARNING]
+> nf-core/datasync is under active development and is not yet recommended for production data transfers. Validate its behaviour with representative data before relying on it.
+
+**nf-core/datasync** is a Nextflow pipeline for copying files and directories between storage locations and documenting their integrity. For every row in an input samplesheet, the pipeline:
+
+1. validates the source against a supplied MD5 and/or SHA-256 checksum manifest using [`rclone checksum`](https://rclone.org/commands/rclone_checksum/);
+2. copies the source to the requested destination with [`rclone copy`](https://rclone.org/);
+3. compares the copied data with the source using [`rclone check`](https://rclone.org/commands/rclone_check/); and
+4. produces detailed `rclone` status files and a consolidated MultiQC report.
+
+Sources and destinations may be local paths, HTTP(S) URLs, or rclone-supported remote storage such as Amazon S3, S3-compatible object storage, or Azure Blob Storage.
 
-**nf-core/datasync** is a system operation pipeline that provides several workflows for handling system operation / automation tasks that are commonly helpful for various tasks in large data processing / analysis facilities. This includes:
+The current tested use case for this pipeline is transfer between S3 buckets.
 
-- Data Synchronization & Checksum generation
-  - Configurable: Can provide YAML file which files to include or exclude from sync
-  - Checksum backend: Can configure which backend to use for checksum generation (e.g. sha256sum, md5, ...)
-  - Configurable whether to include (sub-) folders in the sync or not (search for checkpoint files, e.g. has to have DEMUX_DONE that signals a demultiplexing run was finished & successfully copied)
-- Data Integrity validation
-  - Provided with a directory to check, can validate that file(s) found are matching checksums from Synchronization subworkflow
-- Data Archival & Deletion
-  - Can check source and target location for existence of file(s) and decide based on user configurable rules whether files can be considered archived
-    - Timestamp older than X days
-    - Checksums match Integrity validation report
-    - Create empty files to make it obvious that archival was performed
-    - Optionally: Delete files or create list of files to be deleted for manual deletion process
+Pass an `rclone` configuration with `--rclone_config` whenever a source or destination needs a configured remote, endpoint, or credentials. A single configuration file can contain separate named remotes for multiple providers; for non-S3 layouts, design and validate the provider-specific configuration using the upstream [rclone documentation](https://rclone.org/docs/).
 
-The pipeline can be configured by users to execute any of the aforementioned subworkflows and then produces a report using MultiQC custom content that also serves as a report of _what_ was done by the pipeline for documentation purposes.
+![nf-core/datasync metro map](docs/images/datasync-metromap.png)
 
-## Usage
+## Quick start
 
 > [!NOTE]
-> If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/get_started/environment_setup/overview) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/get_started/run-your-first-pipeline) with `-profile test` before running the workflow on actual data.
+> If you are new to Nextflow and nf-core, see the [nf-core environment setup guide](https://nf-co.re/docs/get_started/environment_setup/overview). Nextflow 25.10.4 or later is required.
 
-Now, you can run the pipeline using:
+To explore the pipeline outputs before preparing your own data, run the bundled `test` profile with a container profile:
 
 ```bash
 nextflow run nf-core/datasync \
-   -profile <docker/singularity/.../institute> \
-   --input samplesheet.csv \
-   --outdir <OUTDIR>
-   --sync
-   --sync_backend 'sha256'
-   --sync_done true #Creates SYNC_DONE file when done in each folder
+    -profile test,docker \
+    --outdir results
 ```
 
-> [!WARNING]
-> Please provide pipeline parameters via the CLI or Nextflow `-params-file` option. Custom config files including those provided by the `-c` Nextflow option can be used to provide any configuration _**except for parameters**_; see [docs](https://nf-co.re/docs/running/run-pipelines#using-parameter-files).
+The `test` profile supplies a small samplesheet and `rclone` configuration automatically. It also enables `--rclone_dry_run`, so no files are actually transferred. This makes it useful for exploring the `rclone/` output folders and `multiqc/multiqc_report.html`; remember that post-copy comparison reports describe whatever is already present at the destination because the dry run does not write transfer data.
+
+To run the pipeline on your own data, create a samplesheet containing one transfer per row:
+
+```csv
+sample,input,output_path,checksum_md5,checksum_sha
+run_001,/data/run_001,s3://archive/runs,/data/manifests/run_001_md5.tsv
+reference,https://example.org/reference.fa,/data/references,,/data/manifests/reference_sha256.tsv
+```
+
+Then launch the pipeline using:
+
+```bash
+nextflow run nf-core/datasync \
+    -r <VERSION> \
+    -profile docker \
+    --input samplesheet.csv \
+    --outdir results \
+    --rclone_config /path/to/rclone.conf
+```
+
+`--rclone_config` is optional only when every source and destination is accessible without a configured rclone remote. See the [`rclone` configuration section](docs/usage.md#configuring-rclone-remotes) for the tested S3-to-S3 use case and guidance on adapting rclone configuration files for other providers. To preview copy operations without transferring data, add `--rclone_dry_run`; note that subsequent comparison reports will then describe the unchanged destination.
 
-For more details and further functionality, please refer to the [usage documentation](https://nf-co.re/datasync/usage) and the [parameter documentation](https://nf-co.re/datasync/parameters).
+See the [usage documentation](docs/usage.md) for samplesheet rules, destination semantics, remote configuration, and reproducible execution. The complete generated parameter reference is available on the [nf-core pipeline page](https://nf-co.re/datasync/parameters).
 
 ## Pipeline output
 
-To see the results of an example test run with a full size dataset refer to the [results](https://nf-co.re/datasync/results) tab on the nf-core website pipeline page.
-For more details about the output files and reports, please refer to the
-[output documentation](https://nf-co.re/datasync/output).
+Results are written below `--outdir`. See the [output documentation](docs/output.md) for file names and status-code interpretation.
 
 ## Credits
 

docs/output.md

@@ -2,60 +2,136 @@
 
 ## Introduction
 
-This document describes the output produced by the pipeline. Most of the plots are taken from the MultiQC report, which summarises results at the end of the pipeline.
+This document describes the reports produced by nf-core/datasync. Paths below are relative to the directory supplied with `--outdir`.
+
+> [!IMPORTANT]
+> The copied payload is written to each samplesheet row's `output_path`. It is not placed in `--outdir` unless `output_path` explicitly points there.
+
+## Output overview
+
+```text
+<OUTDIR>/
+├── rclone/
+│   ├── copy/
+│   │   └── <sample>-rclone-copy.log
+│   ├── checksum/
+│   │   └── <sample>/
+│   │       ├── <sample>.combined.txt
+│   │       ├── <sample>.match.txt
+│   │       ├── <sample>.differ.txt
+│   │       ├── <sample>.missing_on_dst.txt
+│   │       ├── <sample>.missing_on_src.txt
+│   │       └── <sample>.error.txt
+│   └── check/
+│       └── <sample>/
+│           ├── <sample>.combined.txt
+│           ├── <sample>.match.txt
+│           ├── <sample>.differ.txt
+│           ├── <sample>.missing_on_dst.txt
+│           ├── <sample>.missing_on_src.txt
+│           └── <sample>.error.txt
+├── multiqc/
+│   ├── multiqc_report.html
+│   └── multiqc_data/
+└── pipeline_info/
+    ├── nf_core_datasync_software_mqc_versions.yml
+    └── execution_* / pipeline_dag_*
+```
+
+The `rclone/` directory is split by module stage. Copy logs are published to `rclone/copy/`, pre-copy checksum validation reports are published to `rclone/checksum/<sample>/`, and post-copy source-to-destination comparison reports are published to `rclone/check/<sample>/`. The `<sample>` directory name is taken from the `sample` value in the samplesheet row.
+
+## `rclone` directory
 
-The directories listed below will be created in the results directory after the pipeline has finished. All paths are relative to the top-level results directory.
-
-<!-- TODO nf-core: Write this documentation describing your workflow's output -->
+<details markdown="1">
+<summary>Output files</summary>
 
-## Pipeline overview
+- `rclone/copy/`
+  - `<sample>-rclone-copy.log`: informational log from the copy operation.
+- `rclone/checksum/<sample>/`
+  - `<sample>.combined.txt`: combined pre-copy checksum-validation status, one path per line.
+  - `<sample>.match.txt`: paths whose content matched the supplied checksum manifest (`=`).
+  - `<sample>.differ.txt`: paths present in the source and manifest but with different content (`*`).
+  - `<sample>.missing_on_dst.txt`: paths present in the checksum manifest but absent from the checked source (`-`).
+  - `<sample>.missing_on_src.txt`: paths present in the checked source but absent from the checksum manifest (`+`).
+  - `<sample>.error.txt`: paths that could not be read or hashed (`!`).
+- `rclone/check/<sample>/`
+  - `<sample>.combined.txt`: combined post-copy source-to-destination comparison status, one path per line.
+  - `<sample>.match.txt`: paths whose content matched between source and destination (`=`).
+  - `<sample>.differ.txt`: paths present on both sides but with different content (`*`).
+  - `<sample>.missing_on_dst.txt`: paths found in the source but absent from the destination (`-`).
+  - `<sample>.missing_on_src.txt`: paths found at the destination but absent from the source (`+`).
+  - `<sample>.error.txt`: paths that could not be read or hashed (`!`).
 
-The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps:
+</details>
 
-- [FastQC](#fastqc) - Raw read QC
-- [MultiQC](#multiqc) - Aggregate report describing results and QC from the whole pipeline
-- [Pipeline information](#pipeline-information) - Report metrics generated during the workflow execution
+Two integrity stages create reports:
 
-### FastQC
+1. **Pre-copy checksum validation** uses each supplied MD5 and/or SHA-256 manifest to check the source and publishes reports under `rclone/checksum/<sample>/`.
+2. **Post-copy validation** compares the source with the destination after the copy task finishes and publishes reports under `rclone/check/<sample>/`.
 
-<details markdown="1">
-<summary>Output files</summary>
+Both stages use the same `<sample>.*.txt` naming convention and publish to `rclone/`. When a row supplies a checksum manifest, similarly named pre-copy and post-copy files may target the same published path; use the consolidated MultiQC sections for the stage-specific summary and retain the Nextflow work directory if both raw report sets must be audited independently.
 
-- `fastqc/`
-  - `*_fastqc.html`: FastQC report containing quality metrics.
-  - `*_fastqc.zip`: Zip archive containing the FastQC report, tab-delimited data file and plot images.
+The combined files use `rclone`'s one-character status prefixes:
 
-</details>
+| Prefix | Meaning                  | Action                                                                                                                      |
+| ------ | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- |
+| `=`    | File matches             | No action required.                                                                                                         |
+| `-`    | Missing from destination | Investigate an incomplete source checksum set or transfer.                                                                  |
+| `+`    | Missing from source      | Review unexpected destination content. The post-copy check uses `--one-way`, so destination-only files are tolerated there. |
+| `*`    | Content differs          | Re-copy or investigate source/destination mutation.                                                                         |
+| `!`    | Read/hash error          | Inspect permissions, credentials, connectivity, and the copy log.                                                           |
 
-[FastQC](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/) gives general quality metrics about your sequenced reads. It provides information about the quality score distribution across your reads, per base sequence content (%A/T/G/C), adapter contamination and overrepresented sequences. For further reading and documentation see the [FastQC help pages](http://www.bioinformatics.babraham.ac.uk/projects/fastqc/Help/).
+Empty category files mean that `rclone` reported no entries in that category. The commands are designed to preserve these reports rather than terminate the whole workflow on comparison differences. Always inspect the reports; workflow success alone is not an integrity guarantee.
 
-### MultiQC
+## MultiQC
 
 <details markdown="1">
 <summary>Output files</summary>
 
 - `multiqc/`
-  - `multiqc_report.html`: a standalone HTML file that can be viewed in your web browser.
-  - `multiqc_data/`: directory containing parsed statistics from the different tools used in the pipeline.
-  - `multiqc_plots/`: directory containing static images from the report in various formats.
+  - `multiqc_report.html`: standalone report viewable in a browser.
+  - `multiqc_data/`: machine-readable data, logs, source inventory, software versions, and parsed rclone tables.
 
 </details>
 
-[MultiQC](http://multiqc.info) is a visualization tool that generates a single HTML report summarising all samples in your project. Most of the pipeline QC results are visualised in the report and further statistics are available in the report data directory.
+The MultiQC report consolidates:
+
+- checksum validation status for MD5 and/or SHA-256 manifests generated from `rclone checksum`;
+- post-copy source-to-destination validation status generated from `rclone check`;
+- the validated samplesheet and workflow parameter summary; and
+- pipeline and tool versions.
 
-Results generated by MultiQC collate pipeline QC from supported tools e.g. FastQC. The pipeline has special steps which also allow the software versions to be reported in the MultiQC output for future traceability. For more information about how to use MultiQC reports, see <http://multiqc.info>.
+### `rclone checksum` section (source integrity checks)
 
-### Pipeline information
+The MD5 and SHA-256 input-validation sections show the results from [`rclone checksum`](https://rclone.org/commands/rclone_checksum/). Use these sections to confirm that each source file matched the checksum manifest supplied in `checksum_md5` and/or `checksum_sha` before copying.
+
+When a samplesheet `input` points to S3, `rclone checksum` can normally validate MD5 manifests from S3 object hashes, but S3 does not provide SHA-256 object hashes for rclone to read remotely. SHA-256 validation for S3 inputs therefore requires `rclone checksum --download`, which downloads object data and calculates the hash locally during validation. Configure the checksum process to pass `--download` when SHA-256 validation is required for S3 inputs; see the [`rclone checksum` --download documentation](https://rclone.org/commands/rclone_checksum/).
+
+![nf-core/multiqc checksum md5](images/datasync-multiqc-checksum-md5.png)
+
+### `rclone check` section (post-transfer checks)
+
+The source-destination validation section shows the results from `rclone check` after the copy step. Use this section to confirm that copied files at `output_path` match the corresponding source files.
+
+![nf-core/multiqc checksum md5](images/datasync-multiqc-post-transfer-check.png)
+
+Open `multiqc_report.html` after every run and investigate any non-matching, missing, or error entries. In the `rclone` result sections, entries are prioritised so rows needing attention are shown before successful matches: errors, mismatches, and missing-file statuses appear ahead of matching files. Data under `multiqc_data/` can be retained for automated auditing or downstream reporting; exact filenames may vary with the MultiQC version and the checksum types present in the samplesheet.
+
+## Pipeline information
 
 <details markdown="1">
 <summary>Output files</summary>
 
 - `pipeline_info/`
-  - Reports generated by Nextflow: `execution_report.html`, `execution_timeline.html`, `execution_trace.txt` and `pipeline_dag.dot`/`pipeline_dag.svg`.
-  - Reports generated by the pipeline: `pipeline_report.html`, `pipeline_report.txt` and `software_versions.yml`. The `pipeline_report*` files will only be present if the `--email` / `--email_on_fail` parameter's are used when running the pipeline.
-  - Reformatted samplesheet files used as input to the pipeline: `samplesheet.valid.csv`.
-  - Parameters used by the pipeline run: `params.json`.
+  - `nf_core_datasync_software_mqc_versions.yml`: versions of the pipeline and tools collected for MultiQC.
+  - `execution_timeline_<timestamp>.html`: chronological task execution view.
+  - `execution_report_<timestamp>.html`: task runtime and resource report.
+  - `execution_trace_<timestamp>.txt`: tabular task-level execution trace.
+  - `pipeline_dag_<timestamp>.html`: workflow dependency graph.
+  - Completion reports generated when `--email` or `--email_on_fail` is configured may also be present.
 
 </details>
 
-[Nextflow](https://www.nextflow.io/docs/latest/tracing.html) provides excellent functionality for generating various reports relevant to the running and execution of the pipeline. This will allow you to troubleshoot errors with the running of the pipeline, and also provide you with other information such as launch commands, run times and resource usage.
+These files provide operational provenance and help diagnose performance or failures. Archive them with the MultiQC and rclone reports. The Nextflow `work/` directory and `.nextflow.log` remain in the launch directory rather than `--outdir`; keep them until transfer verification is complete if detailed troubleshooting or `-resume` may be needed.
+
+The supplied rclone configuration is an input credential file and is not intentionally copied to `--outdir`. Nevertheless, execution logs may contain remote names and object paths. Review logs before sharing them, and manage `rclone.conf` separately as a secret.