dbverse benchmark reproduction

This repository contains the data manifests, benchmark scripts, and figure renderers used for the dbverse, GiottoDB, and GiottoSeq preprint. Automated scripts install the locked R package environments, download and validate the inputs, and run each benchmark panel.

The workflow was tested on a Mac mini M4 with 16 GB of memory. A complete paper-scale run requires at least 128 GB of free storage and is best run from a fast SSD while the computer is otherwise idle.

Warning

Before running the benchmarks, save any unsaved work and close memory-intensive applications. Run only one benchmark workflow at a time so that concurrent workloads do not distort results or exhaust system memory.

Overview

Directory	Contents
fig1/	Figure 1 panel benchmarks and figure renderer
fig2/	Figure 2 workflow benchmarks and biological analysis
suppfig1/	Supplementary Figure 1 panel benchmarks and renderer
supptable3/	Supplementary Table 3 benchmark implementation
data/	Input manifests, checksums, and downloaded or generated inputs
scripts/	Automated setup, data preparation, and shared entry points
environments/dbsequence-r46/	Locked R 4.6.1 environment for sequence and APA workflows

Each benchmark panel directory contains a run.R entry point and a README describing the comparison in more detail. Downloaded or generated inputs remain under the corresponding data/ subdirectory, regenerated results are written to cache/, and rendered figures are written to figures/.

Quick start

The instructions below are for Apple silicon macOS, matching the system used for the paper benchmarks.

1. Set up the software environments

Install the two required R versions side by side:

• install R 4.5.1 for Apple silicon with the official CRAN installer; and
• install R 4.6.1 with Homebrew: brew install r. If needed, install Homebrew first from brew.sh.

Install Git LFS, then clone the repository and enter its root directory:

brew install git-lfs
git lfs install
git clone https://github.com/Ed2uiz/giottodb-giottoseq-paper-demo.git
cd giottodb-giottoseq-paper-demo

The root renv.lock records the package versions used with R 4.5.1 and Bioconductor 3.22. The second renv.lock records the packages used with R 4.6.1 and Bioconductor 3.23 for dbSequence, GiottoSeq, and APA analyses. Restore both environments from the repository root:

bash scripts/setup.sh

This command verifies the exact R versions, restores each lockfile into its own local renv library, installs SeuratObject 5.4.0 for the untimed Figure 2d input-preparation step, and configures Git LFS for this repository. It does not download benchmark data or run any benchmarks. On a fresh system, allow about 1–2 hours; network speed, cached packages, and local compilation can change the time substantially. Later restores are usually faster because renv reuses downloaded and compiled packages.

The benchmark scripts select the correct environment automatically. Keep the two package libraries and lockfiles separate; combining them would change the software environments used for the reported results.

2. Download and prepare the benchmark inputs

bash scripts/prepare_all_data.sh

This command pulls Git LFS objects, downloads public and Google Drive inputs, generates the remaining synthetic datasets, and validates available SHA-256 checksums. Google Drive downloads use the R package googledrive and may open an OAuth prompt the first time they run. All prepared inputs remain under data/; see the input table for sources and locations. On a fresh checkout on the tested Mac mini and a fast SSD, allow approximately 4–8 hours for this step. Download speed can change this substantially, and valid local files are reused on later runs.

3. Run benchmark panels

Each command writes its working results under cache/. Figure 1 and Supplementary Figure 1 commands also record the completed paper result and its provenance manifest under data/ for rendering. To recreate the complete Figure 1 composite from new results, run all six Figure 1 commands in the listed order before its rendering command in step 4. Figure 2 commands render their corresponding panels directly.

Figure 1

Panel	Command	Approximate time
1d: Matrix Market ingestion	bash fig1/panel-d/run.sh	1–2 h
1e: GeoParquet ingestion	bash fig1/panel-e/run.sh	1–2 h
1f: genomic-interval ingestion	bash fig1/panel-f/run.sh	1–2 h
1g: matrix column means	bash fig1/panel-g/run.sh	2–3 h
1h: point-in-polygon counting	bash fig1/panel-h/run.sh	<30 min
1i: binned genomic coverage	bash fig1/panel-i/run.sh	1–2 h

Figure 2

Panel	Command	Approximate time
2c: simulated Giotto workflows	bash fig2/panel-c/run.sh	3–4 h
2d: public spatial-omics workflows	bash fig2/panel-d/run.sh	2–3 h
2e: 2-µm bin-to-cell pooling	bash fig2/panel-e/run.sh	~45 min
2f: RUD calculation	bash fig2/panel-f/run.sh	~45 min

Note

Panel 2d input preparation requires the R package SeuratObject to deserialize the CosMX object. Timed benchmark processes do not load it.

The shared biological analysis for Figure 2g-j is documented separately in fig2/panels-g-j/.

Supplementary Figure 1 and Supplementary Table 3

Panel or table	Command	Approximate time
Supplementary Figure 1b: matrix storage	bash suppfig1/panel-b/run.sh	1–2 h
Supplementary Figure 1c: matrix indexing	bash suppfig1/panel-c/run.sh	1–2 h
Supplementary Figure 1d: polygon intersection	bash suppfig1/panel-d/run.sh	<30 min
Supplementary Table 3	bash supptable3/run.sh	<30 min

Times are approximate total wall-clock times observed or reconstructed from paper-scale runs on the tested Mac mini. They are not plotted benchmark measurements and will vary with system specifications, network speed, and timeout outcomes.

4. Render figures

After running every Figure 1 panel:

bash fig1/render.sh

After running Supplementary Figure 1b-d, which automatically record the completed panel results used by the renderer:

bash suppfig1/render.sh

Figure 2 benchmark scripts and the Supplementary Table 3 script render their outputs directly. PNG and PDF files are written under figures/.

Reproducibility settings

• successful operations use one untimed warm-up and five timed repetitions;
• each operation attempt has a 600-second time limit;
• setup phases that gate a timed operation are measured separately and have their own 600-second limit;
• complete Figure 2c/d workflows have a 1,000-second time limit;
• benchmark subprocesses use a 14 GiB process-tree memory ceiling on the 16 GB test machine;
• later repetitions are not executed after the first timeout or out-of-memory result for the same method and input size; and
• comparisons use public package APIs and validate equivalent successful outputs outside the timed operation.