diff --git a/notebooks/upload_data.ipynb b/notebooks/01_getting_started/01_upload_data.ipynb similarity index 97% rename from notebooks/upload_data.ipynb rename to notebooks/01_getting_started/01_upload_data.ipynb index 8e32e5fd..a56d309f 100644 --- a/notebooks/upload_data.ipynb +++ b/notebooks/01_getting_started/01_upload_data.ipynb @@ -408,14 +408,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# Next Steps\n", - "\n", "This tutorial covered the main features of the Datamint Python API. For more advanced usage:\n", "\n", - "1. **Check the full documentation**: https://sonanceai.github.io/datamint-python-api/\n", - "2. **Explore other notebooks**:\n", - " - `upload_annotations.ipynb` - For simple annotation management, like image/frame categories.\n", - " - `geometry_annotations.ipynb` - For adding lines, boxes, and other geometric annotations.\n", + "**Check the full documentation**: https://sonanceai.github.io/datamint-python-api/\n", "\n", "Happy coding! 🚀" ] diff --git a/notebooks/exploring_data_tutorial.ipynb b/notebooks/01_getting_started/02_explore_data.ipynb similarity index 100% rename from notebooks/exploring_data_tutorial.ipynb rename to notebooks/01_getting_started/02_explore_data.ipynb diff --git a/notebooks/01_getting_started/README.md b/notebooks/01_getting_started/README.md new file mode 100644 index 00000000..6a5639fc --- /dev/null +++ b/notebooks/01_getting_started/README.md @@ -0,0 +1,8 @@ +# 01 — Getting Started + +Start here. These notebooks cover the basics of connecting to Datamint and working with your first project. + +| Notebook | Description | +|---|---| +| [01_upload_data](01_upload_data.ipynb) | Upload images, DICOM series, and NIfTI volumes to a project | +| [02_explore_data](02_explore_data.ipynb) | List resources, inspect metadata, and browse annotations | diff --git a/notebooks/upload_annotations.ipynb b/notebooks/02_annotations/01_upload_annotations.ipynb similarity index 100% rename from notebooks/upload_annotations.ipynb rename to notebooks/02_annotations/01_upload_annotations.ipynb diff --git a/notebooks/geometry_annotations.ipynb b/notebooks/02_annotations/02_geometry_annotations.ipynb similarity index 100% rename from notebooks/geometry_annotations.ipynb rename to notebooks/02_annotations/02_geometry_annotations.ipynb diff --git a/notebooks/02_annotations/README.md b/notebooks/02_annotations/README.md new file mode 100644 index 00000000..4f743ae8 --- /dev/null +++ b/notebooks/02_annotations/README.md @@ -0,0 +1,8 @@ +# 02 — Annotations + +How to create, upload, and work with annotations programmatically. + +| Notebook | Description | +|---|---| +| [01_upload_annotations](01_upload_annotations.ipynb) | Upload segmentation masks and image-level labels via the API | +| [02_geometry_annotations](02_geometry_annotations.ipynb) | Create and read bounding boxes, points, lines, and polygons | diff --git a/notebooks/project_scoped_splits_tutorial.ipynb b/notebooks/03_datasets/01_project_scoped_splits.ipynb similarity index 100% rename from notebooks/project_scoped_splits_tutorial.ipynb rename to notebooks/03_datasets/01_project_scoped_splits.ipynb diff --git a/notebooks/patient_wise_split.ipynb b/notebooks/03_datasets/02_patient_wise_splits.ipynb similarity index 100% rename from notebooks/patient_wise_split.ipynb rename to notebooks/03_datasets/02_patient_wise_splits.ipynb diff --git a/notebooks/build_dataset_tutorial.ipynb b/notebooks/03_datasets/03_build_dataset.ipynb similarity index 100% rename from notebooks/build_dataset_tutorial.ipynb rename to notebooks/03_datasets/03_build_dataset.ipynb diff --git a/notebooks/volume_dataset_tutorial.ipynb b/notebooks/03_datasets/04_volume_dataset.ipynb similarity index 100% rename from notebooks/volume_dataset_tutorial.ipynb rename to notebooks/03_datasets/04_volume_dataset.ipynb diff --git a/notebooks/03_datasets/README.md b/notebooks/03_datasets/README.md new file mode 100644 index 00000000..a410d7f0 --- /dev/null +++ b/notebooks/03_datasets/README.md @@ -0,0 +1,10 @@ +# 03 — Datasets + +PyTorch dataset classes, data splits, and volume loading. + +| Notebook | Description | +|---|---| +| [01_project_scoped_splits](01_project_scoped_splits.ipynb) | Create reproducible train/val/test splits and persist them to the server | +| [02_patient_wise_splits](02_patient_wise_splits.ipynb) | Patient-level splitting to prevent data leakage in multi-scan datasets | +| [03_build_dataset](03_build_dataset.ipynb) | Use `build_dataset` to auto-detect project type and get the right dataset class | +| [04_volume_dataset](04_volume_dataset.ipynb) | Load 3D volumes, slice along anatomical axes, and apply albumentations transforms | diff --git a/notebooks/mlflow_manual_logging.ipynb b/notebooks/04_experiment_tracking/01_mlflow_manual_logging.ipynb similarity index 97% rename from notebooks/mlflow_manual_logging.ipynb rename to notebooks/04_experiment_tracking/01_mlflow_manual_logging.ipynb index 2ee3f380..bba6f43a 100644 --- a/notebooks/mlflow_manual_logging.ipynb +++ b/notebooks/04_experiment_tracking/01_mlflow_manual_logging.ipynb @@ -141,14 +141,6 @@ "print(f\"View results at: {api.projects.get_by_name(PROJECT_NAME).url}\")\n", "print(\"=\" * 60)" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "1db8c59b", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { diff --git a/notebooks/04_experiment_tracking/README.md b/notebooks/04_experiment_tracking/README.md new file mode 100644 index 00000000..ff980110 --- /dev/null +++ b/notebooks/04_experiment_tracking/README.md @@ -0,0 +1,7 @@ +# 04 — Experiment Tracking + +Logging experiments with MLflow through the Datamint backend. + +| Notebook | Description | +|---|---| +| [01_mlflow_manual_logging](01_mlflow_manual_logging.ipynb) | Log metrics, parameters, and model artifacts manually using `mlflow.set_tracking_uri("datamint://...")` | diff --git a/notebooks/deploy_model_demo.ipynb b/notebooks/05_deployment/01_deploy_registered_model.ipynb similarity index 100% rename from notebooks/deploy_model_demo.ipynb rename to notebooks/05_deployment/01_deploy_registered_model.ipynb diff --git a/notebooks/external_model_deployment_tutorial.ipynb b/notebooks/05_deployment/02_deploy_external_model.ipynb similarity index 100% rename from notebooks/external_model_deployment_tutorial.ipynb rename to notebooks/05_deployment/02_deploy_external_model.ipynb diff --git a/notebooks/05_deployment/README.md b/notebooks/05_deployment/README.md new file mode 100644 index 00000000..2e1abd90 --- /dev/null +++ b/notebooks/05_deployment/README.md @@ -0,0 +1,8 @@ +# 05 — Deployment + +Deploying models as Docker images for remote inference. + +| Notebook | Description | +|---|---| +| [01_deploy_registered_model](01_deploy_registered_model.ipynb) | Start, monitor, and cancel deployment jobs for a model already registered in Datamint | +| [02_deploy_external_model](02_deploy_external_model.ipynb) | Wrap a model trained outside Datamint in a `DatamintModel` adapter and deploy it | diff --git a/notebooks/06_end_to_end/README.md b/notebooks/06_end_to_end/README.md new file mode 100644 index 00000000..cf2acedf --- /dev/null +++ b/notebooks/06_end_to_end/README.md @@ -0,0 +1,22 @@ +# 06 — End-to-End Use Cases + +Complete pipelines from raw data to a deployed model. Each notebook covers data upload, dataset preparation, training, experiment tracking, and deployment. + +## Slice-based (2D) + +Models that operate on individual 2D images or 2D slices extracted from volumes. + +| Notebook | Task | Dataset | Architecture | +|---|---|---|---| +| [01_fracatlas_classification](slice_based/01_fracatlas_classification.ipynb) | Binary classification | FracAtlas (X-rays) | ResNet | +| [02_busi_segmentation](slice_based/02_busi_segmentation.ipynb) | Semantic segmentation | BUSI (ultrasound) | TransUNet | +| [03_bccd_detection](slice_based/03_bccd_detection.ipynb) | Object detection | BCCD (blood cells) | YOLOX | + +## Full 3D + +Models that consume entire volumetric inputs without slicing. + +| Notebook | Task | Dataset | Architecture | +|---|---|---|---| +| [01_synapse_unetrpp](full_3d/01_synapse_unetrpp.ipynb) | Multi-organ segmentation | Synapse CT | UNETR++ | +| [02_synapse_nnunet](full_3d/02_synapse_nnunet.ipynb) | Multi-organ segmentation | Synapse CT | nnU-Net | diff --git a/notebooks/use_cases/unetrpp_synapse_tutorial.ipynb b/notebooks/06_end_to_end/full_3d/01_synapse_unetrpp.ipynb similarity index 100% rename from notebooks/use_cases/unetrpp_synapse_tutorial.ipynb rename to notebooks/06_end_to_end/full_3d/01_synapse_unetrpp.ipynb diff --git a/notebooks/use_cases/nnunet_synapse_tutorial.ipynb b/notebooks/06_end_to_end/full_3d/02_synapse_nnunet.ipynb similarity index 100% rename from notebooks/use_cases/nnunet_synapse_tutorial.ipynb rename to notebooks/06_end_to_end/full_3d/02_synapse_nnunet.ipynb diff --git a/notebooks/06_end_to_end/full_3d/README.md b/notebooks/06_end_to_end/full_3d/README.md new file mode 100644 index 00000000..15cbf3a2 --- /dev/null +++ b/notebooks/06_end_to_end/full_3d/README.md @@ -0,0 +1,8 @@ +# Full 3D Use Cases + +End-to-end pipelines for models that process entire volumetric inputs. + +| Notebook | Task | Dataset | +|---|---|---| +| [01_synapse_unetrpp](01_synapse_unetrpp.ipynb) | Multi-organ segmentation | Synapse CT | +| [02_synapse_nnunet](02_synapse_nnunet.ipynb) | Multi-organ segmentation | Synapse CT | diff --git a/notebooks/use_cases/fracatlas_classification.ipynb b/notebooks/06_end_to_end/slice_based/01_fracatlas_classification.ipynb similarity index 100% rename from notebooks/use_cases/fracatlas_classification.ipynb rename to notebooks/06_end_to_end/slice_based/01_fracatlas_classification.ipynb diff --git a/notebooks/use_cases/busi_2d_segmentation.ipynb b/notebooks/06_end_to_end/slice_based/02_busi_segmentation.ipynb similarity index 100% rename from notebooks/use_cases/busi_2d_segmentation.ipynb rename to notebooks/06_end_to_end/slice_based/02_busi_segmentation.ipynb diff --git a/notebooks/use_cases/yolox_bccd_detection.ipynb b/notebooks/06_end_to_end/slice_based/03_bccd_detection.ipynb similarity index 100% rename from notebooks/use_cases/yolox_bccd_detection.ipynb rename to notebooks/06_end_to_end/slice_based/03_bccd_detection.ipynb diff --git a/notebooks/06_end_to_end/slice_based/README.md b/notebooks/06_end_to_end/slice_based/README.md new file mode 100644 index 00000000..a78456ab --- /dev/null +++ b/notebooks/06_end_to_end/slice_based/README.md @@ -0,0 +1,9 @@ +# Slice-based (2D) Use Cases + +End-to-end pipelines for 2D models — classification, segmentation, and detection. + +| Notebook | Task | Dataset | +|---|---|---| +| [01_fracatlas_classification](01_fracatlas_classification.ipynb) | Fracture classification | FracAtlas | +| [02_busi_segmentation](02_busi_segmentation.ipynb) | Breast lesion segmentation | BUSI | +| [03_bccd_detection](03_bccd_detection.ipynb) | Blood cell detection | BCCD | diff --git a/notebooks/README.md b/notebooks/README.md new file mode 100644 index 00000000..cf9adf24 --- /dev/null +++ b/notebooks/README.md @@ -0,0 +1,49 @@ +# Datamint Notebooks + +Tutorials and end-to-end examples for the Datamint. +Folders are numbered in the recommended learning order. + +## Structure + +| Folder | What you'll learn | +|---|---| +| [01_getting_started](01_getting_started/) | Upload data and explore a project | +| [02_annotations](02_annotations/) | Upload and work with annotations | +| [03_datasets](03_datasets/) | Build PyTorch datasets, splits, and volume loading | +| [04_experiment_tracking](04_experiment_tracking/) | Log metrics and artifacts with MLflow | +| [05_deployment](05_deployment/) | Deploy registered and external models | +| [06_end_to_end](06_end_to_end/) | Full pipelines from data to deployed model | + +## Notebooks + +### 01 — Getting Started +1. [`01_upload_data`](01_getting_started/01_upload_data.ipynb) — Upload images and volumes to a project +2. [`02_explore_data`](01_getting_started/02_explore_data.ipynb) — Browse resources, metadata, and annotations + +### 02 — Annotations +1. [`01_upload_annotations`](02_annotations/01_upload_annotations.ipynb) — Upload segmentation masks and labels +2. [`02_geometry_annotations`](02_annotations/02_geometry_annotations.ipynb) — Work with boxes, points, lines, and polygon annotations + +### 03 — Datasets +1. [`01_project_scoped_splits`](03_datasets/01_project_scoped_splits.ipynb) — Create and persist train/val/test splits on the server +2. [`02_patient_wise_splits`](03_datasets/02_patient_wise_splits.ipynb) — Avoid data leakage with patient-level splitting +3. [`03_build_dataset`](03_datasets/03_build_dataset.ipynb) — Auto-detect dataset type with `build_dataset` +4. [`04_volume_dataset`](03_datasets/04_volume_dataset.ipynb) — Load 3D volumes, slice into 2D, apply albumentations + +### 04 — Experiment Tracking +1. [`01_mlflow_manual_logging`](04_experiment_tracking/01_mlflow_manual_logging.ipynb) — Log metrics, parameters, and models manually with MLflow + +### 05 — Deployment +1. [`01_deploy_registered_model`](05_deployment/01_deploy_registered_model.ipynb) — Deploy a model already registered in Datamint +2. [`02_deploy_external_model`](05_deployment/02_deploy_external_model.ipynb) — Wrap and deploy a model trained outside Datamint + +### 06 — End-to-End Use Cases + +#### Slice-based (2D) +1. [`01_fracatlas_classification`](06_end_to_end/slice_based/01_fracatlas_classification.ipynb) — Binary fracture classification on X-rays (FracAtlas) +2. [`02_busi_segmentation`](06_end_to_end/slice_based/02_busi_segmentation.ipynb) — 2D breast ultrasound segmentation (BUSI) +3. [`03_bccd_detection`](06_end_to_end/slice_based/03_bccd_detection.ipynb) — Object detection on blood cell images (BCCD) with YOLOX + +#### Full 3D +1. [`01_synapse_unetrpp`](06_end_to_end/full_3d/01_synapse_unetrpp.ipynb) — 3D multi-organ segmentation with UNETR++ (Synapse) +2. [`02_synapse_nnunet`](06_end_to_end/full_3d/02_synapse_nnunet.ipynb) — 3D multi-organ segmentation with nnU-Net (Synapse)