diff --git a/README.md b/README.md index 11d57391..35dc5f4a 100644 --- a/README.md +++ b/README.md @@ -1,43 +1,47 @@ +

+ Datamint logo +

+ # Datamint Python API ![Build Status](https://github.com/SonanceAI/datamint-python-api/actions/workflows/run_test.yaml/badge.svg) [![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/) -A comprehensive Python SDK for interacting with the Datamint platform, providing seamless integration for medical imaging workflows, dataset management, and machine learning experiments. +Datamint turns medical imaging ML work. Dataset management, annotation, training, and deployment into a few lines of Python, with built-in support for DICOM/NIfTI/PNG, PyTorch Lightning trainers, and MLflow tracking. + +**Common use cases:** ๐Ÿฉป Segmentation ยท ๐Ÿท๏ธ Classification ยท ๐Ÿ“ฆ Detection ## ๐Ÿ“‹ Table of Contents +- [See it in action](#-see-it-in-action) - [Features](#-features) -- [Installation](#-installation) -- [Quick Setup](#-setup-api-key) +- [Quick Start](#-quick-start) - [Documentation](#-documentation) -- [Command Line Tools](#๏ธ-command-line-tools) - [Support](#-support) +## ๐ŸŽฌ See it in action + +From a project name to a deployed, validated model, patient-wise splitting, training, and deployment included: + +![Datamint pipeline demo](assets/pipeline-demo.gif) + ## ๐Ÿš€ Features - **Dataset Management**: Download, upload, and manage medical imaging datasets using intuitive object-based APIs or CLI tools - **Annotation Tools**: Create, upload, and manage annotations (segmentations, labels, measurements) with ease - **Experiment Tracking**: Seamless support for experiment management via MLflow integration -- **PyTorch Lightning Integration**: Streamlined machine learning workflows featuring specialized `LightningDataModules`, built-in trainers (`SegmentationTrainer`), and automated MLflow checkpoint logging +- **One-line Trainers**: Train segmentation, classification, and detection models with built-in PyTorch Lightning trainers, skipping the dataset class, training loop, and logging setup - **DICOM Support**: Native handling of DICOM files, including powerful anonymization capabilities during upload to protect patient privacy - **Multi-format Support**: Robust support for a wide range of medical imaging formats: PNG, JPEG, NIfTI (NIfTI/NRRD), DICOMs and more -See the full documentation at https://sonanceai.github.io/datamint-python-api/ +## โšก Quick Start -## ๐Ÿ“ฆ Installation - -> [!NOTE] -> We recommend using a virtual environment to avoid package conflicts. - -### From PyPI +**1. Install** `pip install -U datamint` -### Virtual Environment Setup -
-Click to expand virtual environment setup instructions +Using a virtual environment (recommended) We recommend that you install Datamint in a dedicated virtual environment, to avoid conflicting with your system packages. For instance, create the enviroment once with `python3 -m venv datamint-env` and then activate it whenever you need it with: @@ -48,7 +52,7 @@ For instance, create the enviroment once with `python3 -m venv datamint-env` and ``` 2. **Activate the environment** (run whenever you need it): - + | Platform | Command | |----------|---------| | Linux/macOS | `source datamint-env/bin/activate` | @@ -62,125 +66,48 @@ For instance, create the enviroment once with `python3 -m venv datamint-env` and
-## โš™ Setup API key - -To use the Datamint API, you need to setup your API key (ask your administrator if you don't have one). Use one of the following methods to setup your API key: - -### Method 1: Command-line tool (recommended) - -Run ``datamint-config`` (or ``python -m datamint config`` if that doesn't work) in the terminal and follow the instructions. See [command_line_tools](https://sonanceai.github.io/datamint-python-api/command_line_tools.html#configuring-the-datamint-settings) for more details. - -### Method 2: Environment variable - -Specify the API key as an environment variable. - -**Bash:** -```bash -export DATAMINT_API_KEY="my_api_key" -# run your commands (e.g., `datamint-upload`, `python script.py`) -``` - -**Python:** -```python -import os -os.environ["DATAMINT_API_KEY"] = "my_api_key" -``` - -## ๐Ÿ“š Documentation +**2. Configure your API key** -| Resource | Description | -|----------|-------------| -| [๐Ÿš€ Getting Started](https://sonanceai.github.io/datamint-python-api/getting_started.html) | Step-by-step setup and basic usage | -| [๐Ÿ“– API Reference](https://sonanceai.github.io/datamint-python-api/client_api.html) | Complete API documentation | -| [๐Ÿ”ฅ PyTorch Integration](https://sonanceai.github.io/datamint-python-api/pytorch_integration.html) | ML workflow integration | -| [๐Ÿง  Trainer Guide](https://sonanceai.github.io/datamint-python-api/trainer_api.html) | Built-in trainers, trainer lifecycle, and custom model integration | -| [๐Ÿ’ก Examples](examples/) | Practical usage examples | - -## ๐Ÿ› ๏ธ Command Line Tools - -Full documentation at [command_line_tools](https://sonanceai.github.io/datamint-python-api/command_line_tools.html). - -### Upload Resources - -**Upload DICOM files with anonymization:** ```bash -datamint-upload /path/to/dicoms --recursive --channel "training-data" --publish --tag "my_data_tag" -``` -It anonymizes by default. - -**Upload resources with segmentations and associate them with a deployed model:** -```bash -datamint-upload /path/to/dicoms \ - --recursive \ - --segmentation_path /path/to/segmentations \ - --segmentation_names /path/to/segmentation_names.yaml \ - --ai-model "knee-segmentation-v2" \ - --publish -``` -Use `--ai-model` when uploaded segmentation files should be linked to an existing deployed model by name. `--segmentation_names` accepts YAML mappings and ITK-SNAP label export CSV/TXT files. - -### Scaffold a New Project - -```bash -datamint-init -``` - -Generates a numbered set of scripts (`01_upload_data.py` through `06_deploy.py`) for your task (detection, segmentation, or classification). Run it once in a new directory and follow the scripts in order. - -### Configuration Management - -```bash -# Interactive setup datamint-config - -# Set API key -datamint-config --api-key "your-key" ``` -## ๐Ÿ”’ SSL Certificate Troubleshooting - -If you encounter SSL certificate verification errors like: -``` -SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate -``` +Follow the prompts (ask your administrator if you don't have a key yet). Environment variable and programmatic options are in the [Setup API Key guide](https://sonanceai.github.io/datamint-python-api/getting_started.html#setup-api-key). -### Quick Fix +**3. Scaffold a project โ€” the fastest way to start** -**1. Upgrade certifi:** ```bash -pip install --upgrade certifi -``` - -**2. Set environment variables:** -```bash -export SSL_CERT_FILE=$(python -m certifi) -export REQUESTS_CA_BUNDLE=$(python -m certifi) +datamint-init ``` -**3. Run your script:** -```bash -python your_script.py -``` +This is the recommended on-ramp: it asks for a project name and task type (**segmentation**, **classification**, or **detection**), then generates a ready-to-run, numbered set of scripts (`01_upload_data.py` โ†’ `06_deploy.py`) โ€” upload data, train, and deploy by running them in order. -### Alternative Solutions +**4. ...or write it yourself** -**Option 1: Use Custom CA Bundle** ```python from datamint import Api +from datamint.lightning import UNetPPTrainer -api = Api(verify_ssl="/path/to/your/ca-bundle.crt") +api = Api() +api.projects.create(name="my-project", exists_ok=True) + +trainer = UNetPPTrainer(project="my-project") +results = trainer.fit() ``` -**Option 2: Disable SSL Verification (Development Only)** -```python -from datamint import Api +## ๐Ÿ“š Documentation -# โš ๏ธ WARNING: Only use in development with self-signed certificates -api = Api(verify_ssl=False) -``` +| Resource | Description | +|----------|-------------| +| [๐Ÿš€ Getting Started](https://sonanceai.github.io/datamint-python-api/getting_started.html) | Step-by-step setup and basic usage | +| [๐Ÿ“– API Reference](https://sonanceai.github.io/datamint-python-api/client_api.html) | Complete API documentation | +| [๐Ÿ”ฅ PyTorch Integration](https://sonanceai.github.io/datamint-python-api/pytorch_integration.html) | ML workflow integration | +| [๐Ÿง  Trainer Guide](https://sonanceai.github.io/datamint-python-api/trainer_api.html) | Built-in trainers, trainer lifecycle, and custom model integration | +| [๐Ÿ› ๏ธ Command Line Tools](https://sonanceai.github.io/datamint-python-api/command_line_tools.html) | Full reference for `datamint-upload`, `datamint-init`, and `datamint-config` | +| [๐Ÿ”’ SSL Troubleshooting](https://sonanceai.github.io/datamint-python-api/ssl_troubleshooting.html) | Fixing `SSLCertVerificationError` | +| [๐Ÿ““ Notebooks](notebooks/) | Numbered, runnable tutorials. Start at `01_getting_started` and work through annotations, datasets, experiment tracking, deployment, and a full end-to-end example | ## ๐Ÿ†˜ Support [Full Documentation](https://sonanceai.github.io/datamint-python-api) [GitHub Issues](https://github.com/SonanceAI/datamint-python-api/issues) - diff --git a/assets/logo.png b/assets/logo.png new file mode 100644 index 00000000..02954d7c Binary files /dev/null and b/assets/logo.png differ diff --git a/assets/pipeline-demo.gif b/assets/pipeline-demo.gif new file mode 100644 index 00000000..ee64bf58 Binary files /dev/null and b/assets/pipeline-demo.gif differ diff --git a/assets/pipeline-demo.tape b/assets/pipeline-demo.tape new file mode 100644 index 00000000..996948eb --- /dev/null +++ b/assets/pipeline-demo.tape @@ -0,0 +1,99 @@ +# Regenerate the README demo GIF with: vhs assets/pipeline-demo.tape +# (run from the repo root; needs vhs: https://github.com/charmbracelet/vhs) +Output assets/pipeline-demo.gif + +Set Shell "bash" +Set FontSize 17 +Set Width 1000 +Set Height 340 +Set Theme "Dracula" +Set Padding 20 +Set WindowBar Colorful +Set BorderRadius 8 +Set TypingSpeed 35ms + +Hide +Type `python3 -iq assets/pipeline_demo_mock.py` +Enter +Sleep 500ms +Type `step("๐Ÿ“ฆ 1/5 Build a dataset โ€” no format handling needed")` +Enter +Sleep 300ms +Show + +Type@55ms `dataset = build_dataset("A_Cool_Project")` +Enter +Sleep 1400ms +Type@55ms `print(f"{type(dataset).__name__}: {len(dataset)} resources")` +Enter +Sleep 2600ms + +Hide +Type `step("โœ‚๏ธ 2/5 Patient-wise split โ€” no leakage between sets")` +Enter +Sleep 300ms +Show + +Type `parts = dataset.split(train=0.7, val=0.15, test=0.15, by_patient=True, seed=42)` +Enter +Sleep 900ms +Type `print({name: len(ds) for name, ds in parts.items()})` +Enter +Sleep 900ms +Type `parts.save()` +Enter +Sleep 2200ms + +Hide +Type `step("๐Ÿง  3/5 Train")` +Enter +Sleep 300ms +Show + +Type `trainer = UNetPPTrainer(project="A_Cool_Project", max_epochs=5)` +Enter +Sleep 400ms +Type `results = trainer.fit()` +Enter +Sleep 2200ms +Type@55ms `metrics = results['test_results'][0]` +Enter +Sleep 900ms +Type@55ms `print(f"dice: {metrics['dice']:.4f}")` +Enter +Sleep 1000ms +Type@55ms `print(f"iou: {metrics['iou']:.4f}")` +Enter +Sleep 2600ms + +Hide +Type `step("โœ… 4/5 Validate before shipping it")` +Enter +Sleep 300ms +Show + +Type `report = validate_model(results["model"], dataset=dataset)` +Enter +Sleep 800ms +Type `print(report)` +Enter +Sleep 2200ms + +Hide +Type `step("๐Ÿš€ 5/5 Deploy")` +Enter +Sleep 300ms +Show + +Type `job = api.deploy.start(model_name="A_Cool_Project", model_alias="champion")` +Enter +Sleep 800ms +Type `print(f"Status: {job.status}")` +Enter +Sleep 1800ms + +Hide +Type `finale()` +Enter +Show +Sleep 4500ms diff --git a/assets/pipeline_demo_mock.py b/assets/pipeline_demo_mock.py new file mode 100644 index 00000000..38d0262e --- /dev/null +++ b/assets/pipeline_demo_mock.py @@ -0,0 +1,85 @@ +# Backs the README demo GIF (see pipeline-demo.tape) with realistic, instant output +# instead of hitting a real server/GPU. Call signatures and print idioms mirror the +# real API and the notebooks/ tutorials. Regenerate with: vhs assets/pipeline-demo.tape +import time + + +def step(title): + print("\033[H\033[2J", end="") + print(title) + print("โ”€" * len(title)) + print() + + +def finale(): + print("\033[H\033[2J", end="") + print("๐Ÿ From raw data to a deployed model in 7 lines") + print() + print("โœ… Dataset loaded โ€” any format, auto-detected") + print("โœ… Patient-wise split โ€” no data leakage") + print("โœ… Trained & tracked in MLflow") + print("โœ… Validated before shipping") + print("โœ… Deployed and live") + + +def _bar(pct, width=22): + filled = int(width * pct) + return "โ”" * filled + "โ”€" * (width - filled) + + +class SplitResult(dict): + def save(self): + time.sleep(0.3) + + +class ImageDataset: + def __init__(self, project, n=780): + self.project = project + self._n = n + + def __len__(self): + return self._n + + def split(self, *, by_patient=False, seed=None, **ratios): + time.sleep(0.4) + return SplitResult(train=range(546), val=range(117), test=range(117)) + + +def build_dataset(project): + time.sleep(0.4) + return ImageDataset(project) + + +class UNetPPTrainer: + def __init__(self, project, **kwargs): + self.project = project + + def fit(self): + for epoch in range(1, 6): + pct = epoch / 5 + print(f"\rEpoch {epoch}/5 {_bar(pct)} loss={0.51 - 0.08 * epoch:.3f}", end="", flush=True) + time.sleep(0.25) + print() + return {"model": object(), "test_results": [{"dice": 0.8912, "iou": 0.8054}]} + + +def validate_model(model, dataset=None, **kwargs): + time.sleep(0.5) + return "[v] task_type matches project\n[v] inference smoke test passed" + + +class DeployJob: + status = "running" + + +class _Deploy: + def start(self, **kwargs): + time.sleep(0.5) + return DeployJob() + + +class _Api: + deploy = _Deploy() + + +api = _Api() diff --git a/docs/source/index.rst b/docs/source/index.rst index 0d7cf8e3..b4e236c5 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -32,6 +32,7 @@ Datamint trainer_api tutorials datamint_vs_raw_pytorch + ssl_troubleshooting .. toctree:: :maxdepth: 1 diff --git a/docs/source/ssl_troubleshooting.rst b/docs/source/ssl_troubleshooting.rst new file mode 100644 index 00000000..df8ff384 --- /dev/null +++ b/docs/source/ssl_troubleshooting.rst @@ -0,0 +1,51 @@ +SSL Certificate Troubleshooting +================================ + +If you encounter SSL certificate verification errors like:: + + SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate + +Quick Fix +--------- + +**1. Upgrade certifi:** + +.. code-block:: bash + + pip install --upgrade certifi + +**2. Set environment variables:** + +.. code-block:: bash + + export SSL_CERT_FILE=$(python -m certifi) + export REQUESTS_CA_BUNDLE=$(python -m certifi) + +**3. Run your script:** + +.. code-block:: bash + + python your_script.py + +Alternative Solutions +---------------------- + +**Option 1: Use Custom CA Bundle** + +.. code-block:: python + + from datamint import Api + + api = Api(verify_ssl="/path/to/your/ca-bundle.crt") + +**Option 2: Disable SSL Verification (Development Only)** + +.. warning:: + + Only use in development with self-signed certificates. + +.. code-block:: python + + from datamint import Api + + api = Api(verify_ssl=False)