Feat/incremental update dat 643 - #52
Merged
Merged
Conversation
…date - Introduced a new `Annotation` class to encapsulate annotation data and provide methods for serialization and deserialization. - Updated `DatamintBaseDataset` to utilize the new `Annotation` class, converting annotations from dictionaries to `Annotation` objects. - Refactored methods to handle annotations, including filtering and loading segmentations, to work with the new class structure. - Enhanced dataset metadata handling to support incremental updates for new and deleted resources and annotations. - Improved error handling and logging for better traceability during dataset operations.
…otationAPIHandler and update DatamintBaseDataset to utilize this feature. Bump version to 1.7.0.
There was a problem hiding this comment.
Pull Request Overview
This pull request introduces incremental dataset updates and new annotation management features to the datamint library, along with version bump to 1.7.0. The changes enable efficient dataset synchronization through batch downloading operations and structured annotation handling.
- Added incremental update functionality for datasets to sync changes without full re-download
- Introduced structured
Annotationclass to replace dictionary-based annotation handling - Added asynchronous batch downloading methods for resources and segmentation files
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Version bump from 1.6.3-post1 to 1.7.0 |
| datamint/dataset/annotation.py | New Annotation dataclass with serialization and validation methods |
| datamint/dataset/base_dataset.py | Major refactoring for incremental updates and Annotation class integration |
| datamint/dataset/dataset.py | Updated to use Annotation objects instead of dictionaries |
| datamint/apihandler/root_api_handler.py | Added batch resource downloading and project resource retrieval methods |
| datamint/apihandler/base_api_handler.py | Enhanced async request handling with content download support |
| datamint/apihandler/annotation_api_handler.py | Added batch segmentation downloading and individual annotation retrieval |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant enhancements and new features to the
datamintAPI handlers and dataset modules. Key changes include the addition of new methods for downloading and managing resources and annotations, the introduction of anAnnotationclass for structured annotation handling, and various improvements to asynchronous operations and code readability.New Features and Enhancements
Annotation Management:
download_segmentation_filemethod to download segmentation files with optional file path saving (datamint/apihandler/annotation_api_handler.py).download_multiple_segmentationsbatch downloading of segmentation files with asynchronous support (datamint/apihandler/annotation_api_handler.py).Resource Management:
download_multiple_resourcesmethods for downloading multiple resources asynchronously (datamint/apihandler/root_api_handler.py).Annotation Class:
Annotationclass to represent and manage annotations, including methods for serialization, validation, and utility functions (datamint/dataset/annotation.py). We will probably change the location of this file.Improvements to Asynchronous Operations