TODO.md

TODO

Large Sensor Array Operations

Open3D is a useful model for this project: keep the API small, composable, NumPy-first, and fast enough for very large robotics datasets. The operations below should work on in-memory buffers first, then TileDB-backed arrays where practical.

Reference categories: Open3D point cloud tutorial, Open3D PointCloud API, and Open3D ICP registration tutorial.

Prioritized Next Work

1. P0 - Finish navigation quality: covariance propagation plus quality/status masks.
2. P0 - Add operation pipelines that stream directly from DataSources, write to DataBuffer, and persist to TileDB without materializing full topics.
3. P1 - Add progress reporting, cancellation, and resumable operation checkpoints for long source and topic pipelines.
4. P1 - Finish DEM terrain operations: terrain patches, roughness/traversability, and DEM-to-point-cloud/mesh conversion.
5. P2 - Add optional parallel execution for independent chunks/topics.
6. P2 - Add benchmark tests for core operations on synthetic image, point cloud, IMU, odometry, navsat, DEM, and TileDB workloads.
7. P2 - Add ML-ready exports, deterministic splits, augmentations, and mixed-rate collation.
8. P3 - Finish DEM tile reprojection, resampling, and cache support.
9. P3 - Work through the package-and-publish checklist for TestPyPI and PyPI.

Backlog

• Define a common operation interface for buffered topics:
  • Add map(topic, fn), filter(topic, predicate), reduce(topic, fn), and window(topic, size|seconds) helpers.
  • Support eager NumPy output and lazy/chunked iteration for larger-than-memory arrays.
  • Preserve message metadata: timestamp, topic, name, frame id, shape, dtype, and source URI.
  • Add consistent copy, out, and chunk_size options for memory-sensitive workflows.
• Add dataset-level selection and indexing:
  • Select by topic, timestamp range, message index range, frame id, geographic bounds, and spatial bounds.
  • Add timestamp range and message index range selection helpers.
  • Add nearest-time lookup and bounded nearest alignment helpers.
  • Add generic numeric time-series interpolation helpers.
  • Add topic alignment modes: exact timestamp, nearest neighbor, bounded tolerance, fixed-rate resampling, and rolling window joins.
  • Add persistent secondary indexes for TileDB-backed timestamp and message-name queries.
  • Add persistent secondary indexes for frame id and spatial bounds queries.
• Add geometry and coordinate-frame operations:
  • Apply SE(3) transforms to point clouds, odometry poses, navsat-derived local coordinates, and DEM grids.
  • Add SE(3) transform helpers for XYZ point arrays.
  • Convert IMU, odometry, and navsat streams into common pose/trajectory arrays.
  • Add frame graph support for static and time-varying transforms.
  • Add projection helpers between point clouds, depth images, RGB images, DEM tiles, and camera frames.
  • Add crop/select helpers for axis-aligned bounds, oriented bounds, masks, and geographic bounding boxes.
  • Add axis-aligned XYZ bounds cropping with mask output.
• Add point cloud operations:
  • Downsample by voxel grid, uniform sampling, random sampling, and farthest-point sampling.
  • Add voxel-grid downsampling.
  • Estimate normals, local covariance, curvature-like descriptors, and nearest-neighbor distance statistics.
  • Add normal estimation.
  • Remove outliers with statistical and radius-based filters.
  • Cluster and segment with DBSCAN, plane fitting, connected components, and ground/non-ground separation.
  • Add DBSCAN clustering and RANSAC-style plane fitting.
  • Add nearest-neighbor search with KNN, radius search, and hybrid search.
  • Add KNN and radius search.
  • Add registration helpers for point-to-point ICP, point-to-plane ICP, multi-scale ICP, and odometry-seeded registration.
  • Add loop closure candidate search and ICP verification for datasets with point cloud and pose streams.
  • Calibrate relative point clouds to accurate metric point clouds and apply the fitted scale/offset.
  • Add conversion adapters to and from Open3D point clouds when open3d is installed.
• Add image and depth operations:
  • Resize, crop, pad, normalize, color convert, and dtype convert image sequences.
  • Add resize-nearest, pad, normalize, and RGB-to-gray helpers.
  • Add masks, morphology, thresholding, gradients, pyramids, and local statistics.
  • Add depth-image operations: valid-depth masks, backprojection to point clouds, depth-to-normal, and RGB-D fusion.
  • Add valid-depth masks and depth backprojection to point clouds.
  • Calibrate relative depth images to accurate metric point clouds and apply the fitted scale/offset.
  • Add frame-to-frame optical flow, image alignment, and motion-compensated rolling windows.
  • Add camera model utilities for intrinsics, distortion, rectification, and projection.
• Add IMU, odometry, and navsat operations:
  • Resample and interpolate orientation, angular velocity, linear acceleration, position, velocity, and covariance.
  • Add generic numeric time-series interpolation.
  • Add quaternion normalization, SLERP, Euler conversion, gravity compensation, and bias correction helpers.
  • Add quaternion normalization and SLERP.
  • Convert WGS84 navsat samples to local ENU/NED frames and back.
  • Add approximate WGS84 to local ENU conversion and inverse conversion.
  • Add trajectory smoothing, differentiation, integration, and dead-reckoning helpers.
  • Add covariance propagation and quality/status masks for navigation streams.
• Add DEM and raster operations:
  • Mosaic, crop, reproject, resample, and cache DEM tiles.
  • Add mosaic, crop, bilinear sampling, and nearest sampling helpers.
  • Compute slope, aspect, hillshade, normals, gradients, roughness, and traversability maps.
  • Add slope, aspect, and hillshade helpers.
  • Sample elevation at navsat/trajectory points and generate local terrain patches around a vehicle pose.
  • Add raster grid sampling helper.
  • Convert DEM windows to point clouds, meshes, or height grids for fusion with sensor topics.
• Add large-array execution features:
  • Add chunked operation execution for buffered topic arrays that do not fit in memory.
  • Add lazy buffered-topic pipelines with explicit collect(), iter_chunks(), iter_rows(), reduce(), and sliding-window execution.
  • Push lazy buffered-topic time and index constraints into TileDB before reading data chunks.
  • Add guarded materialization limits for explicit collect() calls.
  • Reopen existing TileDB datasets without the original source.
  • Resume partial TileDB ingest by replaying the source and skipping stored per-topic offsets.
  • Add operation pipelines that can stream from DataSources, write to DataBuffer, and persist to TileDB.
  • Add optional parallel execution for independent chunks/topics.
  • Add progress reporting, cancellation, and resumable operation checkpoints.
  • Add benchmark tests for core operations on synthetic image, point cloud, IMU, odometry, navsat, DEM, and TileDB workloads.
• Add ML-ready dataset operations:
  • Export topic windows to PyTorch, NumPy, and plain iterator datasets.
  • Add deterministic train/validation/test splits by time, sequence, geography, or source file.
  • Add augmentation operations for images, point clouds, trajectories, and DEM patches.
  • Add batch collation for variable-size point clouds and mixed-rate sensor windows.

Package And Publish To Python Registries

• Confirm the package metadata in pyproject.toml:

  • Package name is correct for the registry: arraydataengine.
  • Version is bumped for the release.
  • Description, README, license, authors, URLs, classifiers, and requires-python are accurate.
  • Optional dependency groups cover supported installs: dev, image, ros, dem, tiledb, visualization, notebook, and ml.

• Add release tooling if it is not already installed:

  python -m pip install --upgrade build twine

• Run the pre-release checks from a clean working tree:

  python -m pytest -q
  python -m compileall -q ade tests
  git diff --check

• Build the source distribution and wheel:

  python -m build

• Validate the built artifacts:

  python -m twine check dist/*
  python -m pip install --force-reinstall dist/*.whl
  python -m pytest -q

• Publish to TestPyPI first:

  python -m twine upload --repository testpypi dist/*

• Verify the TestPyPI install in a fresh virtual environment:

  python -m venv /tmp/ade-testpypi
  /tmp/ade-testpypi/bin/python -m pip install --upgrade pip
  /tmp/ade-testpypi/bin/python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ arraydataengine
  /tmp/ade-testpypi/bin/python -c "import ade; print(ade.__file__)"

• Create and push the release commit and tag:

  git add pyproject.toml README.md TODO.md
  git commit -m "Release vX.Y.Z"
  git tag vX.Y.Z
  git push origin main --tags

• Publish the same checked artifacts to PyPI:

  python -m twine upload dist/*

• Verify the PyPI install in a fresh virtual environment:

  python -m venv /tmp/ade-pypi
  /tmp/ade-pypi/bin/python -m pip install --upgrade pip
  /tmp/ade-pypi/bin/python -m pip install arraydataengine
  /tmp/ade-pypi/bin/python -c "import ade; print(ade.__file__)"

• Create a GitHub release from the pushed tag and attach the generated dist/ artifacts.

• Record the released version, PyPI URL, TestPyPI URL, and release notes in the project README or GitHub release notes.