Rename node API and remove TreeDB namespace

Adopt BaseNode and DataNode as the public model and move the implementation fully into the lotdb package so the storage and measurement APIs use one consistent namespace.

Author: Spitz

README.md

@@ -6,13 +6,18 @@ LOTDB stands for Light Object Tree DB.
 
 ## What it is
 
-LOTDB stores data in `StorageTree` nodes:
+LOTDB stores data in `BaseNode` nodes:
 - each node can contain child nodes
 - each node can store arbitrary attributes
 - the full tree can be persisted with ZODB
 
 This is especially useful for dataset pipelines where folder structure, file references, and metadata all belong together.
 
+LOTDB now supports both:
+- file references for external assets
+- native ZODB blob storage for large sensor measurements and array payloads
+- backend-linked measurements via blob or Zarr
+
 ## Installation
 
 Local editable install:
@@ -26,9 +31,9 @@ With optional IO helpers:
 ## Basic usage
 
 ```python
-from lotdb import StorageTree
+from lotdb import BaseNode
 
-tree = StorageTree(key="dataset")
+tree = BaseNode(key="dataset")
 node = tree.get_node_path(["speaker_01", "session_a", "clip_001"])
 node.set_attribute("label", "hello")
 
@@ -49,25 +54,87 @@ db.close_connection()
 db.close()
 ```
 
-## Backward compatibility
+## Sensor/blob storage usage
+
+```python
+import numpy as np
+from lotdb import BlobReader, BlobWriter, LOTDB
+
+db = LOTDB(path="./data", name="lotdb.fs", new=True)
+tree = db.open_connection()
+
+node = tree.get_node_path(["sensor_01", "capture_0001"])
+BlobWriter.write_array(
+    node,
+    np.random.randn(1000, 6).astype("float32"),
+    blob_attribute="measurement",
+    metadata={"samplerate_hz": 1000, "sensor_type": "imu"},
+)
+
+db.commit()
+measurement = BlobReader.read_array(node, "measurement")
+db.close_connection()
+db.close()
+```
+
+## Data backend usage
+
+```python
+import numpy as np
+from lotdb import DataReader, DataWriter, LOTDB
+
+db = LOTDB(path="./data", name="lotdb.fs", new=True)
+tree = db.open_connection()
+node = tree.get_node_path(["sensor_01", "capture_0002"])
+
+DataWriter.write_array(
+    node,
+    np.random.randn(2000, 6).astype("float32"),
+    samplerate_hz=1000,
+    backend="zarr",  # or "blob"
+    data_attribute="imu",
+    database=db,
+    metadata={"sensor_type": "imu", "unit": "m/s^2"},
+)
+
+for block in DataReader.iter_blocks(
+    node,
+    "imu",
+    block_size=0.5,
+    block_unit="seconds",
+):
+    process(block)
+
+second_one_to_two = DataReader.read_seconds(node, "imu", 1.0, 2.0)
+
+db.commit()
+db.close_connection()
+db.close()
+```
+
+## Data node API
 
-The old short method names still work:
-- `ga()`
-- `gn()`
-- `gns()`
-- `gna()`
+```python
+data_node = tree.get_data_node(
+    ["sensor_01", "capture_0003"],
+    samplerate_hz=1000,
+    backend="zarr",
+    data_attribute="imu",
+)
 
-Readable wrappers were added so the API is easier to maintain and easier for weaker coding models to follow.
+data_node.write_data(data, database=db)
+
+for block in data_node.iter_data_blocks(0.25, block_unit="seconds"):
+    process(block)
+
+window = data_node.read_seconds(1.0, 2.0)
+```
 
-Import compatibility is also preserved for now:
-- new import: `import lotdb`
-- legacy imports still work: `import TreeDB`, `import StorageTree`
-- new database class: `LOTDB`
-- legacy alias still works: `StorageTreeDatabase`
+This is the recommended high-level API for large binary/data payloads.
 
 ## Development
 
-- source packages live in `src/lotdb` and `src/TreeDB`
+- source package lives in `src/lotdb`
 - tests live in `tests/`
 - API notes live in `docs/API.md`
 

docs/API.md

@@ -2,7 +2,7 @@
 
 ## Core classes
 
-### `StorageTree`
+### `BaseNode`
 - Hierarchical persistent node.
 - Stores child nodes in `node_storage`.
 - Stores metadata/attributes in `attribute_storage`.
@@ -28,14 +28,11 @@ Important behaviors:
 - `delete_node(..., only_node=True)` removes a node but promotes its children.
 
 ### `LOTDB`
-- ZODB-backed storage for a root `StorageTree`.
+- ZODB-backed storage for a root `BaseNode`.
 - `open_connection()` opens a transaction-scoped connection.
 - `commit()` must be called explicitly for writes.
 - `connection()` provides a context manager that closes the connection automatically.
 
-Compatibility alias:
-- `StorageTreeDatabase`
-
 ### `PathFileObj`
 - Persistent file path object storing root and filename separately.
 - Useful for keeping file references inside the tree.
@@ -48,6 +45,53 @@ Attach a `PathFileObj` to a node attribute.
 ### `FileReader` / `FileWriter`
 Helpers for wav/txt/npy oriented workflows.
 
+## Blob helpers
+
+### `BlobObject`
+- ZODB-managed binary payload stored inside the database/blob directory.
+- Can store raw bytes or NumPy arrays.
+- Supports metadata like sensor type, sample rate, units, and shape.
+
+### `BlobReader` / `BlobWriter`
+- `BlobWriter.write_bytes()` stores arbitrary binary payloads on a node.
+- `BlobWriter.write_array()` stores NumPy arrays in `.npy` format inside a blob.
+- `BlobReader.read_bytes()` and `BlobReader.read_array()` restore the payload.
+
+Typical use case:
+- sensor measurement sequences
+- multichannel waveform data
+- binary captures that should be managed by LOTDB instead of external file paths
+
+## Data helpers
+
+### `DataWriter`
+- Stores sensor arrays using a selected backend: `blob` or `zarr`.
+- Attaches a data reference object to a node.
+- Records metadata like sample rate, dtype, shape, dataset path, and custom sensor metadata.
+
+### `DataReader`
+- Reads data ranges in samples or seconds.
+- Provides `iter_blocks(...)` for streaming/block-based consumption.
+- Converts second-based ranges into sample indices using `samplerate_hz`.
+
+### `DataObject`
+- Persistent data reference stored on a node.
+- For `blob`, the payload lives in a ZODB blob.
+- For `zarr`, the node stores a link to a Zarr dataset path on disk.
+
+### `DataNode`
+- Specialized data-oriented node that inherits from `BaseNode`.
+- Stores default data configuration like backend, sample rate, and data attribute name.
+- Convenience methods:
+  - `write_data(...)`
+  - `read_data(...)`
+  - `read_seconds(...)`
+  - `iter_data_blocks(...)`
+
+### `BaseNode.get_data_node()`
+- Creates or upgrades the final node in a path to a `DataNode`.
+- This is the easiest way to create sensor capture nodes with automatic backend setup.
+
 ## Bulk loading helpers
 
 ### `LOTDB.load_files_folder()`

pyproject.toml

@@ -24,13 +24,16 @@ io = [
     "pandas",
     "scipy",
 ]
+measurements = [
+    "zarr",
+]
 dev = [
     "pytest",
+    "zarr",
 ]
 
 [tool.setuptools]
 package-dir = {"" = "src"}
-py-modules = ["StorageTree"]
 
 [tool.setuptools.packages.find]
 where = ["src"]

src/StorageTree.py

@@ -1 +1 @@
-from TreeDB.StorageTree import *
+from .api import *

src/TreeDB/__init__.py

@@ -1,7 +1,9 @@
-from .database import LOTDB, StorageTreeConnection, StorageTreeDatabase, load_small_files_directory, load_small_files_folder
+from .blobs import BlobObject, BlobReader, BlobWriter, add_blob
+from .database import LOTDB, NodeConnection, load_small_files_directory, load_small_files_folder
 from .files import FileReader, FileWriter, add_file, rename_file
+from .measurements import DataObject, DataReader, DataWriter
 from .paths import PathFileObj
-from .tree import StorageTree, StorageTreeUnconnected
+from .tree import BaseNode, BaseNodeUnconnected, DataNode
 from .utils import (
     apply_args_and_kwargs,
     filter_node_list,
@@ -15,14 +17,21 @@
 )
 
 __all__ = [
-    "StorageTree",
-    "StorageTreeUnconnected",
+    "BaseNode",
+    "BaseNodeUnconnected",
+    "DataNode",
+    "BlobObject",
+    "BlobReader",
+    "BlobWriter",
+    "DataObject",
+    "DataReader",
+    "DataWriter",
     "LOTDB",
-    "StorageTreeConnection",
-    "StorageTreeDatabase",
+    "NodeConnection",
     "PathFileObj",
     "FileReader",
     "FileWriter",
+    "add_blob",
     "add_file",
     "rename_file",
     "load_small_files_directory",

src/lotdb/StorageTree.py

@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+import shutil
+from typing import Any, Mapping, Optional
+
+import persistent
+from BTrees.OOBTree import OOBTree
+from ZODB.blob import Blob
+from numpy import ndarray
+
+
+class BlobObject(persistent.Persistent):
+    def __init__(
+        self,
+        content_type: str = "application/octet-stream",
+        metadata: Optional[Mapping[str, Any]] = None,
+    ) -> None:
+        self.blob = Blob()
+        self.metadata = OOBTree()
+        self.metadata["content_type"] = content_type
+
+        for key, value in (metadata or {}).items():
+            self.metadata[str(key)] = value
+
+    @property
+    def content_type(self) -> str:
+        return str(self.metadata.get("content_type", "application/octet-stream"))
+
+    def set_metadata(self, key: str, value: Any) -> Any:
+        self.metadata[str(key)] = value
+        return value
+
+    def get_metadata(self, key: str, default: Any = None) -> Any:
+        return self.metadata.get(str(key), default)
+
+    def metadata_dict(self) -> dict[str, Any]:
+        return dict(self.metadata.items())
+
+    def write_bytes(self, data: bytes) -> None:
+        with self.blob.open("w") as file_handle:
+            file_handle.write(data)
+
+    def read_bytes(self) -> bytes:
+        with self.blob.open("r") as file_handle:
+            return file_handle.read()
+
+    def write_array(self, array: ndarray, allow_pickle: bool = False) -> None:
+        from numpy import save
+
+        with self.blob.open("w") as file_handle:
+            save(file_handle, array, allow_pickle=allow_pickle)
+
+        self.metadata["format"] = "npy"
+        self.metadata["dtype"] = str(array.dtype)
+        self.metadata["shape"] = tuple(array.shape)
+
+    def read_array(self, allow_pickle: bool = False):
+        from numpy import load
+
+        with self.blob.open("r") as file_handle:
+            return load(file_handle, allow_pickle=allow_pickle)
+
+    def copy_for_tree(self) -> "BlobObject":
+        clone = BlobObject(content_type=self.content_type, metadata=self.metadata_dict())
+        with self.blob.open("r") as source_handle, clone.blob.open("w") as target_handle:
+            shutil.copyfileobj(source_handle, target_handle)
+        return clone
+
+
+def add_blob(
+    node,
+    blob_attribute: str = "_blob",
+    data: bytes = b"",
+    content_type: str = "application/octet-stream",
+    metadata: Optional[Mapping[str, Any]] = None,
+) -> BlobObject:
+    blob_object = BlobObject(content_type=content_type, metadata=metadata)
+    blob_object.write_bytes(data)
+    node.set_attribute(blob_attribute, blob_object)
+    return blob_object
+
+
+class BlobWriter:
+    @staticmethod
+    def write_bytes(
+        node,
+        data: bytes,
+        blob_attribute: str = "_blob",
+        content_type: str = "application/octet-stream",
+        metadata: Optional[Mapping[str, Any]] = None,
+    ) -> BlobObject:
+        blob_object = BlobObject(content_type=content_type, metadata=metadata)
+        blob_object.write_bytes(data)
+        node.set_attribute(blob_attribute, blob_object)
+        return blob_object
+
+    @staticmethod
+    def write_array(
+        node,
+        array: ndarray,
+        blob_attribute: str = "_blob_array",
+        metadata: Optional[Mapping[str, Any]] = None,
+        content_type: str = "application/x-npy",
+        allow_pickle: bool = False,
+    ) -> BlobObject:
+        blob_object = BlobObject(content_type=content_type, metadata=metadata)
+        blob_object.write_array(array, allow_pickle=allow_pickle)
+        node.set_attribute(blob_attribute, blob_object)
+        return blob_object
+
+
+class BlobReader:
+    @staticmethod
+    def read_bytes(node, blob_attribute: str = "_blob") -> bytes:
+        return node.get_attribute(blob_attribute).read_bytes()
+
+    @staticmethod
+    def read_array(node, blob_attribute: str = "_blob_array", allow_pickle: bool = False):
+        return node.get_attribute(blob_attribute).read_array(allow_pickle=allow_pickle)