Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,15 +69,26 @@ with serial.open_serial_device(behavior, port="COM3") as device:
```python
from harp import data

# Finds device.yml in the folder, builds the device, returns a ready-to-use reader.
reader = data.create_dataset_reader("session.harp")
behavior = reader.device_module
df = reader.read(behavior.AnalogData) # by register class
df = reader.read(44) # or by address
everything = reader.read_all() # {register_name: DataFrame}
# Finds device.yml in the folder, builds the device, returns a ready-to-use reader
reader = data.open_dataset("session.harp")
df = reader.read("AnalogData") # by name
df = reader.read(44) # or by address

# `contents` names every register the folder holds
frames = {name: reader.read(name) for name in reader.contents}
```

Given a device package already in hand, pass it as the second argument and read by register class. This is the form that type-checks, and it also checks the device identity against the `device.yml` in the folder:

```python
from harp import data
from harp.device import behavior

reader = data.open_dataset("session.harp", behavior)
df = reader.read(behavior.AnalogData)
```

Both paths are based on a device schema. Given only a `device.yml` and no pre-generated package, `create_device_module` compiles it into a module of register classes at runtime, with no code-generation step. This is exactly what `create_dataset_reader` does internally:
Both paths are based on a device schema. Given only a `device.yml` and no pre-generated package, `create_device_module` compiles it into a module of register classes at runtime, with no code-generation step. This is exactly what `open_dataset` does internally:

```python
from pathlib import Path
Expand Down
2 changes: 1 addition & 1 deletion docs/api/data.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

---

::: harp.data.create_dataset_reader
::: harp.data.open_dataset
::: harp.data.DatasetReader
::: harp.data.default_file_resolver
::: harp.data.parse_to_dataframe
Expand Down
2 changes: 1 addition & 1 deletion docs/examples/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,5 @@ Talking to a device:

Reading recorded data:

- [Reading a Whole Dataset Folder](./read_dataset/read_dataset.md) - load an entire recorded session folder into pandas DataFrames with `DatasetReader`.
- [Reading a Whole Dataset Folder](./read_dataset/read_dataset.md) - read registers from a recorded session folder into pandas DataFrames, decoded against the device schema.
- [Reading Data into a DataFrame](./read_data_to_dataframe/read_data_to_dataframe.md) - decode the binary file of a single register into a pandas DataFrame.
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
This example demonstrates how to load the binary data file of a **single** Harp register into a pandas DataFrame using `harp.data`. The register definition tells `parse_to_dataframe` how to decode each frame, so the result carries named columns and decoded enums.

!!! tip
For a whole recorded session folder rather than one loose file, use [`DatasetReader`](../read_dataset/read_dataset.md), which reads every register in a dataset folder based on the device schema.
For a recorded session folder rather than one loose file, use [`open_dataset`](../read_dataset/read_dataset.md), which resolves each register against the device schema so any of them can be read by class, by name, or by address.

<!--codeinclude-->
```python
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@
with open("OperationControl.bin", "rb") as f:
df = data.parse_to_dataframe(core.OperationControl, f)

# To read a whole recorded session folder at once, covering many registers based on
# the device schema, use `harp.data.DatasetReader`. See the "Reading a Whole Dataset
# Folder" example.
# To read registers from a recorded session folder, resolved against the device
# schema, use `harp.data.open_dataset`. See the "Reading a Whole Dataset Folder"
# example.
2 changes: 1 addition & 1 deletion docs/examples/read_dataset/read_dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A Harp acquisition is usually saved as a **de-multiplexed dataset folder**: one

This is the recommended entry point for a recorded session on disk. To decode a single loose `.bin` file instead, see [Reading Data into a DataFrame](../read_data_to_dataframe/read_data_to_dataframe.md).

The quickest way in is `create_dataset_reader(folder)`. It finds the `device.yml` inside the folder, builds the device module, and returns a reader ready to go. Given a device module already in hand, for example from a pre-generated package, construct `DatasetReader(module, folder)` directly instead. A register is then read by class or by address, or every register at once with `read_all()`. Timestamps are detected automatically and placed on the `"Time"` index, as float seconds or an absolute `DatetimeIndex` when an `epoch` is passed.
The quickest way in is `open_dataset(folder)`. It finds the `device.yml` inside the folder, builds the device module, and returns a reader ready to go. Given a device module already in hand, for example from a pre-generated package, pass it as the second argument, `open_dataset(folder, module)`. A register is then read by class, by name, or by address. The Harp time becomes the `"Time"` index, as float seconds or an absolute `DatetimeIndex` when an `epoch` is passed.

<!--codeinclude-->
```python
Expand Down
34 changes: 21 additions & 13 deletions docs/examples/read_dataset/read_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,38 +11,46 @@
# ┣ ...
# ┗ 📜 device.yml
#
# `create_dataset_reader` does the right thing: it finds `device.yml` inside the
# folder, builds the module of register classes that knows how to decode each
# register, and hands back a reader ready to go.
reader = data.create_dataset_reader("session.harp")
# `open_dataset` does the right thing: it finds `device.yml` inside the folder,
# builds the module of register classes that knows how to decode each register,
# and hands back a reader ready to go.
reader = data.open_dataset("session.harp")

# Read one register into a DataFrame by register class, which covers any register
# in the device map, including common ones such as `OperationControl`.
df = reader.read(core.OperationControl)

# A register can also be read by address. Timestamps are auto-detected from the
# frames, and when present they become the DataFrame index, named "Time", holding
# float seconds from device start.
# A register can also be read by name. Names resolve through the device register
# map rather than the module namespace, so common registers are reachable too.
df = reader.read("OperationControl")

# Or by address. The Harp time becomes the DataFrame index, named "Time",
# holding float seconds from device start.
df = reader.read(44)
print(df.head())

# Read every register that has a file on disk at once, keyed by register name.
everything = reader.read_all()
print(list(everything))
# `contents` names every register that has data in this folder. Most datasets log
# every register by default.
frames = {name: reader.read(name) for name in reader.contents}
print(list(frames))

# Pass an epoch to turn the "Time" index into an absolute `DatetimeIndex` instead
# of float seconds. `REFERENCE_EPOCH` is time zero of the Harp clock in UTC.
absolute = reader.read(44, epoch=data.REFERENCE_EPOCH)
print(absolute.index[:3])

# --- Working from a device module already in hand ----------------------------
# A pre-generated device package, or one built with `create_device_module`,
# can be passed to the reader directly as `DatasetReader(module, folder)`:
# A pre-generated device package, or one built with `create_device_module`, is
# passed as the second argument. Either way the device identity is checked against
# the `device.yml` in the folder, so a module paired with the wrong session fails
# here rather than decoding against the wrong register map. A generated package
# adds register classes a type checker can verify:
#
# from pathlib import Path
#
# from harp import data
# from harp.device import schema
#
# behavior = schema.create_device_module((Path("session.harp") / "device.yml").read_bytes())
# reader = data.DatasetReader(behavior, "session.harp")
# reader = data.open_dataset("session.harp", behavior)
# df = reader.read(behavior.AnalogData)
34 changes: 23 additions & 11 deletions src/packages/harp-data/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,33 +19,45 @@ A Harp acquisition is usually saved as a de-multiplexed folder, one binary file
┗ 📜 device.yml
```

Reading is based on a [device module](../harp-device) that describes how to decode each register. `create_dataset_reader` supplies one automatically. It finds the `device.yml` in the folder, builds the module, and returns a ready-to-use reader:
Reading is based on a [device module](../harp-device) that describes how to decode each register. `open_dataset` supplies one automatically. It finds the `device.yml` in the folder, builds the module, and returns a ready-to-use reader:

```python
from harp import data

reader = data.create_dataset_reader("session.harp")
behavior = reader.device_module
df = reader.read(behavior.AnalogData) # by register class
df = reader.read(44) # by address
everything = reader.read_all() # {register_name: DataFrame}
reader = data.open_dataset("session.harp")
df = reader.read("AnalogData") # by name
df = reader.read(44) # by address
```

`contents` maps every register with data in the folder to its address, keyed by register name. It is the place to start on an unfamiliar dataset, and since its keys are exactly what `read` takes, loading a whole dataset can be done with a comprehension:

```python
reader.contents # {'WhoAmI': 0, 'AnalogData': 33, ...}

frames = {name: reader.read(name) for name in reader.contents}
```

Given a device module already in hand, either a pre-generated package or one built with `create_device_module`, pass it to `DatasetReader` directly:
A name is resolved through the device register map rather than the module namespace, so the common registers are reachable by name too.

A register declared in the device register map with no data present in the folder reads as an empty DataFrame carrying the same columns, since the schema describes the structure of the data regardless of whether anything was recorded. `contents` is what tells the two cases apart. A register the device does not declare at all raises `KeyError`.

Given a device module already in hand, either a pre-generated package or one built with `create_device_module`, pass it as the second argument:

```python
from harp import data
from harp.device import behavior

reader = data.DatasetReader(behavior, "session.harp")
df = reader.read(behavior.AnalogData)
reader = data.open_dataset("session.harp", behavior)
df = reader.read(behavior.AnalogData) # by register class
```

Timestamps are auto-detected per register and placed on the DataFrame index named `"Time"`: float seconds by default, or an absolute `DatetimeIndex` when `epoch=REFERENCE_EPOCH` is passed. Multi-chunk registers logged as `<DeviceName>_<address>_<suffix>.bin` are concatenated in filename order; pass a `resolver` to support an alternative on-disk layout.
Prefer the register class where a generated package supplies one, since it is the only form that type-checks and a misspelling is caught before the folder is read. A module built by `create_device_module` resolves its registers as `Any`, so there the class verifies no more than the name does.

The Harp time becomes the DataFrame index named `"Time"`, as float seconds by default or an absolute `DatetimeIndex` when `epoch=REFERENCE_EPOCH` is passed. Data carrying no timestamp raise unless `timestamp=False` is passed. Multi-chunk registers logged as `<DeviceName>_<address>_<suffix>.bin` are concatenated in filename order; pass a `resolver` to support an alternative on-disk layout. `paths` reports what the resolver found, keyed by address, which is where a custom layout or a chunked register can be checked.

The `<DeviceName>` prefix comes from the `DEVICE_NAME` declared by the device module. Pass `name=` to override it, or to supply one when the module declares an empty name.

When the folder carries a `device.yml` and the module declares an identity, their `whoAmI` values are checked against each other. Reusing a module across sessions and reaching the wrong folder then fails on construction rather than decoding the files against the wrong register map. Pass `validate=False` to turn off every check the reader performs, so a folder whose `device.yml` is damaged can be read with a module obtained elsewhere.
When a device module declaring an identity is supplied and the folder carries a `device.yml`, their `whoAmI` values are checked against each other. Reusing a module across sessions and reaching the wrong folder then fails on construction rather than decoding the files against the wrong register map. Pass `validate=False` to turn off every check the reader performs, so a folder whose `device.yml` is damaged can be read with a module obtained elsewhere.

## Read a single register file

Expand Down
4 changes: 2 additions & 2 deletions src/packages/harp-data/src/harp/data/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from ._dataset import DatasetReader, create_dataset_reader, default_file_resolver
from ._dataset import DatasetReader, default_file_resolver, open_dataset
from ._read import read
from ._reader import REFERENCE_EPOCH, parse_to_dataframe, payload_to_dataframe
from ._write import to_buffer, to_file
Expand All @@ -10,7 +10,7 @@
"to_buffer",
"to_file",
"DatasetReader",
"create_dataset_reader",
"open_dataset",
"default_file_resolver",
"REFERENCE_EPOCH",
]
Loading