Skip to content

Tutorial edits #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
FrancescAlted merged 3 commits into from
Oct 30, 2025
Merged

Tutorial edits #254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dd3642b
Corrections to tutorials
lshaw8317 Oct 30, 2025
f208c74
Corrections to tutorials
lshaw8317 Oct 30, 2025
c9406f0
Clarify browse command
lshaw8317 Oct 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/tutorials/API.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using the client APIs
# Using the Python client APIs

To follow these instructions, make sure that you have installed ``caterva2`` and ``blosc2``.

Expand All @@ -17,7 +17,7 @@ client = cat2.Client("https://cat2.cloud/demo")
client.get_roots()
# {'@public': {'name': '@public'}}
```
or, if one has created a user, using the user credentials:
or - if one has created a user - using the user credentials:

```python
client = cat2.Client("https://cat2.cloud/demo", ("user@example.com", "password1"))
Expand Down
32 changes: 21 additions & 11 deletions doc/tutorials/cli.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@ To create a user, you can use the `cat2-admin adduser` command. For example:
cat2-admin adduser user@example.com foobar11
```

To start the server use the following command:
```sh
CATERVA2_SECRET=c2sikrit cat2-server &
```

Now that the services are running, we can use the `cat2-client` client to talk
to the server. In another shell, let's list all the available roots in the system:

Expand All @@ -26,17 +31,22 @@ cat2-client --user "user@example.com" --pass "foobar11" roots
@shared
```

First let's upload a file from the `root-example`folder to the `@personal` root:
First let's generate a file and save it locally. Run the following script in the terminal to save a `b2nd` file to your current directory.
```
python -c "import blosc2; blosc2.arange(0, 1000, 1, blocks = (10,), chunks=(100,), urlpath='ds-1d.b2nd')"
```

Let's upload the file to the `@personal` root:

```sh
cat2-client --username user@example.com --password foobar11 upload root-example/ds-1d.b2nd @personal/ds-1d.b2nd
cat2-client --username user@example.com --password foobar11 upload ds-1d.b2nd @personal/ds-1d.b2nd
```

```
Dataset stored in @personal/ds-1d.b2nd
```

Now, one can list the datasets in the `@personal` root and see that the uploaded file appears
Now, one may list the datasets in the `@personal` root and see that the uploaded file appears

```sh
cat2-client --username user@example.com --password foobar11 list @personal
Expand All @@ -59,32 +69,32 @@ chunks: [100]
blocks: [10]
dtype : int64
nbytes: 7.81 KiB
cbytes: 4.90 KiB
ratio : 1.59x
mtime : 2025-10-08T11:09:03.955154Z
cbytes: 3.86 KiB
ratio : 2.02x
mtime : 2025-10-30T11:16:17.012415Z
cparams:
codec : ZSTD (5)
clevel : 1
clevel : 5
filters: [SHUFFLE]
```

As you see, this command returns digested information of the dataset's metadata.

You can also see the contents of the dataset:
You can also see the contents of the dataset from the terminal.

```sh
cat2-client --username user@example.com --password foobar11 show @personal/ds-1d.b2nd
```

When the dataset is small, the contents are printed to the screen, otherwise a pager is used.

If you want to use a browser to view the contents of the dataset:
If you want to use a browser to view the contents of the dataset you can do that too. First authenticate yourself via the browser window accessible via the url associated with your server instance (probably something like `http://localhost:8000`). Then run the following command in the terminal to visualise the dataset directly.

```shell
cat2-client --username user@example.com --password foobar11 browse @personal/ds-1d.b2nd
cat2-client browse @personal/ds-1d.b2nd
```

Although you will need to authenticate with the server first; after that, this command will open a new tab in your default browser with the contents of the dataset.
You do need to authenticate with the server first; after that, this command will open a new tab in your default browser with the contents of the dataset.

There are more commands available in the `cat2-client` client; ask for help with:

Expand Down
218 changes: 109 additions & 109 deletions doc/tutorials/hdf5.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,109 +1,109 @@
(hdf5)=
# Working with HDF5 files
Caterva2 offers native support for working with HDF5 files. See [here](https://ironarray.wistia.com/medias/y64r14mojw) for a video demonstration. The notebook used is available on [github](https://github.com/ironArray/Caterva2/blob/main/examples/Video6-Cat2Cloud_and_HDF5.ipynb). In this tutorial, we will cover the same material: This tutorial demonstrates how to use the Cat2Cloud system with large HDF5 files. You'll learn how to upload, access, manipulate, and visualize HDF5 data both through proxy objects and Blosc2 arrays using the Caterva2 Python client.
## What is an HDF5 file?
HDF stands for Hierarchical Data Format. [HDF5](https://www.hdfgroup.org/solutions/hdf5/) is a file format and set of tools for managing complex data. It is designed to store and organize large amounts of heterogeneous data, enabling quick access and efficient storage, using compression for example. HDF5 files are made up of a root (`/`) which may then contain the file contents. These contents are organised using the two main structures of HDF5 files - groups (which are like subdirectories) and datasets. Both datasets and groups possess metadata, which may include special, bespoke information about the object in the form of attributes (`attrs`).
```{figure} images/hdf5-file.webp
---
class: with-border
---
Schematic of hdf5 file structure
```
## Loading a HDF5 file
HDF5 files are common in many scientific and industrial applications, and so examples abound online. We're going to use some example diffraction data from a synchrotron, provided by the [silx project](http://www.silx.org/). This may be downloaded locally (after importing necessary libraries - see the notebook mentioned at the beginning of this tutorial):
```
dir_path = "kevlar"
if not os.path.exists(f"{dir_path}.h5"):
response = requests.get("http://www.silx.
org/pub/pyFAI/pyFAI_UM_2020/data_ID13
/kevlar.h5")
with open(f"{dir_path}.h5", "wb") as file:
file.write(response.content)
```
## Unfolding the file
In order to handle the `.h5` file using Cat2Cloud, we must expose the hierarchical structure of the HDF5 file on the server, which in our case will be the `demo` server at https://cat2.cloud/demo. This is done by uploading the file to the Caterva2 server and then unfolding it, using a memory-light structure of subdirectories and proxy datasets.
```
url = "https://cat2.cloud/demo"
client = cat2.Client(url,
("user@example.com", "foobar11"))
myroot = client.get("@shared")
print(f"Before uploading and unfolding:
{myroot.file_list}")
local_address = f"{dir_path}.h5"
remote_address = myroot.name + "/" + local_address
apath = client.upload(local_address, remote_address)
bloscpath = client.unfold(apath)
```
By running the line ``print(f"After uploading and unfolding: {myroot.file_list}")``, one can check that the `.h5` indeed has been correctly exposed. Note that one may also use the `unfold` command in the prompt on the web client, applying it to the uploaded file:
```{image} images/hdf5-unfold.webp
:alt: Unfold command
:width: 49.5%
```
```{image} images/hdf5-unfold2.webp
:alt: Unfold result
:width: 49.5%
```
The unfolded file structure is clearly visible in the second image.
We may now perform operations on the `.b2nd` proxy.
## Examining the unfolded data
We can define a local reference which points to the data on the server (specifically the proxy), and we can obtain the easily obtain some of the data and plot it like so:
```
proxy = myroot["kevlar/entry/data/data.b2nd"]
cmap = plt.cm.viridis
example_image = proxy[5]
fig = plt.figure()
plt.imshow(example_image / 65535,
figure=fig, cmap=cmap, vmax=1, vmin=0)
```
```{figure} images/hdf5-output1.webp
---
class: with-border
---
First visualisation
```
As is clear, not much detail can be seen in the image; this is due to outlier pixels maxing out the image range. We can use a `lazyexpr` to apply a function to the data, which will be executed on the server side, and then we can visualise the result. The following code applies a conditional expression to the data, increasing the signal pixels.
```
image = client.lazyexpr("expr",
"where(a < 10, a * 32000, a)",
{"a": proxy.path})
example_image = client.get(image)[5] / 65535
fig = plt.figure()
plt.imshow(example_image, figure=fig,
cmap=cmap, vmax=1, vmin=0)
```
The result is an image with the desired diffraction pattern visible, as shown below:
```{figure} images/hdf5-output2.webp
---
class: with-border
---
Second visualisation
```
We can also go to the web client and directly visualize the lazy expression we have just generated and saved via the Tomography tab for the saved expression:
```{figure} images/hdf5-tomo.webp
---
class: with-border
---
Second visualisation
```
(hdf5)=
# Working with HDF5 files
Caterva2 offers native support for working with HDF5 files. See [here](https://ironarray.wistia.com/medias/y64r14mojw) for a video demonstration. The notebook used is available on [github](https://github.com/ironArray/Caterva2/blob/main/examples/Video6-Cat2Cloud_and_HDF5.ipynb). In this tutorial, we will cover the same material: This tutorial demonstrates how to use the Cat2Cloud system with large HDF5 files. You'll learn how to upload, access, manipulate, and visualize HDF5 data both through proxy objects and Blosc2 arrays using the Caterva2 Python client.

## What is an HDF5 file?
HDF stands for Hierarchical Data Format. [HDF5](https://www.hdfgroup.org/solutions/hdf5/) is a file format and set of tools for managing complex data. It is designed to store and organize large amounts of heterogeneous data, enabling quick access and efficient storage, using compression for example. HDF5 files are made up of a root (`/`) which may then contain the file contents. These contents are organised using the two main structures of HDF5 files - groups (which are like subdirectories) and datasets. Both datasets and groups possess metadata, which may include special, bespoke information about the object in the form of attributes (`attrs`).

```{figure} images/hdf5-file.webp
---
class: with-border

---

Schematic of hdf5 file structure
```


## Loading a HDF5 file
HDF5 files are common in many scientific and industrial applications, and so examples abound online. We're going to use some example diffraction data from a synchrotron, provided by the [silx project](http://www.silx.org/). This may be downloaded locally (after importing necessary libraries - see the notebook mentioned at the beginning of this tutorial):

```
dir_path = "kevlar"
if not os.path.exists(f"{dir_path}.h5"):
response = requests.get("http://www.silx.
org/pub/pyFAI/pyFAI_UM_2020/data_ID13
/kevlar.h5")
with open(f"{dir_path}.h5", "wb") as file:
file.write(response.content)
```

## Unfolding the file
In order to handle the `.h5` file using Cat2Cloud, we must expose the hierarchical structure of the HDF5 file on the server, which in our case will be the `demo` server at https://cat2.cloud/demo. This is done by uploading the file to the Caterva2 server and then unfolding it, using a memory-light structure of subdirectories and proxy datasets.

```
url = "https://cat2.cloud/demo"
client = cat2.Client(url,
("user@example.com", "foobar11"))
myroot = client.get("@shared")
print(f"Before uploading and unfolding:
{myroot.file_list}")
local_address = f"{dir_path}.h5"
remote_address = myroot.name + "/" + local_address
apath = client.upload(local_address, remote_address)
bloscpath = client.unfold(apath)
```

By running the line ``print(f"After uploading and unfolding: {myroot.file_list}")``, one can check that the `.h5` indeed has been correctly exposed. Note that one may also use the `unfold` command in the prompt on the web client, applying it to the uploaded file:
```{image} images/hdf5-unfold.webp
:alt: Unfold command
:width: 49.5%
```
```{image} images/hdf5-unfold2.webp
:alt: Unfold result
:width: 49.5%
```
The unfolded file structure is clearly visible in the second image.
We may now perform operations on the `.b2nd` proxy.

## Examining the unfolded data
We can define a local reference which points to the data on the server (specifically the proxy), and we can obtain the easily obtain some of the data and plot it like so:
```
proxy = myroot["kevlar/entry/data/data.b2nd"]
cmap = plt.cm.viridis
example_image = proxy[5]
fig = plt.figure()
plt.imshow(example_image / 65535,
figure=fig, cmap=cmap, vmax=1, vmin=0)
```

```{figure} images/hdf5-output1.webp
---
class: with-border

---

First visualisation
```

As is clear, not much detail can be seen in the image; this is due to outlier pixels maxing out the image range. We can use a `lazyexpr` to apply a function to the data, which will be executed on the server side, and then we can visualise the result. The following code applies a conditional expression to the data, increasing the signal pixels.

```
image = client.lazyexpr("expr",
"where(a < 10, a * 32000, a)",
{"a": proxy.path})
example_image = client.get(image)[5] / 65535
fig = plt.figure()
plt.imshow(example_image, figure=fig,
cmap=cmap, vmax=1, vmin=0)
```
The result is an image with the desired diffraction pattern visible, as shown below:

```{figure} images/hdf5-output2.webp
---
class: with-border

---

Second visualisation
```
We can also go to the web client and directly visualize the lazy expression we have just generated and saved via the Tomography tab for the saved expression:

```{figure} images/hdf5-tomo.webp
---
class: with-border

---

Second visualisation
```