diff --git a/doc/tutorials/API.md b/doc/tutorials/API.md index 17c7f691..ff428e11 100644 --- a/doc/tutorials/API.md +++ b/doc/tutorials/API.md @@ -1,4 +1,4 @@ -# Using the client APIs +# Using the Python client APIs To follow these instructions, make sure that you have installed ``caterva2`` and ``blosc2``. @@ -17,7 +17,7 @@ client = cat2.Client("https://cat2.cloud/demo") client.get_roots() # {'@public': {'name': '@public'}} ``` -or, if one has created a user, using the user credentials: +or - if one has created a user - using the user credentials: ```python client = cat2.Client("https://cat2.cloud/demo", ("user@example.com", "password1")) diff --git a/doc/tutorials/cli.md b/doc/tutorials/cli.md index 9c89c314..e299e1d5 100644 --- a/doc/tutorials/cli.md +++ b/doc/tutorials/cli.md @@ -13,6 +13,11 @@ To create a user, you can use the `cat2-admin adduser` command. For example: cat2-admin adduser user@example.com foobar11 ``` +To start the server use the following command: +```sh +CATERVA2_SECRET=c2sikrit cat2-server & +``` + Now that the services are running, we can use the `cat2-client` client to talk to the server. In another shell, let's list all the available roots in the system: @@ -26,17 +31,22 @@ cat2-client --user "user@example.com" --pass "foobar11" roots @shared ``` -First let's upload a file from the `root-example`folder to the `@personal` root: +First let's generate a file and save it locally. Run the following script in the terminal to save a `b2nd` file to your current directory. +``` +python -c "import blosc2; blosc2.arange(0, 1000, 1, blocks = (10,), chunks=(100,), urlpath='ds-1d.b2nd')" +``` + +Let's upload the file to the `@personal` root: ```sh -cat2-client --username user@example.com --password foobar11 upload root-example/ds-1d.b2nd @personal/ds-1d.b2nd +cat2-client --username user@example.com --password foobar11 upload ds-1d.b2nd @personal/ds-1d.b2nd ``` ``` Dataset stored in @personal/ds-1d.b2nd ``` -Now, one can list the datasets in the `@personal` root and see that the uploaded file appears +Now, one may list the datasets in the `@personal` root and see that the uploaded file appears ```sh cat2-client --username user@example.com --password foobar11 list @personal @@ -59,18 +69,18 @@ chunks: [100] blocks: [10] dtype : int64 nbytes: 7.81 KiB -cbytes: 4.90 KiB -ratio : 1.59x -mtime : 2025-10-08T11:09:03.955154Z +cbytes: 3.86 KiB +ratio : 2.02x +mtime : 2025-10-30T11:16:17.012415Z cparams: codec : ZSTD (5) - clevel : 1 + clevel : 5 filters: [SHUFFLE] ``` As you see, this command returns digested information of the dataset's metadata. -You can also see the contents of the dataset: +You can also see the contents of the dataset from the terminal. ```sh cat2-client --username user@example.com --password foobar11 show @personal/ds-1d.b2nd @@ -78,13 +88,13 @@ cat2-client --username user@example.com --password foobar11 show @personal/ds-1d When the dataset is small, the contents are printed to the screen, otherwise a pager is used. -If you want to use a browser to view the contents of the dataset: +If you want to use a browser to view the contents of the dataset you can do that too. First authenticate yourself via the browser window accessible via the url associated with your server instance (probably something like `http://localhost:8000`). Then run the following command in the terminal to visualise the dataset directly. ```shell -cat2-client --username user@example.com --password foobar11 browse @personal/ds-1d.b2nd +cat2-client browse @personal/ds-1d.b2nd ``` -Although you will need to authenticate with the server first; after that, this command will open a new tab in your default browser with the contents of the dataset. +You do need to authenticate with the server first; after that, this command will open a new tab in your default browser with the contents of the dataset. There are more commands available in the `cat2-client` client; ask for help with: diff --git a/doc/tutorials/hdf5.md b/doc/tutorials/hdf5.md index 73c69645..374fcfe9 100644 --- a/doc/tutorials/hdf5.md +++ b/doc/tutorials/hdf5.md @@ -1,109 +1,109 @@ -(hdf5)= -# Working with HDF5 files -Caterva2 offers native support for working with HDF5 files. See [here](https://ironarray.wistia.com/medias/y64r14mojw) for a video demonstration. The notebook used is available on [github](https://github.com/ironArray/Caterva2/blob/main/examples/Video6-Cat2Cloud_and_HDF5.ipynb). In this tutorial, we will cover the same material: This tutorial demonstrates how to use the Cat2Cloud system with large HDF5 files. You'll learn how to upload, access, manipulate, and visualize HDF5 data both through proxy objects and Blosc2 arrays using the Caterva2 Python client. - -## What is an HDF5 file? -HDF stands for Hierarchical Data Format. [HDF5](https://www.hdfgroup.org/solutions/hdf5/) is a file format and set of tools for managing complex data. It is designed to store and organize large amounts of heterogeneous data, enabling quick access and efficient storage, using compression for example. HDF5 files are made up of a root (`/`) which may then contain the file contents. These contents are organised using the two main structures of HDF5 files - groups (which are like subdirectories) and datasets. Both datasets and groups possess metadata, which may include special, bespoke information about the object in the form of attributes (`attrs`). - -```{figure} images/hdf5-file.webp ---- -class: with-border - ---- - -Schematic of hdf5 file structure -``` - - -## Loading a HDF5 file -HDF5 files are common in many scientific and industrial applications, and so examples abound online. We're going to use some example diffraction data from a synchrotron, provided by the [silx project](http://www.silx.org/). This may be downloaded locally (after importing necessary libraries - see the notebook mentioned at the beginning of this tutorial): - -``` -dir_path = "kevlar" -if not os.path.exists(f"{dir_path}.h5"): - response = requests.get("http://www.silx. - org/pub/pyFAI/pyFAI_UM_2020/data_ID13 - /kevlar.h5") - with open(f"{dir_path}.h5", "wb") as file: - file.write(response.content) -``` - -## Unfolding the file -In order to handle the `.h5` file using Cat2Cloud, we must expose the hierarchical structure of the HDF5 file on the server, which in our case will be the `demo` server at https://cat2.cloud/demo. This is done by uploading the file to the Caterva2 server and then unfolding it, using a memory-light structure of subdirectories and proxy datasets. - -``` -url = "https://cat2.cloud/demo" -client = cat2.Client(url, - ("user@example.com", "foobar11")) -myroot = client.get("@shared") -print(f"Before uploading and unfolding: - {myroot.file_list}") -local_address = f"{dir_path}.h5" -remote_address = myroot.name + "/" + local_address -apath = client.upload(local_address, remote_address) -bloscpath = client.unfold(apath) -``` - -By running the line ``print(f"After uploading and unfolding: {myroot.file_list}")``, one can check that the `.h5` indeed has been correctly exposed. Note that one may also use the `unfold` command in the prompt on the web client, applying it to the uploaded file: -```{image} images/hdf5-unfold.webp -:alt: Unfold command -:width: 49.5% -``` -```{image} images/hdf5-unfold2.webp -:alt: Unfold result -:width: 49.5% -``` -The unfolded file structure is clearly visible in the second image. -We may now perform operations on the `.b2nd` proxy. - -## Examining the unfolded data -We can define a local reference which points to the data on the server (specifically the proxy), and we can obtain the easily obtain some of the data and plot it like so: -``` -proxy = myroot["kevlar/entry/data/data.b2nd"] -cmap = plt.cm.viridis -example_image = proxy[5] -fig = plt.figure() -plt.imshow(example_image / 65535, - figure=fig, cmap=cmap, vmax=1, vmin=0) -``` - -```{figure} images/hdf5-output1.webp ---- -class: with-border - ---- - -First visualisation -``` - -As is clear, not much detail can be seen in the image; this is due to outlier pixels maxing out the image range. We can use a `lazyexpr` to apply a function to the data, which will be executed on the server side, and then we can visualise the result. The following code applies a conditional expression to the data, increasing the signal pixels. - -``` -image = client.lazyexpr("expr", - "where(a < 10, a * 32000, a)", - {"a": proxy.path}) -example_image = client.get(image)[5] / 65535 -fig = plt.figure() -plt.imshow(example_image, figure=fig, - cmap=cmap, vmax=1, vmin=0) -``` -The result is an image with the desired diffraction pattern visible, as shown below: - -```{figure} images/hdf5-output2.webp ---- -class: with-border - ---- - -Second visualisation -``` -We can also go to the web client and directly visualize the lazy expression we have just generated and saved via the Tomography tab for the saved expression: - -```{figure} images/hdf5-tomo.webp ---- -class: with-border - ---- - -Second visualisation -``` +(hdf5)= +# Working with HDF5 files +Caterva2 offers native support for working with HDF5 files. See [here](https://ironarray.wistia.com/medias/y64r14mojw) for a video demonstration. The notebook used is available on [github](https://github.com/ironArray/Caterva2/blob/main/examples/Video6-Cat2Cloud_and_HDF5.ipynb). In this tutorial, we will cover the same material: This tutorial demonstrates how to use the Cat2Cloud system with large HDF5 files. You'll learn how to upload, access, manipulate, and visualize HDF5 data both through proxy objects and Blosc2 arrays using the Caterva2 Python client. + +## What is an HDF5 file? +HDF stands for Hierarchical Data Format. [HDF5](https://www.hdfgroup.org/solutions/hdf5/) is a file format and set of tools for managing complex data. It is designed to store and organize large amounts of heterogeneous data, enabling quick access and efficient storage, using compression for example. HDF5 files are made up of a root (`/`) which may then contain the file contents. These contents are organised using the two main structures of HDF5 files - groups (which are like subdirectories) and datasets. Both datasets and groups possess metadata, which may include special, bespoke information about the object in the form of attributes (`attrs`). + +```{figure} images/hdf5-file.webp +--- +class: with-border + +--- + +Schematic of hdf5 file structure +``` + + +## Loading a HDF5 file +HDF5 files are common in many scientific and industrial applications, and so examples abound online. We're going to use some example diffraction data from a synchrotron, provided by the [silx project](http://www.silx.org/). This may be downloaded locally (after importing necessary libraries - see the notebook mentioned at the beginning of this tutorial): + +``` +dir_path = "kevlar" +if not os.path.exists(f"{dir_path}.h5"): + response = requests.get("http://www.silx. + org/pub/pyFAI/pyFAI_UM_2020/data_ID13 + /kevlar.h5") + with open(f"{dir_path}.h5", "wb") as file: + file.write(response.content) +``` + +## Unfolding the file +In order to handle the `.h5` file using Cat2Cloud, we must expose the hierarchical structure of the HDF5 file on the server, which in our case will be the `demo` server at https://cat2.cloud/demo. This is done by uploading the file to the Caterva2 server and then unfolding it, using a memory-light structure of subdirectories and proxy datasets. + +``` +url = "https://cat2.cloud/demo" +client = cat2.Client(url, + ("user@example.com", "foobar11")) +myroot = client.get("@shared") +print(f"Before uploading and unfolding: + {myroot.file_list}") +local_address = f"{dir_path}.h5" +remote_address = myroot.name + "/" + local_address +apath = client.upload(local_address, remote_address) +bloscpath = client.unfold(apath) +``` + +By running the line ``print(f"After uploading and unfolding: {myroot.file_list}")``, one can check that the `.h5` indeed has been correctly exposed. Note that one may also use the `unfold` command in the prompt on the web client, applying it to the uploaded file: +```{image} images/hdf5-unfold.webp +:alt: Unfold command +:width: 49.5% +``` +```{image} images/hdf5-unfold2.webp +:alt: Unfold result +:width: 49.5% +``` +The unfolded file structure is clearly visible in the second image. +We may now perform operations on the `.b2nd` proxy. + +## Examining the unfolded data +We can define a local reference which points to the data on the server (specifically the proxy), and we can obtain the easily obtain some of the data and plot it like so: +``` +proxy = myroot["kevlar/entry/data/data.b2nd"] +cmap = plt.cm.viridis +example_image = proxy[5] +fig = plt.figure() +plt.imshow(example_image / 65535, + figure=fig, cmap=cmap, vmax=1, vmin=0) +``` + +```{figure} images/hdf5-output1.webp +--- +class: with-border + +--- + +First visualisation +``` + +As is clear, not much detail can be seen in the image; this is due to outlier pixels maxing out the image range. We can use a `lazyexpr` to apply a function to the data, which will be executed on the server side, and then we can visualise the result. The following code applies a conditional expression to the data, increasing the signal pixels. + +``` +image = client.lazyexpr("expr", + "where(a < 10, a * 32000, a)", + {"a": proxy.path}) +example_image = client.get(image)[5] / 65535 +fig = plt.figure() +plt.imshow(example_image, figure=fig, + cmap=cmap, vmax=1, vmin=0) +``` +The result is an image with the desired diffraction pattern visible, as shown below: + +```{figure} images/hdf5-output2.webp +--- +class: with-border + +--- + +Second visualisation +``` +We can also go to the web client and directly visualize the lazy expression we have just generated and saved via the Tomography tab for the saved expression: + +```{figure} images/hdf5-tomo.webp +--- +class: with-border + +--- + +Second visualisation +```