From dc0ab5d3d7b74f496b75cc0177e34807a929b860 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 1 Oct 2025 13:50:25 +0200 Subject: [PATCH 1/4] First attempt at subscriber -> server and publisher -> client --- README-DEVELOPERS.md | 2 +- README.md | 20 ++++----- RELEASE_NOTES.md | 8 ++-- SPECS.md | 78 +++++++++++++++++----------------- caterva2.sample.toml | 16 +++---- caterva2/client.py | 12 +++--- caterva2/clients/cli.py | 4 +- caterva2/services/settings.py | 2 +- caterva2/tests/services.py | 2 +- doc/tutorials/cli.md | 6 +-- doc/tutorials/configuration.md | 16 +++---- examples/query_server.py | 6 +-- pyproject.toml | 2 +- 13 files changed, 87 insertions(+), 87 deletions(-) diff --git a/README-DEVELOPERS.md b/README-DEVELOPERS.md index 5a6e61c8..44861a39 100644 --- a/README-DEVELOPERS.md +++ b/README-DEVELOPERS.md @@ -12,7 +12,7 @@ As the config files are already there, this essentially boils down to: pre-commit install ``` -Also, for running the tests, one needs to run manually the subscriber. +Also, for running the tests, one needs to run manually the server. There is a `caterva2.tests.services` script that does this. ## Build CSS and JS diff --git a/README.md b/README.md index 4802b87f..94c4d6d5 100644 --- a/README.md +++ b/README.md @@ -64,15 +64,15 @@ You may install Caterva2 in several ways: python -m pip install -e . ``` -When a user uses a client (web GUI, REST API, Python API, or command line) to query datasets, the client will connect to a Caterva2 **subscriber** service, which -accesses the relevant datasets stored either locally or remotely. The subscriber services may be managed via the command line by installing the `caterva2` package with the `[subscriber]` extra feature (we also wish to use the command line client, so we will also install the `clients` extra too): +When a user uses a client (web GUI, REST API, Python API, or command line) to query datasets, the client will connect to a Caterva2 **server** service, which +accesses the relevant datasets stored either locally or remotely. The server services may be managed via the command line by installing the `caterva2` package with the `[server]` extra feature (we also wish to use the command line client, so we will also install the `clients` extra too): ```sh - python -m pip install caterva2 [subscriber, clients] + python -m pip install caterva2 [server, clients] ``` In general, if you intend to run Caterva2 services, client programs, or the test suite, you need to enable the proper extra features by appending `[feature1,feature2...]` to the last argument of `pip` commands above. The following extras are supported: -- `subscriber` for running the Caterva2 subscriber service +- `subscriber` for running the Caterva2 server service - `clients` to use Caterva2 client programs (command-line or terminal) - `blosc2-plugins` to enable extra Blosc2 features like Btune or JPEG 2000 support - `plugins` to enable web GUI features like the tomography display @@ -117,18 +117,18 @@ Now: - create a virtual environment and install Caterva2 with the `[subscriber,clients]` extras (see above). - copy the configuration file `caterva2.sample.toml` to `caterva2.toml`. -Subscribers (and clients, to a limited extent) may get their configuration from a `caterva2.toml` file at the current directory (or an alternative file given with the `--conf` option). +Servers (and clients, to a limited extent) may get their configuration from a `caterva2.toml` file at the current directory (or an alternative file given with the `--conf` option). See also [configuration.md](configuration.md) in Caterva2 tutorials. -Then run the subscriber: +Then run the server: ```sh -CATERVA2_SECRET=c2sikrit cat2sub & # subscriber +CATERVA2_SECRET=c2sikrit cat2sub & # server ``` The `CATERVA2_SECRET` environment variable is obligatory and is explained below in the following section. ### User authentication -The Caterva2 subscriber includes some support for authenticating users. To enable it, run the subscriber with the environment variable `CATERVA2_SECRET` set to some non-empty, secure string that will be used for various user management operations. Note that new accounts may be registered, but their addresses are not verified. Password recovery does not work either. +The Caterva2 server includes some support for authenticating users. To enable it, run the server with the environment variable `CATERVA2_SECRET` set to some non-empty, secure string that will be used for various user management operations. Note that new accounts may be registered, but their addresses are not verified. Password recovery does not work either. To create a user, you can use the `cat2adduser` command line client. For example: @@ -146,7 +146,7 @@ client = cat2.Client("https://cat2.cloud/demo", ('user@example.com', 'foobar11') ### The command line client Now that the services are running, we can use the `cat2cli` client to talk -to the subscriber. In another shell, let's list all the available roots in the system: +to the server. In another shell, let's list all the available roots in the system: ```sh cat2cli --user "user@example.com" --pass "foobar11" roots @@ -170,7 +170,7 @@ cat2cli --username user@example.com --password foobar11 list @personal >> ds-1d.b2nd ``` -Let's ask the subscriber for more info about the dataset: +Let's ask the server for more info about the dataset: ```sh cat2cli --username user@example.com --password foobar11 info @personal/ds-1d.b2nd diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index 83dfbd86..b8a373e9 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -158,15 +158,15 @@ * When visualizing an image, resize it to fit the window. ### Web backend -* Blosc2 Proxy class adoption. That allows the subscriber to fetch data from a publisher in chunks that are needed, instead of the whole dataset at once. This is useful for large datasets, where the subscriber may not have enough memory to hold the whole dataset. +* Blosc2 Proxy class adoption. That allows the server to fetch data from a client in chunks that are needed, instead of the whole dataset at once. This is useful for large datasets, where the server may not have enough memory to hold the whole dataset. * New @personal and @shared areas in the server, where users can create new datasets and upload files accessible to them and to a group of users, respectively. * New @public area in the server, where users can publish datasets that are accessible to everyone. -* Allow for the subscriber to run standalone, without a publisher. +* Allow for the server to run standalone, without a client. * New quota configuration in the server, to limit the amount of data that can be stored. * Support for Unix Domain Sockets. ### Client API -* New upload/remove/move/copy commands in the subscriber. +* New upload/remove/move/copy commands in the server. * New adduser/deluser/listusers APIs. * New c2context context manager to handle the connection to the server. * Support for new lazy expressions in Python-Blosc2 3.0 (with support for saving reductions). @@ -192,7 +192,7 @@ * Web client: New Download button to download a dataset as a file. * Web client: New Delete button to delete a dataset from @personal area. * Web client: Support to detect tomographies automatically using heuristics -- 3D integer datasets in greyscale and RGB(A) (using a 4th dim). -* Client API: Support for subscriber user authentication in client code. +* Client API: Support for server user authentication in client code. * Client API: Support for creating lazy expressions in the @personal area. The resulting data is not computed on creation, but on demand. diff --git a/SPECS.md b/SPECS.md index a119b14e..7b685714 100644 --- a/SPECS.md +++ b/SPECS.md @@ -2,38 +2,38 @@ ## Introduction -This document describes the minimal specifications for the project. It is meant to describe the functionality of the client, whereas the implementation details of the publisher and subscriber are left to the developer (as long as the specification works). +This document describes the minimal specifications for the project. It is meant to describe the functionality of the client, whereas the implementation details of the client and server are left to the developer (as long as the specification works). ## Vocabulary - **Root**: The root of a group of datasets that are published together. It is identified by a name. -- **Dataset**: A dataset is a file that is published by the publisher. It is identified by a path. +- **Dataset**: A dataset is a file that is published by the client. It is identified by a path. E.g. `foo/bar.b2nd` is a dataset path with root `foo`. -- **Broker**: The broker is the entity that manages the communication between publishers and subscribers. It is also responsible for keeping a list of roots available to subscribers. -- **Publisher**: The publisher is the entity that makes datasets available to subscribers. It is responsible for creating a root and adding datasets to it. -- **Subscriber**: The subscriber is the entity that follows changes in a root and allows the download of datasets from publishers. -- **Client**: The client is a subscriber consumer (e.g. a command line tool) for the user to access the datasets; it connects to a subscriber. +- **Broker**: The broker is the entity that manages the communication between clients and servers. It is also responsible for keeping a list of roots available to servers. +- **Client**: The client is the entity that makes datasets available to servers. It is responsible for creating a root and adding datasets to it. +- **Server**: The server is the entity that follows changes in a root and allows the download of datasets from clients. +- **User Client**: The user client is a server consumer (e.g. a command line tool) for the user to access the datasets; it connects to a server. ## Services -The three services (broker, publisher and subscriber) have a number of common options: +The three services (broker, client and server) have a number of common options: - `--http`: the hostname and port that it listens, e.g. `localhost:8000` -- `--subscriber`: the base of URLs provided by the subscriber, if different from `http://:` (only for subscriber) +- `--server`: the base of URLs provided by the server, if different from `http://:` (only for server) - `--loglevel`: by default `warning` - `--statedir`: directory where to store the service state files (cache, logs, pid file, etc.) -- `--broker`: the hostname and port where the broker runs (only for publisher and subscriber) +- `--broker`: the hostname and port where the broker runs (only for client and server) In production deployments it's recommended to use Systemd services. -## Client commands +## User Client commands -The client must implement the following commands: +The user client must implement the following commands: - `roots`: List all the available roots in a broker. - `subscribe `: Request access to the datasets in a root. - `list `: List all the available datasets in a root. Needs to be subscribed to the root. -- `url `: Subscriber URL from where a dataset can be downloaded. +- `url `: Server URL from where a dataset can be downloaded. - `info `: Get metadata about a dataset. - `show `: Show the data of a dataset. `slice` is optional. - `download `: Get the data of a dataset and save it to a local `output_dir` folder. @@ -43,7 +43,7 @@ The client must implement the following commands: There should be a configuration file (by default $CWD/caterva2.toml) where the configuration for each service is specified. For example: ``` -[subscriber] +[server] http = "localhost:8000" urlbase = "https://cat2.example.com" # e.g. served by reverse proxy statedir = "_caterva2/sub" @@ -51,25 +51,25 @@ loglevel = "warning" ``` -## Client implementation +## User Client implementation -The client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a subscriber and sends commands to it. The subscriber must be running before the client is started. If the subscriber is not running, the client must print an error message and exit. The publisher is expected to be running before the subscriber is started; if not, the subscriber will only serve its cached data. +The user client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a server and sends commands to it. The server must be running before the user client is started. If the server is not running, the user client must print an error message and exit. The client is expected to be running before the server is started; if not, the server will only serve its cached data. ### Command line interface -- When a `roots` command is issued, the client must send a request to the subscriber to list all the available roots. The subscriber will reply with a list of roots (if possible, with flags indicating if a root is subscribed). +- When a `roots` command is issued, the user client must send a request to the server to list all the available roots. The server will reply with a list of roots (if possible, with flags indicating if a root is subscribed). -- When a `subscribe` command is issued, the client must send a request to the subscriber to subscribe to the given root. The subscriber will reply with a success or failure message. If successful, the subscriber must store the root metadata in its local cache. +- When a `subscribe` command is issued, the user client must send a request to the server to subscribe to the given root. The server will reply with a success or failure message. If successful, the server must store the root metadata in its local cache. -- When a `list` command is issued, the client must send a request to the subscriber to list the datasets in the given root. The subscriber will reply with a list of datasets. +- When a `list` command is issued, the user client must send a request to the server to list the datasets in the given root. The server will reply with a list of datasets. -- When a `url` command is issued, the client must show the URL from where the given dataset may be downloaded. +- When a `url` command is issued, the user client must show the URL from where the given dataset may be downloaded. -- When an `info` command is issued, the client must send a request to the subscriber to get the metadata of the given dataset. The subscriber will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. +- When an `info` command is issued, the user client must send a request to the server to get the metadata of the given dataset. The server will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. -- When a `show` command is issued, the client must send a request to the subscriber to retrieve the data of the given dataset. The subscriber will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). +- When a `show` command is issued, the user client must send a request to the server to retrieve the data of the given dataset. The server will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). -- When a `download` command is issued, the client must send a request to the subscriber to retrieve the data of the dataset. The subscriber will reply with the data and client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). +- When a `download` command is issued, the user client must send a request to the server to retrieve the data of the dataset. The server will reply with the data and user client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). The sequence diagram below summarizes how different messages flow between the components of the system. @@ -77,21 +77,21 @@ The sequence diagram below summarizes how different messages flow between the co ## Cache management details -Whenever the subscriber gets a request to `subscribe` to a root, it must check if metadata (not the data itself) for all the datasets in a root is already in the cache. If it is, it must check if the root has changed in the publisher. If it has, it must update the cache. If it hasn't, it must use the cached data. If the root metadata is not in the cache, it must fetch it and add it to the cache. +Whenever the server gets a request to `subscribe` to a root, it must check if metadata (not the data itself) for all the datasets in a root is already in the cache. If it is, it must check if the root has changed in the client. If it has, it must update the cache. If it hasn't, it must use the cached data. If the root metadata is not in the cache, it must fetch it and add it to the cache. Metadata can be fetched and consolidated as uninitialized datasets in cache by using the API described in the [Metadata](#metadata) section below. -There will be not an in-memory cache in the subscriber, but a folder in the filesystem. The reason is that cache files that are accessed frequently will be cached automatically by the OS, so there is no need to duplicate it (at least initially). The folder will be called `$(cwd)/_caterva2/cache/` and it will contain the metadata and data of the datasets. The data and metadata will be stored in Blosc2 format. +There will be not an in-memory cache in the server, but a folder in the filesystem. The reason is that cache files that are accessed frequently will be cached automatically by the OS, so there is no need to duplicate it (at least initially). The folder will be called `$(cwd)/_caterva2/cache/` and it will contain the metadata and data of the datasets. The data and metadata will be stored in Blosc2 format. -Updates to the cache for a given root should happen in an atomic fashion. The subscriber should get and store all new required information in a temporary location of the same device where the current database and cache are located, and only once the storage operation is complete shall it replace the old data. +Updates to the cache for a given root should happen in an atomic fashion. The server should get and store all new required information in a temporary location of the same device where the current database and cache are located, and only once the storage operation is complete shall it replace the old data. -When a publisher has to serve a data file that is not in Blosc2 format (e.g. a text file), it will be compressed locally (initially in one go with the technique shown in [section "Compressing general files"](#compressing-general-files)), and stored in `$(cwd)/_caterva2/cache/`. The file will be named `$(dataset_path).b2` (e.g. `foo/bar.txt` will be stored as `$(cwd)/_caterva2/cache/foo/bar.txt.b2`). +When a client has to serve a data file that is not in Blosc2 format (e.g. a text file), it will be compressed locally (initially in one go with the technique shown in [section "Compressing general files"](#compressing-general-files)), and stored in `$(cwd)/_caterva2/cache/`. The file will be named `$(dataset_path).b2` (e.g. `foo/bar.txt` will be stored as `$(cwd)/_caterva2/cache/foo/bar.txt.b2`). -The publisher will serve the data in its own cache as-is, without decompressing it. The subscriber will store and send the data as-is too, and only the client will be responsible to decompress it (it will receive a Blosc2 frame than can be opened with `blosc2.open()` and data can be retrieved using slicing). +The client will serve the data in its own cache as-is, without decompressing it. The server will store and send the data as-is too, and only the user client will be responsible to decompress it (it will receive a Blosc2 frame than can be opened with `blosc2.open()` and data can be retrieved using slicing). -Whenever a `show` or `download` command is issued, the subscriber must check if the data in dataset is already in the cache. If it is, it must check if the dataset has changed in the publisher; for this, it will ask the publisher for the `mtime` in the dataset, and compare it against the `mtime` field in the general JSON database. If it has changed, it must update the cache. If it hasn't, it must use the cached data. If the data of the dataset is not in the cache, it must fetch it and add it to the cache. +Whenever a `show` or `download` command is issued, the server must check if the data in dataset is already in the cache. If it is, it must check if the dataset has changed in the client; for this, it will ask the client for the `mtime` in the dataset, and compare it against the `mtime` field in the general JSON database. If it has changed, it must update the cache. If it hasn't, it must use the cached data. If the data of the dataset is not in the cache, it must fetch it and add it to the cache. -`show` or `download` commands will make the subscriber download the whole data from publisher and will store it in its cache folder. When a `slice` is provided (only for the `show` command), subscriber will download only the chunks in `[slice]` that are not in cache yet. +`show` or `download` commands will make the server download the whole data from client and will store it in its cache folder. When a `slice` is provided (only for the `show` command), server will download only the chunks in `[slice]` that are not in cache yet. ## Metadata @@ -207,15 +207,15 @@ As we will be checking for the validity of the data in the cache (see above), we This is a list of possible actions: -* When a subscriber sends a command to the publisher, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the subscriber will just serve its cached data to the client. In the latter case, if there is no cached data, it will return an error message instead. +* When a server sends a command to the client, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the server will just serve its cached data to the user client. In the latter case, if there is no cached data, it will return an error message instead. -* As a particular case of the former, when the client requests data from the subscriber using `show` or `download`, the subscriber should first get and store locally all the data from the publisher needed to fulfill the client's request. If the transfer of any piece of data fails, the subscriber should return an error message before starting the transfer of data to the client. +* As a particular case of the former, when the user client requests data from the server using `show` or `download`, the server should first get and store locally all the data from the client needed to fulfill the user client's request. If the transfer of any piece of data fails, the server should return an error message before starting the transfer of data to the user client. -* When a client sends a command to the subscriber, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the client will print an error message and exit. +* When a user client sends a command to the server, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the user client will print an error message and exit. -* When a subscriber needs to update its database and cache for a given root, if the communication fails or a reply from the publisher is not received in a certain amount of time, or there is some other local problem (like lack of storage space), since the update should be atomic, the temporary data should be discarded and the cached one used according to the previous points. +* When a server needs to update its database and cache for a given root, if the communication fails or a reply from the client is not received in a certain amount of time, or there is some other local problem (like lack of storage space), since the update should be atomic, the temporary data should be discarded and the cached one used according to the previous points. -* When a publisher is down, and the root files are added/updated, when the publisher comes up again, it should announce the new/updated root files to the broker. The broker should then notify the subscribers that the root files have changed, and the subscribers should update their local database and cache. +* When a client is down, and the root files are added/updated, when the client comes up again, it should announce the new/updated root files to the broker. The broker should then notify the servers that the root files have changed, and the servers should update their local database and cache. TODO: think about other situations. @@ -247,17 +247,17 @@ For the time being, `.b2` files can be made in one shot (i.e. a single `schunk.a ## Internal database -There will be an internal database for publishers and subscribers for storing different metadata. It will be a JSON file called `$(cwd)/_caterva2/db.json` and it will contain the following fields (J. David: please check this): +There will be an internal database for clients and servers for storing different metadata. It will be a JSON file called `$(cwd)/_caterva2/db.json` and it will contain the following fields (J. David: please check this): * `version`: The version of the database. * `roots`: A list of roots. Each root is a dictionary with the following fields: * `name`: The name of the root. - * `url`: The publisher URL where the root is accessible (e.g. `http://localhost:5000/foo`). + * `url`: The client URL where the root is accessible (e.g. `http://localhost:5000/foo`). * `subscribed`: A boolean indicating if the root is subscribed. - * `mtime`: The modification time of the root in the publisher. + * `mtime`: The modification time of the root in the client. * `datasets`: A list of datasets. Each dataset is a dictionary with the following fields: * `path`: The path of the dataset. - * `mtime`: The modification time of the dataset in the publisher. + * `mtime`: The modification time of the dataset in the client. * `meta`: The metadata of the dataset. * `vlmeta`: The variable length metadata of the dataset. @@ -265,4 +265,4 @@ The ``meta`` and ``vlmeta`` fields above are the same as described in the [Metad ## TODO -- Broker: add API to remove a root (only the publisher that creates it can remove it) +- Broker: add API to remove a root (only the client that creates it can remove it) diff --git a/caterva2.sample.toml b/caterva2.sample.toml index 3e7ae0c2..3e0f5914 100644 --- a/caterva2.sample.toml +++ b/caterva2.sample.toml @@ -1,13 +1,13 @@ -# Example configuration for a standalone subscriber +# Example configuration for a standalone server # -# It's possible to run only the subscriber. Then the configuration has only a -# section for the subscriber. And maybe another one for the client. +# It's possible to run only the server. Then the configuration has only a +# section for the server. And maybe another one for the client. -# The subscriber section must define: +# The server section must define: # -# - statedir: the directory where the subcriber's data will be stored (default: _caterva2/sub) -# - http: where the subscriber listens to (a unix socket or a host/port) (default: localhost:8000) -# - urlbase: the base url users will use to reach the subscriber (default: http://localhost:8000) +# - statedir: the directory where the server's data will be stored (default: _caterva2/sub) +# - http: where the server listens to (a unix socket or a host/port) (default: localhost:8000) +# - urlbase: the base url users will use to reach the server (default: http://localhost:8000) # - quota: if defined, it will limit the disk usage (default: 0, no limit) # - maxusers: if defined, it will limit the number of users (default: 0, no limit) # - login: if true, users will need to authenticate (default: true) @@ -23,7 +23,7 @@ maxusers = 5 register = true # allow users to register # The client section defines the credentials for the client to authenticate -# against the subscriber. +# against the server. [client] username = "" password = "" diff --git a/caterva2/client.py b/caterva2/client.py index 6f2f9969..8d2f7e5f 100644 --- a/caterva2/client.py +++ b/caterva2/client.py @@ -14,7 +14,7 @@ from . import api_utils, utils sub_urlbase_default = "http://localhost:8000" -"""The default base of URLs provided by the subscriber.""" +"""The default base of URLs provided by the server.""" def _format_paths(urlbase, path=None): @@ -760,7 +760,7 @@ def __init__(self, urlbase, auth=None, timeout=5): Parameters ---------- urlbase : str, optional - Base URL of the subscriber to query. Default to + Base URL of the server to query. Default to :py:obj:`caterva2.sub_urlbase_default`. auth : tuple, BasicAuth, optional @@ -1481,7 +1481,7 @@ def lazyexpr(self, name, expression, operands=None, compute=False): def adduser(self, newuser, password=None, superuser=False): """ - Adds a user to the subscriber. + Adds a user to the server. Parameters ---------- @@ -1517,7 +1517,7 @@ def adduser(self, newuser, password=None, superuser=False): def deluser(self, user): """ - Deletes a user from the subscriber. + Deletes a user from the server. Parameters ---------- @@ -1546,7 +1546,7 @@ def deluser(self, user): def listusers(self, username=None): """ - Lists the users in the subscriber. + Lists the users in the server. Parameters ---------- @@ -1556,7 +1556,7 @@ def listusers(self, username=None): Returns ------- list of dict - A list of user dictionaries in the subscriber. + A list of user dictionaries in the server. Examples -------- diff --git a/caterva2/clients/cli.py b/caterva2/clients/cli.py index cc06ca6d..dcc34087 100644 --- a/caterva2/clients/cli.py +++ b/caterva2/clients/cli.py @@ -489,7 +489,7 @@ def main(): subparser.set_defaults(func=cmd_move) # remove - help = "Remove a dataset from the subscriber." + help = "Remove a dataset from the server." subparser = subparsers.add_parser("remove", aliases=["rm"], help=help) subparser.add_argument("dataset", type=pathlib.Path) subparser.set_defaults(func=cmd_remove) @@ -538,7 +538,7 @@ def main(): subparser.set_defaults(func=cmd_download) # upload - help = "Upload a local dataset to subscriber." + help = "Upload a local dataset to server." subparser = subparsers.add_parser("upload", help=help) subparser.add_argument("localpath", type=pathlib.Path) subparser.add_argument("dataset", type=pathlib.Path) diff --git a/caterva2/services/settings.py b/caterva2/services/settings.py index 1792e77d..b6e98950 100644 --- a/caterva2/services/settings.py +++ b/caterva2/services/settings.py @@ -1,5 +1,5 @@ """ -Configuration for the subscriber only. +Configuration for the server only. TODO Move toml config here. """ diff --git a/caterva2/tests/services.py b/caterva2/tests/services.py index 57f492c5..92dad4b6 100644 --- a/caterva2/tests/services.py +++ b/caterva2/tests/services.py @@ -1,6 +1,6 @@ """Caterva2 services for tests. -This ensures that Caterva2 subscriber service is running before proceeding to tests. +This ensures that Caterva2 server service is running before proceeding to tests. It has three modes of operation: - Standalone script: when run as a script, it starts the services as children diff --git a/doc/tutorials/cli.md b/doc/tutorials/cli.md index 7424f4c2..811e0450 100644 --- a/doc/tutorials/cli.md +++ b/doc/tutorials/cli.md @@ -1,7 +1,7 @@ (Using-the-command-line-client)= # Using the command-line client -For quick queries to a subscriber or for use in shell scripts, Caterva2 ships the `cat2cli` program. To use it, you need to install Caterva2 with the `clients` extra, as well as `subscriber` in order to be able to query something. +For quick queries to a server or for use in shell scripts, Caterva2 ships the `cat2cli` program. To use it, you need to install Caterva2 with the `clients` extra, as well as `subscriber` in order to be able to query something. ```sh python -m pip install caterva2[clients,subscriber] @@ -14,7 +14,7 @@ cat2adduser user@example.com foobar11 ``` Now that the services are running, we can use the `cat2cli` client to talk -to the subscriber. In another shell, let's list all the available roots in the system: +to the server. In another shell, let's list all the available roots in the system: ```sh cat2cli --user "user@example.com" --pass "foobar11" roots @@ -38,7 +38,7 @@ cat2cli --username user@example.com --password foobar11 list @personal >> ds-1d.b2nd ``` -Let's ask the subscriber for more info about the dataset: +Let's ask the server for more info about the dataset: ```sh cat2cli --username user@example.com --password foobar11 info @personal/ds-1d.b2nd diff --git a/doc/tutorials/configuration.md b/doc/tutorials/configuration.md index a89ed1ca..9cd5f9cb 100644 --- a/doc/tutorials/configuration.md +++ b/doc/tutorials/configuration.md @@ -3,19 +3,19 @@ We've seen that the `cat2cli` program accepts some command-line options to tune its operation (check the `--help` option). This is even more important for services as we shall see in following sections. Thus, Caterva2 programs support getting some settings from a TOML configuration file, by default `caterva2.toml` in the current directory (though you may override it with the `--conf` option). -The configuration file may hold settings for different programs, with a separate section for each program. Thus, a program may check the file for its own settings, but also for those of other programs which may be of use to itself. This allows compact configurations in a single file. For instance, below is a sample configuration file for the subscriber program and some client app: +The configuration file may hold settings for different programs, with a separate section for each program. Thus, a program may check the file for its own settings, but also for those of other programs which may be of use to itself. This allows compact configurations in a single file. For instance, below is a sample configuration file for the server program and some client app: ```toml -# Example configuration for a standalone subscriber +# Example configuration for a standalone server # -# It's possible to run only the subscriber. Then the configuration has only a -# section for the subscriber. And maybe another one for the client. +# It's possible to run only the server. Then the configuration has only a +# section for the server. And maybe another one for the client. -# The subscriber section must define: +# The server section must define: # -# - statedir: the directory where the subcriber's data will be stored (default: _caterva2/sub) -# - http: where the subscriber listens to (a unix socket or a host/port) (default: localhost:8002) -# - urlbase: the base url users will use to reach the subscriber (default: http://localhost:8002) +# - statedir: the directory where the server's data will be stored (default: _caterva2/sub) +# - http: where the server listens to (a unix socket or a host/port) (default: localhost:8002) +# - urlbase: the base url users will use to reach the server (default: http://localhost:8002) # - quota: if defined, it will limit the disk usage (default: 0, no limit) # - maxusers: if defined, it will limit the number of users (default: 0, no limit) # - login: if true, users will need to authenticate (default: true) diff --git a/examples/query_server.py b/examples/query_server.py index e8b8b31f..17393750 100644 --- a/examples/query_server.py +++ b/examples/query_server.py @@ -10,7 +10,7 @@ import blosc2 -# Small example of how to query a dataset on a subscriber +# Small example of how to query a dataset on a server import httpx import numpy as np @@ -22,7 +22,7 @@ user_auth = None # Uncomment the following line and use your username and password -# if the subscriber requires authentication. +# if the server requires authentication. # user_auth = {'username': 'user@example.com', 'password': 'foobar11'} client = cat2.Client(URLBASE, auth=user_auth) @@ -50,7 +50,7 @@ mem_array = blosc2.ndarray_from_cframe(data.content) print(mem_array.info) -# 2. Fetch the data from the subscriber +# 2. Fetch the data from the server t0 = time() mem_array2 = array[:] t = time() - t0 diff --git a/pyproject.toml b/pyproject.toml index 1c6926d6..89abe98c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,7 +77,7 @@ dev = [ "pytest-cov", "ruff", ] -# Currently used to enable HDF5 support at the publisher. +# Currently used to enable HDF5 support at the client. hdf5 = [ "h5py", "b2h5py>=0.5.1", From 35acea0db0c01fd54f5c17784386ea8e4443cf20 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 1 Oct 2025 14:13:49 +0200 Subject: [PATCH 2/4] Remove the references to the broker (only some remain in independent-services.md) --- SPECS.md | 10 +++---- doc/tutorials/independent-services.md | 41 ++++----------------------- doc/tutorials/launching-services.md | 6 ++-- 3 files changed, 12 insertions(+), 45 deletions(-) diff --git a/SPECS.md b/SPECS.md index 7b685714..20bab95f 100644 --- a/SPECS.md +++ b/SPECS.md @@ -9,20 +9,18 @@ This document describes the minimal specifications for the project. It is meant - **Root**: The root of a group of datasets that are published together. It is identified by a name. - **Dataset**: A dataset is a file that is published by the client. It is identified by a path. E.g. `foo/bar.b2nd` is a dataset path with root `foo`. -- **Broker**: The broker is the entity that manages the communication between clients and servers. It is also responsible for keeping a list of roots available to servers. - **Client**: The client is the entity that makes datasets available to servers. It is responsible for creating a root and adding datasets to it. - **Server**: The server is the entity that follows changes in a root and allows the download of datasets from clients. - **User Client**: The user client is a server consumer (e.g. a command line tool) for the user to access the datasets; it connects to a server. ## Services -The three services (broker, client and server) have a number of common options: +The two services (client and server) have a number of common options: - `--http`: the hostname and port that it listens, e.g. `localhost:8000` - `--server`: the base of URLs provided by the server, if different from `http://:` (only for server) - `--loglevel`: by default `warning` - `--statedir`: directory where to store the service state files (cache, logs, pid file, etc.) -- `--broker`: the hostname and port where the broker runs (only for client and server) In production deployments it's recommended to use Systemd services. @@ -30,7 +28,7 @@ In production deployments it's recommended to use Systemd services. The user client must implement the following commands: -- `roots`: List all the available roots in a broker. +- `roots`: List all the available roots in the server. - `subscribe `: Request access to the datasets in a root. - `list `: List all the available datasets in a root. Needs to be subscribed to the root. - `url `: Server URL from where a dataset can be downloaded. @@ -215,7 +213,7 @@ This is a list of possible actions: * When a server needs to update its database and cache for a given root, if the communication fails or a reply from the client is not received in a certain amount of time, or there is some other local problem (like lack of storage space), since the update should be atomic, the temporary data should be discarded and the cached one used according to the previous points. -* When a client is down, and the root files are added/updated, when the client comes up again, it should announce the new/updated root files to the broker. The broker should then notify the servers that the root files have changed, and the servers should update their local database and cache. +* When a client is down, and the root files are added/updated, when the client comes up again, it should announce the new/updated root files to the server. The server should then update its local database and cache for those roots. TODO: think about other situations. @@ -265,4 +263,4 @@ The ``meta`` and ``vlmeta`` fields above are the same as described in the [Metad ## TODO -- Broker: add API to remove a root (only the client that creates it can remove it) +- Add API to remove a root (only the client that creates it can remove it) diff --git a/doc/tutorials/independent-services.md b/doc/tutorials/independent-services.md index 934d4852..c5f7fc46 100644 --- a/doc/tutorials/independent-services.md +++ b/doc/tutorials/independent-services.md @@ -5,12 +5,11 @@ The services that we used til now are enough for testing, but not for a real dep In this section we'll setup a more realistic deployment for a fictional organization: -- A broker at host `broker.example.org`. -- Two publishers at host `pub.lab.example.org` at a data collection laboratory, serving a different root each. -- A subscriber at host `sub.edu.example.org` at a research & education branch. +- Two clients at host `pub.lab.example.org` at a data collection laboratory, serving a different root each. +- A server at host `sub.edu.example.org` at a research & education branch. - A custom API client in a workstation at the latter branch. -The broker, publisher and subscriber hosts need a Caterva2 installation with the `services` extra: +The client and server hosts need a Caterva2 installation with the `services` extra: ```sh python -m pip install caterva2[services] @@ -24,39 +23,9 @@ python -m pip install caterva2[clients] (If you're going to try this tutorial on a single machine, just install `caterva2[services,clients]`.) -## Broker +## Clients -Our example broker shall listen on port 3104 of host `broker.example.org`. At that host, it may be run like this: - -```sh -cat2bro --http *:3104 -``` - -The broker will create a `_caterva2/bro` directory for its state files and listen in all network interfaces. Let's restrict that to just the public interface, and set the directory to `cat2-bro`. Stop the broker with Ctrl+C and run this (using the host name of your machine or `localhost`): - -```sh -cat2bro --http broker.example.org:3104 --statedir ./cat2-bro -``` - -(The ``./`` is not needed, but it shows that the `--statedir` option allows both relative and absolute paths, not necessarily under the current directory.) - -Let's put those options in the `caterva2.toml` configuration file: - -```toml -[broker] -http = "broker.example.org:3104" -statedir = "./cat2-bro" -``` - -You may now stop the broker and run it with just: - -```sh -cat2bro -``` - -## Publishers - -Here we will setup at the `pub.lab.example.org` host two publishers, each serving one of the roots which we shall name `foo` and `bar`. We'll create their respective Caterva2 directories with the (arbitrary but meaningful) names `foo-root` and `bar-root`, with simple text files inside: +Here we will setup at the `pub.lab.example.org` host two clients, each serving one of the roots which we shall name `foo` and `bar`. We'll create their respective Caterva2 directories with the (arbitrary but meaningful) names `foo-root` and `bar-root`, with simple text files inside: ```sh mkdir foo-root diff --git a/doc/tutorials/launching-services.md b/doc/tutorials/launching-services.md index e93cbdfc..8d04ec70 100644 --- a/doc/tutorials/launching-services.md +++ b/doc/tutorials/launching-services.md @@ -1,7 +1,7 @@ (Launching-Caterva2-services)= # Launching Caterva2 services -To do anything useful with Caterva2, you need at least a running broker, publisher (with some datasets) and subscriber. For the following tutorials we'll run our own services in the local machine, with the publisher serving some example datasets included in the Caterva2 package (the `root-example` directory). +To do anything useful with Caterva2, you need at least a running client (with some datasets) and server. For the following tutorials we'll run our own services in the local machine, with the client serving some example datasets included in the Caterva2 package (the `root-example` directory). First of all, you need to install Caterva2 with the `services` extra: @@ -15,9 +15,9 @@ The easiest way to run a set of services with example datasets is to launch the python -m caterva2.tests.services ``` -This will run a broker, a publisher and a subscriber listening for HTTP requests on `localhost:8000`, `localhost:8001` and `localhost:8002` respectively. They will put their private files under the `_caterva2` directory, respectively in `broker`, `publisher.foo` and `subscriber`. The publisher will be serving a root called `foo`, whose datasets (copied from `root-example`) sit in `_caterva2/data.foo`. You may want to browse that directory. +This will run a client and a server listening for HTTP requests on `localhost:8001` and `localhost:8002` respectively. They will put their private files under the `_caterva2` directory, respectively in `client.foo` and `server`. The client will be serving a root called `foo`, whose datasets (copied from `root-example`) sit in `_caterva2/data.foo`. You may want to browse that directory. -**Note:** You may provide command-line arguments to `caterva2.tests.services` to use a different state directory, root name and dataset source instead of `_caterva2` and `foo=root-example`. Multiple `ROOT_NAME=DATASET_SOURCE` arguments may be given, each to be served by a different publisher (run `caterva2.tests.services` with `--help` for more information). Moreover, you may customize some service settings (except those set by the previous arguments) via a `caterva2.toml` configuration file in the current directory (see [](caterva2.toml) and [](Running-independent-Caterva2-services) for more information). +**Note:** You may provide command-line arguments to `caterva2.tests.services` to use a different state directory, root name and dataset source instead of `_caterva2` and `foo=root-example`. Multiple `ROOT_NAME=DATASET_SOURCE` arguments may be given, each to be served by a different client (run `caterva2.tests.services` with `--help` for more information). Moreover, you may customize some service settings (except those set by the previous arguments) via a `caterva2.toml` configuration file in the current directory (see [](caterva2.toml) and [](Running-independent-Caterva2-services) for more information). **Note:** If you want to test user authentication, you may run the previous command with some secret value in the `CATERVA2_SECRET` environment variable, e.g. `env CATERVA2_SECRET=c2sikrit python -m caterva2.tests.services`. This will also create a sample user named `user@example.com` with password `foobar`. From 30500aa804fe62e44e51f1ccff1861efc21e7adc Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 1 Oct 2025 14:26:25 +0200 Subject: [PATCH 3/4] Revert release notes to keep original pubsub parlance --- RELEASE_NOTES.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md index b8a373e9..83dfbd86 100644 --- a/RELEASE_NOTES.md +++ b/RELEASE_NOTES.md @@ -158,15 +158,15 @@ * When visualizing an image, resize it to fit the window. ### Web backend -* Blosc2 Proxy class adoption. That allows the server to fetch data from a client in chunks that are needed, instead of the whole dataset at once. This is useful for large datasets, where the server may not have enough memory to hold the whole dataset. +* Blosc2 Proxy class adoption. That allows the subscriber to fetch data from a publisher in chunks that are needed, instead of the whole dataset at once. This is useful for large datasets, where the subscriber may not have enough memory to hold the whole dataset. * New @personal and @shared areas in the server, where users can create new datasets and upload files accessible to them and to a group of users, respectively. * New @public area in the server, where users can publish datasets that are accessible to everyone. -* Allow for the server to run standalone, without a client. +* Allow for the subscriber to run standalone, without a publisher. * New quota configuration in the server, to limit the amount of data that can be stored. * Support for Unix Domain Sockets. ### Client API -* New upload/remove/move/copy commands in the server. +* New upload/remove/move/copy commands in the subscriber. * New adduser/deluser/listusers APIs. * New c2context context manager to handle the connection to the server. * Support for new lazy expressions in Python-Blosc2 3.0 (with support for saving reductions). @@ -192,7 +192,7 @@ * Web client: New Download button to download a dataset as a file. * Web client: New Delete button to delete a dataset from @personal area. * Web client: Support to detect tomographies automatically using heuristics -- 3D integer datasets in greyscale and RGB(A) (using a 4th dim). -* Client API: Support for server user authentication in client code. +* Client API: Support for subscriber user authentication in client code. * Client API: Support for creating lazy expressions in the @personal area. The resulting data is not computed on creation, but on demand. From 950611a0582c228996d485f2463b2c079d281508 Mon Sep 17 00:00:00 2001 From: Francesc Alted Date: Wed, 1 Oct 2025 17:24:50 +0200 Subject: [PATCH 4/4] Another update to the SPECS (not totally finished yet, but good to go) --- SPECS.md | 72 ++++++++++++++------------------------------------------ 1 file changed, 18 insertions(+), 54 deletions(-) diff --git a/SPECS.md b/SPECS.md index 20bab95f..1b0c3c16 100644 --- a/SPECS.md +++ b/SPECS.md @@ -7,11 +7,10 @@ This document describes the minimal specifications for the project. It is meant ## Vocabulary - **Root**: The root of a group of datasets that are published together. It is identified by a name. -- **Dataset**: A dataset is a file that is published by the client. It is identified by a path. +- **Dataset**: A dataset is a file that is uploaded by the client. It is identified by a path. E.g. `foo/bar.b2nd` is a dataset path with root `foo`. -- **Client**: The client is the entity that makes datasets available to servers. It is responsible for creating a root and adding datasets to it. - **Server**: The server is the entity that follows changes in a root and allows the download of datasets from clients. -- **User Client**: The user client is a server consumer (e.g. a command line tool) for the user to access the datasets; it connects to a server. +- **Client**: The client is a server consumer (e.g. a command line tool or a web interface) for the user to access the datasets; it connects to a server, and can add/download datasets (or slices) to it. ## Services @@ -24,9 +23,9 @@ The two services (client and server) have a number of common options: In production deployments it's recommended to use Systemd services. -## User Client commands +## Client commands -The user client must implement the following commands: +The client must implement the following commands: - `roots`: List all the available roots in the server. - `subscribe `: Request access to the datasets in a root. @@ -49,48 +48,30 @@ loglevel = "warning" ``` -## User Client implementation +## Client implementation -The user client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a server and sends commands to it. The server must be running before the user client is started. If the server is not running, the user client must print an error message and exit. The client is expected to be running before the server is started; if not, the server will only serve its cached data. +The client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a server and sends commands to it. The server must be running before the client can be used. If the server is not running, the client must print an error message and exit. The client is expected to be running before the server is started; if not, the server will only serve its cached data. ### Command line interface -- When a `roots` command is issued, the user client must send a request to the server to list all the available roots. The server will reply with a list of roots (if possible, with flags indicating if a root is subscribed). +- When a `roots` command is issued, the client must send a request to the server to list all the available roots. The server will reply with a list of roots (if possible, with flags indicating if a root is subscribed). -- When a `subscribe` command is issued, the user client must send a request to the server to subscribe to the given root. The server will reply with a success or failure message. If successful, the server must store the root metadata in its local cache. +- When a `subscribe` command is issued, the client must send a request to the server to subscribe to the given root. The server will reply with a success or failure message. If successful, the server must store the root metadata in its local cache. -- When a `list` command is issued, the user client must send a request to the server to list the datasets in the given root. The server will reply with a list of datasets. +- When a `list` command is issued, the client must send a request to the server to list the datasets in the given root. The server will reply with a list of datasets. -- When a `url` command is issued, the user client must show the URL from where the given dataset may be downloaded. +- When a `url` command is issued, the client must show the URL from where the given dataset may be downloaded. -- When an `info` command is issued, the user client must send a request to the server to get the metadata of the given dataset. The server will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. +- When an `info` command is issued, the client must send a request to the server to get the metadata of the given dataset. The server will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. -- When a `show` command is issued, the user client must send a request to the server to retrieve the data of the given dataset. The server will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). +- When a `show` command is issued, the client must send a request to the server to retrieve the data of the given dataset. The server will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). -- When a `download` command is issued, the user client must send a request to the server to retrieve the data of the dataset. The server will reply with the data and user client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). +- When a `download` command is issued, the client must send a request to the server to retrieve the data of the dataset. The server will reply with the data and client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). The sequence diagram below summarizes how different messages flow between the components of the system. ![Figure: Caterva2 sequence diagram](./doc/_static/caterva2-seq.svg) -## Cache management details - -Whenever the server gets a request to `subscribe` to a root, it must check if metadata (not the data itself) for all the datasets in a root is already in the cache. If it is, it must check if the root has changed in the client. If it has, it must update the cache. If it hasn't, it must use the cached data. If the root metadata is not in the cache, it must fetch it and add it to the cache. - -Metadata can be fetched and consolidated as uninitialized datasets in cache by using the API described in the [Metadata](#metadata) section below. - -There will be not an in-memory cache in the server, but a folder in the filesystem. The reason is that cache files that are accessed frequently will be cached automatically by the OS, so there is no need to duplicate it (at least initially). The folder will be called `$(cwd)/_caterva2/cache/` and it will contain the metadata and data of the datasets. The data and metadata will be stored in Blosc2 format. - -Updates to the cache for a given root should happen in an atomic fashion. The server should get and store all new required information in a temporary location of the same device where the current database and cache are located, and only once the storage operation is complete shall it replace the old data. - -When a client has to serve a data file that is not in Blosc2 format (e.g. a text file), it will be compressed locally (initially in one go with the technique shown in [section "Compressing general files"](#compressing-general-files)), and stored in `$(cwd)/_caterva2/cache/`. The file will be named `$(dataset_path).b2` (e.g. `foo/bar.txt` will be stored as `$(cwd)/_caterva2/cache/foo/bar.txt.b2`). - -The client will serve the data in its own cache as-is, without decompressing it. The server will store and send the data as-is too, and only the user client will be responsible to decompress it (it will receive a Blosc2 frame than can be opened with `blosc2.open()` and data can be retrieved using slicing). - -Whenever a `show` or `download` command is issued, the server must check if the data in dataset is already in the cache. If it is, it must check if the dataset has changed in the client; for this, it will ask the client for the `mtime` in the dataset, and compare it against the `mtime` field in the general JSON database. If it has changed, it must update the cache. If it hasn't, it must use the cached data. If the data of the dataset is not in the cache, it must fetch it and add it to the cache. - -`show` or `download` commands will make the server download the whole data from client and will store it in its cache folder. When a `slice` is provided (only for the `show` command), server will download only the chunks in `[slice]` that are not in cache yet. - ## Metadata - `meta`: The metadata of the dataset. @@ -199,28 +180,9 @@ You can find an example of a data root in the `root-example` folder. It contain a = np.arange(120, dtype="complex128").reshape(2, 3, 4, 5) blosc2.asarray(a+a*1j, chunks=(1, 2, 3, 4), blocks=(1, 2, 2, 2), urlpath="dir2/ds-4d.b2nd", mode="w") -## Communication failures - -As we will be checking for the validity of the data in the cache (see above), we will be able to implement communication failure handling in a next version. For the time being, we will just assume that the communication is always successful, but we can start thinking on actions to be done when something fails. - -This is a list of possible actions: - -* When a server sends a command to the client, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the server will just serve its cached data to the user client. In the latter case, if there is no cached data, it will return an error message instead. - -* As a particular case of the former, when the user client requests data from the server using `show` or `download`, the server should first get and store locally all the data from the client needed to fulfill the user client's request. If the transfer of any piece of data fails, the server should return an error message before starting the transfer of data to the user client. - -* When a user client sends a command to the server, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the user client will print an error message and exit. - -* When a server needs to update its database and cache for a given root, if the communication fails or a reply from the client is not received in a certain amount of time, or there is some other local problem (like lack of storage space), since the update should be atomic, the temporary data should be discarded and the cached one used according to the previous points. - -* When a client is down, and the root files are added/updated, when the client comes up again, it should announce the new/updated root files to the server. The server should then update its local database and cache for those roots. - - -TODO: think about other situations. - ## Data transmission -Whenever possible, data should be transmitted in [Blosc2 frame format](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst). +Whenever possible, data should be transmitted in [Blosc2 frame format](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst). That is, when a dataset (or a slice of it) is requested, the server should send the data in Blosc2 frame format. The client should be able to read the data in this format return it to the user. As Blosc2 frames can be read as-is, there will be no penalty in de-serializing the data. ## Compressing general files @@ -243,7 +205,7 @@ Out[5]: True For the time being, `.b2` files can be made in one shot (i.e. a single `schunk.append_data()` call), but in a next version we should be able to compress files larger than available memory by using a chunked algorithm (i.e. reading and writing chunk-by-chunk). -## Internal database +## Internal database (TODO: this is obsolete) There will be an internal database for clients and servers for storing different metadata. It will be a JSON file called `$(cwd)/_caterva2/db.json` and it will contain the following fields (J. David: please check this): @@ -263,4 +225,6 @@ The ``meta`` and ``vlmeta`` fields above are the same as described in the [Metad ## TODO -- Add API to remove a root (only the client that creates it can remove it) +- Revise the document for clarity and completeness. +- The original pubsub model may still live in parts if this document; try to remove it. +- Add examples of the client commands.