diff --git a/README-DEVELOPERS.md b/README-DEVELOPERS.md index 5a6e61c8..44861a39 100644 --- a/README-DEVELOPERS.md +++ b/README-DEVELOPERS.md @@ -12,7 +12,7 @@ As the config files are already there, this essentially boils down to: pre-commit install ``` -Also, for running the tests, one needs to run manually the subscriber. +Also, for running the tests, one needs to run manually the server. There is a `caterva2.tests.services` script that does this. ## Build CSS and JS diff --git a/README.md b/README.md index 4802b87f..94c4d6d5 100644 --- a/README.md +++ b/README.md @@ -64,15 +64,15 @@ You may install Caterva2 in several ways: python -m pip install -e . ``` -When a user uses a client (web GUI, REST API, Python API, or command line) to query datasets, the client will connect to a Caterva2 **subscriber** service, which -accesses the relevant datasets stored either locally or remotely. The subscriber services may be managed via the command line by installing the `caterva2` package with the `[subscriber]` extra feature (we also wish to use the command line client, so we will also install the `clients` extra too): +When a user uses a client (web GUI, REST API, Python API, or command line) to query datasets, the client will connect to a Caterva2 **server** service, which +accesses the relevant datasets stored either locally or remotely. The server services may be managed via the command line by installing the `caterva2` package with the `[server]` extra feature (we also wish to use the command line client, so we will also install the `clients` extra too): ```sh - python -m pip install caterva2 [subscriber, clients] + python -m pip install caterva2 [server, clients] ``` In general, if you intend to run Caterva2 services, client programs, or the test suite, you need to enable the proper extra features by appending `[feature1,feature2...]` to the last argument of `pip` commands above. The following extras are supported: -- `subscriber` for running the Caterva2 subscriber service +- `subscriber` for running the Caterva2 server service - `clients` to use Caterva2 client programs (command-line or terminal) - `blosc2-plugins` to enable extra Blosc2 features like Btune or JPEG 2000 support - `plugins` to enable web GUI features like the tomography display @@ -117,18 +117,18 @@ Now: - create a virtual environment and install Caterva2 with the `[subscriber,clients]` extras (see above). - copy the configuration file `caterva2.sample.toml` to `caterva2.toml`. -Subscribers (and clients, to a limited extent) may get their configuration from a `caterva2.toml` file at the current directory (or an alternative file given with the `--conf` option). +Servers (and clients, to a limited extent) may get their configuration from a `caterva2.toml` file at the current directory (or an alternative file given with the `--conf` option). See also [configuration.md](configuration.md) in Caterva2 tutorials. -Then run the subscriber: +Then run the server: ```sh -CATERVA2_SECRET=c2sikrit cat2sub & # subscriber +CATERVA2_SECRET=c2sikrit cat2sub & # server ``` The `CATERVA2_SECRET` environment variable is obligatory and is explained below in the following section. ### User authentication -The Caterva2 subscriber includes some support for authenticating users. To enable it, run the subscriber with the environment variable `CATERVA2_SECRET` set to some non-empty, secure string that will be used for various user management operations. Note that new accounts may be registered, but their addresses are not verified. Password recovery does not work either. +The Caterva2 server includes some support for authenticating users. To enable it, run the server with the environment variable `CATERVA2_SECRET` set to some non-empty, secure string that will be used for various user management operations. Note that new accounts may be registered, but their addresses are not verified. Password recovery does not work either. To create a user, you can use the `cat2adduser` command line client. For example: @@ -146,7 +146,7 @@ client = cat2.Client("https://cat2.cloud/demo", ('user@example.com', 'foobar11') ### The command line client Now that the services are running, we can use the `cat2cli` client to talk -to the subscriber. In another shell, let's list all the available roots in the system: +to the server. In another shell, let's list all the available roots in the system: ```sh cat2cli --user "user@example.com" --pass "foobar11" roots @@ -170,7 +170,7 @@ cat2cli --username user@example.com --password foobar11 list @personal >> ds-1d.b2nd ``` -Let's ask the subscriber for more info about the dataset: +Let's ask the server for more info about the dataset: ```sh cat2cli --username user@example.com --password foobar11 info @personal/ds-1d.b2nd diff --git a/SPECS.md b/SPECS.md index a119b14e..1b0c3c16 100644 --- a/SPECS.md +++ b/SPECS.md @@ -2,27 +2,24 @@ ## Introduction -This document describes the minimal specifications for the project. It is meant to describe the functionality of the client, whereas the implementation details of the publisher and subscriber are left to the developer (as long as the specification works). +This document describes the minimal specifications for the project. It is meant to describe the functionality of the client, whereas the implementation details of the client and server are left to the developer (as long as the specification works). ## Vocabulary - **Root**: The root of a group of datasets that are published together. It is identified by a name. -- **Dataset**: A dataset is a file that is published by the publisher. It is identified by a path. +- **Dataset**: A dataset is a file that is uploaded by the client. It is identified by a path. E.g. `foo/bar.b2nd` is a dataset path with root `foo`. -- **Broker**: The broker is the entity that manages the communication between publishers and subscribers. It is also responsible for keeping a list of roots available to subscribers. -- **Publisher**: The publisher is the entity that makes datasets available to subscribers. It is responsible for creating a root and adding datasets to it. -- **Subscriber**: The subscriber is the entity that follows changes in a root and allows the download of datasets from publishers. -- **Client**: The client is a subscriber consumer (e.g. a command line tool) for the user to access the datasets; it connects to a subscriber. +- **Server**: The server is the entity that follows changes in a root and allows the download of datasets from clients. +- **Client**: The client is a server consumer (e.g. a command line tool or a web interface) for the user to access the datasets; it connects to a server, and can add/download datasets (or slices) to it. ## Services -The three services (broker, publisher and subscriber) have a number of common options: +The two services (client and server) have a number of common options: - `--http`: the hostname and port that it listens, e.g. `localhost:8000` -- `--subscriber`: the base of URLs provided by the subscriber, if different from `http://:` (only for subscriber) +- `--server`: the base of URLs provided by the server, if different from `http://:` (only for server) - `--loglevel`: by default `warning` - `--statedir`: directory where to store the service state files (cache, logs, pid file, etc.) -- `--broker`: the hostname and port where the broker runs (only for publisher and subscriber) In production deployments it's recommended to use Systemd services. @@ -30,10 +27,10 @@ In production deployments it's recommended to use Systemd services. The client must implement the following commands: -- `roots`: List all the available roots in a broker. +- `roots`: List all the available roots in the server. - `subscribe `: Request access to the datasets in a root. - `list `: List all the available datasets in a root. Needs to be subscribed to the root. -- `url `: Subscriber URL from where a dataset can be downloaded. +- `url `: Server URL from where a dataset can be downloaded. - `info `: Get metadata about a dataset. - `show `: Show the data of a dataset. `slice` is optional. - `download `: Get the data of a dataset and save it to a local `output_dir` folder. @@ -43,7 +40,7 @@ The client must implement the following commands: There should be a configuration file (by default $CWD/caterva2.toml) where the configuration for each service is specified. For example: ``` -[subscriber] +[server] http = "localhost:8000" urlbase = "https://cat2.example.com" # e.g. served by reverse proxy statedir = "_caterva2/sub" @@ -53,46 +50,28 @@ loglevel = "warning" ## Client implementation -The client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a subscriber and sends commands to it. The subscriber must be running before the client is started. If the subscriber is not running, the client must print an error message and exit. The publisher is expected to be running before the subscriber is started; if not, the subscriber will only serve its cached data. +The client must be implemented in Python 3 (3.11 being the minimal supported version). It must be a library with a command line interface that connects to a server and sends commands to it. The server must be running before the client can be used. If the server is not running, the client must print an error message and exit. The client is expected to be running before the server is started; if not, the server will only serve its cached data. ### Command line interface -- When a `roots` command is issued, the client must send a request to the subscriber to list all the available roots. The subscriber will reply with a list of roots (if possible, with flags indicating if a root is subscribed). +- When a `roots` command is issued, the client must send a request to the server to list all the available roots. The server will reply with a list of roots (if possible, with flags indicating if a root is subscribed). -- When a `subscribe` command is issued, the client must send a request to the subscriber to subscribe to the given root. The subscriber will reply with a success or failure message. If successful, the subscriber must store the root metadata in its local cache. +- When a `subscribe` command is issued, the client must send a request to the server to subscribe to the given root. The server will reply with a success or failure message. If successful, the server must store the root metadata in its local cache. -- When a `list` command is issued, the client must send a request to the subscriber to list the datasets in the given root. The subscriber will reply with a list of datasets. +- When a `list` command is issued, the client must send a request to the server to list the datasets in the given root. The server will reply with a list of datasets. - When a `url` command is issued, the client must show the URL from where the given dataset may be downloaded. -- When an `info` command is issued, the client must send a request to the subscriber to get the metadata of the given dataset. The subscriber will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. +- When an `info` command is issued, the client must send a request to the server to get the metadata of the given dataset. The server will reply with the [metadata](#metadata). See below for the [metadata](#metadata) format. -- When a `show` command is issued, the client must send a request to the subscriber to retrieve the data of the given dataset. The subscriber will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). +- When a `show` command is issued, the client must send a request to the server to retrieve the data of the given dataset. The server will reply with the data. The format is inferred from the extension of the output file: `.b2nd` for Blosc2 NDim and `.b2frame` for Blosc2 frames; an n-dim NumPy array and a 1-dim NumPy array will be shown respectively. All other extensions will be delivered as a raw buffer (e.g. `foo/path/README.md` will be shown as text). -- When a `download` command is issued, the client must send a request to the subscriber to retrieve the data of the dataset. The subscriber will reply with the data and client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). +- When a `download` command is issued, the client must send a request to the server to retrieve the data of the dataset. The server will reply with the data and client should be responsible to store it in its local `` folder. The name of the file will be the same as the dataset path (e.g. `foo/bar.b2nd` will be stored as `/foo/bar.b2nd`). The sequence diagram below summarizes how different messages flow between the components of the system. ![Figure: Caterva2 sequence diagram](./doc/_static/caterva2-seq.svg) -## Cache management details - -Whenever the subscriber gets a request to `subscribe` to a root, it must check if metadata (not the data itself) for all the datasets in a root is already in the cache. If it is, it must check if the root has changed in the publisher. If it has, it must update the cache. If it hasn't, it must use the cached data. If the root metadata is not in the cache, it must fetch it and add it to the cache. - -Metadata can be fetched and consolidated as uninitialized datasets in cache by using the API described in the [Metadata](#metadata) section below. - -There will be not an in-memory cache in the subscriber, but a folder in the filesystem. The reason is that cache files that are accessed frequently will be cached automatically by the OS, so there is no need to duplicate it (at least initially). The folder will be called `$(cwd)/_caterva2/cache/` and it will contain the metadata and data of the datasets. The data and metadata will be stored in Blosc2 format. - -Updates to the cache for a given root should happen in an atomic fashion. The subscriber should get and store all new required information in a temporary location of the same device where the current database and cache are located, and only once the storage operation is complete shall it replace the old data. - -When a publisher has to serve a data file that is not in Blosc2 format (e.g. a text file), it will be compressed locally (initially in one go with the technique shown in [section "Compressing general files"](#compressing-general-files)), and stored in `$(cwd)/_caterva2/cache/`. The file will be named `$(dataset_path).b2` (e.g. `foo/bar.txt` will be stored as `$(cwd)/_caterva2/cache/foo/bar.txt.b2`). - -The publisher will serve the data in its own cache as-is, without decompressing it. The subscriber will store and send the data as-is too, and only the client will be responsible to decompress it (it will receive a Blosc2 frame than can be opened with `blosc2.open()` and data can be retrieved using slicing). - -Whenever a `show` or `download` command is issued, the subscriber must check if the data in dataset is already in the cache. If it is, it must check if the dataset has changed in the publisher; for this, it will ask the publisher for the `mtime` in the dataset, and compare it against the `mtime` field in the general JSON database. If it has changed, it must update the cache. If it hasn't, it must use the cached data. If the data of the dataset is not in the cache, it must fetch it and add it to the cache. - -`show` or `download` commands will make the subscriber download the whole data from publisher and will store it in its cache folder. When a `slice` is provided (only for the `show` command), subscriber will download only the chunks in `[slice]` that are not in cache yet. - ## Metadata - `meta`: The metadata of the dataset. @@ -201,28 +180,9 @@ You can find an example of a data root in the `root-example` folder. It contain a = np.arange(120, dtype="complex128").reshape(2, 3, 4, 5) blosc2.asarray(a+a*1j, chunks=(1, 2, 3, 4), blocks=(1, 2, 2, 2), urlpath="dir2/ds-4d.b2nd", mode="w") -## Communication failures - -As we will be checking for the validity of the data in the cache (see above), we will be able to implement communication failure handling in a next version. For the time being, we will just assume that the communication is always successful, but we can start thinking on actions to be done when something fails. - -This is a list of possible actions: - -* When a subscriber sends a command to the publisher, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the subscriber will just serve its cached data to the client. In the latter case, if there is no cached data, it will return an error message instead. - -* As a particular case of the former, when the client requests data from the subscriber using `show` or `download`, the subscriber should first get and store locally all the data from the publisher needed to fulfill the client's request. If the transfer of any piece of data fails, the subscriber should return an error message before starting the transfer of data to the client. - -* When a client sends a command to the subscriber, it will wait for a reply. If the communication fails or a reply is not received in a certain amount of time, the client will print an error message and exit. - -* When a subscriber needs to update its database and cache for a given root, if the communication fails or a reply from the publisher is not received in a certain amount of time, or there is some other local problem (like lack of storage space), since the update should be atomic, the temporary data should be discarded and the cached one used according to the previous points. - -* When a publisher is down, and the root files are added/updated, when the publisher comes up again, it should announce the new/updated root files to the broker. The broker should then notify the subscribers that the root files have changed, and the subscribers should update their local database and cache. - - -TODO: think about other situations. - ## Data transmission -Whenever possible, data should be transmitted in [Blosc2 frame format](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst). +Whenever possible, data should be transmitted in [Blosc2 frame format](https://github.com/Blosc/c-blosc2/blob/main/README_CFRAME_FORMAT.rst). That is, when a dataset (or a slice of it) is requested, the server should send the data in Blosc2 frame format. The client should be able to read the data in this format return it to the user. As Blosc2 frames can be read as-is, there will be no penalty in de-serializing the data. ## Compressing general files @@ -245,19 +205,19 @@ Out[5]: True For the time being, `.b2` files can be made in one shot (i.e. a single `schunk.append_data()` call), but in a next version we should be able to compress files larger than available memory by using a chunked algorithm (i.e. reading and writing chunk-by-chunk). -## Internal database +## Internal database (TODO: this is obsolete) -There will be an internal database for publishers and subscribers for storing different metadata. It will be a JSON file called `$(cwd)/_caterva2/db.json` and it will contain the following fields (J. David: please check this): +There will be an internal database for clients and servers for storing different metadata. It will be a JSON file called `$(cwd)/_caterva2/db.json` and it will contain the following fields (J. David: please check this): * `version`: The version of the database. * `roots`: A list of roots. Each root is a dictionary with the following fields: * `name`: The name of the root. - * `url`: The publisher URL where the root is accessible (e.g. `http://localhost:5000/foo`). + * `url`: The client URL where the root is accessible (e.g. `http://localhost:5000/foo`). * `subscribed`: A boolean indicating if the root is subscribed. - * `mtime`: The modification time of the root in the publisher. + * `mtime`: The modification time of the root in the client. * `datasets`: A list of datasets. Each dataset is a dictionary with the following fields: * `path`: The path of the dataset. - * `mtime`: The modification time of the dataset in the publisher. + * `mtime`: The modification time of the dataset in the client. * `meta`: The metadata of the dataset. * `vlmeta`: The variable length metadata of the dataset. @@ -265,4 +225,6 @@ The ``meta`` and ``vlmeta`` fields above are the same as described in the [Metad ## TODO -- Broker: add API to remove a root (only the publisher that creates it can remove it) +- Revise the document for clarity and completeness. +- The original pubsub model may still live in parts if this document; try to remove it. +- Add examples of the client commands. diff --git a/caterva2.sample.toml b/caterva2.sample.toml index 3e7ae0c2..3e0f5914 100644 --- a/caterva2.sample.toml +++ b/caterva2.sample.toml @@ -1,13 +1,13 @@ -# Example configuration for a standalone subscriber +# Example configuration for a standalone server # -# It's possible to run only the subscriber. Then the configuration has only a -# section for the subscriber. And maybe another one for the client. +# It's possible to run only the server. Then the configuration has only a +# section for the server. And maybe another one for the client. -# The subscriber section must define: +# The server section must define: # -# - statedir: the directory where the subcriber's data will be stored (default: _caterva2/sub) -# - http: where the subscriber listens to (a unix socket or a host/port) (default: localhost:8000) -# - urlbase: the base url users will use to reach the subscriber (default: http://localhost:8000) +# - statedir: the directory where the server's data will be stored (default: _caterva2/sub) +# - http: where the server listens to (a unix socket or a host/port) (default: localhost:8000) +# - urlbase: the base url users will use to reach the server (default: http://localhost:8000) # - quota: if defined, it will limit the disk usage (default: 0, no limit) # - maxusers: if defined, it will limit the number of users (default: 0, no limit) # - login: if true, users will need to authenticate (default: true) @@ -23,7 +23,7 @@ maxusers = 5 register = true # allow users to register # The client section defines the credentials for the client to authenticate -# against the subscriber. +# against the server. [client] username = "" password = "" diff --git a/caterva2/client.py b/caterva2/client.py index 6f2f9969..8d2f7e5f 100644 --- a/caterva2/client.py +++ b/caterva2/client.py @@ -14,7 +14,7 @@ from . import api_utils, utils sub_urlbase_default = "http://localhost:8000" -"""The default base of URLs provided by the subscriber.""" +"""The default base of URLs provided by the server.""" def _format_paths(urlbase, path=None): @@ -760,7 +760,7 @@ def __init__(self, urlbase, auth=None, timeout=5): Parameters ---------- urlbase : str, optional - Base URL of the subscriber to query. Default to + Base URL of the server to query. Default to :py:obj:`caterva2.sub_urlbase_default`. auth : tuple, BasicAuth, optional @@ -1481,7 +1481,7 @@ def lazyexpr(self, name, expression, operands=None, compute=False): def adduser(self, newuser, password=None, superuser=False): """ - Adds a user to the subscriber. + Adds a user to the server. Parameters ---------- @@ -1517,7 +1517,7 @@ def adduser(self, newuser, password=None, superuser=False): def deluser(self, user): """ - Deletes a user from the subscriber. + Deletes a user from the server. Parameters ---------- @@ -1546,7 +1546,7 @@ def deluser(self, user): def listusers(self, username=None): """ - Lists the users in the subscriber. + Lists the users in the server. Parameters ---------- @@ -1556,7 +1556,7 @@ def listusers(self, username=None): Returns ------- list of dict - A list of user dictionaries in the subscriber. + A list of user dictionaries in the server. Examples -------- diff --git a/caterva2/clients/cli.py b/caterva2/clients/cli.py index cc06ca6d..dcc34087 100644 --- a/caterva2/clients/cli.py +++ b/caterva2/clients/cli.py @@ -489,7 +489,7 @@ def main(): subparser.set_defaults(func=cmd_move) # remove - help = "Remove a dataset from the subscriber." + help = "Remove a dataset from the server." subparser = subparsers.add_parser("remove", aliases=["rm"], help=help) subparser.add_argument("dataset", type=pathlib.Path) subparser.set_defaults(func=cmd_remove) @@ -538,7 +538,7 @@ def main(): subparser.set_defaults(func=cmd_download) # upload - help = "Upload a local dataset to subscriber." + help = "Upload a local dataset to server." subparser = subparsers.add_parser("upload", help=help) subparser.add_argument("localpath", type=pathlib.Path) subparser.add_argument("dataset", type=pathlib.Path) diff --git a/caterva2/services/settings.py b/caterva2/services/settings.py index 1792e77d..b6e98950 100644 --- a/caterva2/services/settings.py +++ b/caterva2/services/settings.py @@ -1,5 +1,5 @@ """ -Configuration for the subscriber only. +Configuration for the server only. TODO Move toml config here. """ diff --git a/caterva2/tests/services.py b/caterva2/tests/services.py index 57f492c5..92dad4b6 100644 --- a/caterva2/tests/services.py +++ b/caterva2/tests/services.py @@ -1,6 +1,6 @@ """Caterva2 services for tests. -This ensures that Caterva2 subscriber service is running before proceeding to tests. +This ensures that Caterva2 server service is running before proceeding to tests. It has three modes of operation: - Standalone script: when run as a script, it starts the services as children diff --git a/doc/tutorials/cli.md b/doc/tutorials/cli.md index 7424f4c2..811e0450 100644 --- a/doc/tutorials/cli.md +++ b/doc/tutorials/cli.md @@ -1,7 +1,7 @@ (Using-the-command-line-client)= # Using the command-line client -For quick queries to a subscriber or for use in shell scripts, Caterva2 ships the `cat2cli` program. To use it, you need to install Caterva2 with the `clients` extra, as well as `subscriber` in order to be able to query something. +For quick queries to a server or for use in shell scripts, Caterva2 ships the `cat2cli` program. To use it, you need to install Caterva2 with the `clients` extra, as well as `subscriber` in order to be able to query something. ```sh python -m pip install caterva2[clients,subscriber] @@ -14,7 +14,7 @@ cat2adduser user@example.com foobar11 ``` Now that the services are running, we can use the `cat2cli` client to talk -to the subscriber. In another shell, let's list all the available roots in the system: +to the server. In another shell, let's list all the available roots in the system: ```sh cat2cli --user "user@example.com" --pass "foobar11" roots @@ -38,7 +38,7 @@ cat2cli --username user@example.com --password foobar11 list @personal >> ds-1d.b2nd ``` -Let's ask the subscriber for more info about the dataset: +Let's ask the server for more info about the dataset: ```sh cat2cli --username user@example.com --password foobar11 info @personal/ds-1d.b2nd diff --git a/doc/tutorials/configuration.md b/doc/tutorials/configuration.md index a89ed1ca..9cd5f9cb 100644 --- a/doc/tutorials/configuration.md +++ b/doc/tutorials/configuration.md @@ -3,19 +3,19 @@ We've seen that the `cat2cli` program accepts some command-line options to tune its operation (check the `--help` option). This is even more important for services as we shall see in following sections. Thus, Caterva2 programs support getting some settings from a TOML configuration file, by default `caterva2.toml` in the current directory (though you may override it with the `--conf` option). -The configuration file may hold settings for different programs, with a separate section for each program. Thus, a program may check the file for its own settings, but also for those of other programs which may be of use to itself. This allows compact configurations in a single file. For instance, below is a sample configuration file for the subscriber program and some client app: +The configuration file may hold settings for different programs, with a separate section for each program. Thus, a program may check the file for its own settings, but also for those of other programs which may be of use to itself. This allows compact configurations in a single file. For instance, below is a sample configuration file for the server program and some client app: ```toml -# Example configuration for a standalone subscriber +# Example configuration for a standalone server # -# It's possible to run only the subscriber. Then the configuration has only a -# section for the subscriber. And maybe another one for the client. +# It's possible to run only the server. Then the configuration has only a +# section for the server. And maybe another one for the client. -# The subscriber section must define: +# The server section must define: # -# - statedir: the directory where the subcriber's data will be stored (default: _caterva2/sub) -# - http: where the subscriber listens to (a unix socket or a host/port) (default: localhost:8002) -# - urlbase: the base url users will use to reach the subscriber (default: http://localhost:8002) +# - statedir: the directory where the server's data will be stored (default: _caterva2/sub) +# - http: where the server listens to (a unix socket or a host/port) (default: localhost:8002) +# - urlbase: the base url users will use to reach the server (default: http://localhost:8002) # - quota: if defined, it will limit the disk usage (default: 0, no limit) # - maxusers: if defined, it will limit the number of users (default: 0, no limit) # - login: if true, users will need to authenticate (default: true) diff --git a/doc/tutorials/independent-services.md b/doc/tutorials/independent-services.md index 934d4852..c5f7fc46 100644 --- a/doc/tutorials/independent-services.md +++ b/doc/tutorials/independent-services.md @@ -5,12 +5,11 @@ The services that we used til now are enough for testing, but not for a real dep In this section we'll setup a more realistic deployment for a fictional organization: -- A broker at host `broker.example.org`. -- Two publishers at host `pub.lab.example.org` at a data collection laboratory, serving a different root each. -- A subscriber at host `sub.edu.example.org` at a research & education branch. +- Two clients at host `pub.lab.example.org` at a data collection laboratory, serving a different root each. +- A server at host `sub.edu.example.org` at a research & education branch. - A custom API client in a workstation at the latter branch. -The broker, publisher and subscriber hosts need a Caterva2 installation with the `services` extra: +The client and server hosts need a Caterva2 installation with the `services` extra: ```sh python -m pip install caterva2[services] @@ -24,39 +23,9 @@ python -m pip install caterva2[clients] (If you're going to try this tutorial on a single machine, just install `caterva2[services,clients]`.) -## Broker +## Clients -Our example broker shall listen on port 3104 of host `broker.example.org`. At that host, it may be run like this: - -```sh -cat2bro --http *:3104 -``` - -The broker will create a `_caterva2/bro` directory for its state files and listen in all network interfaces. Let's restrict that to just the public interface, and set the directory to `cat2-bro`. Stop the broker with Ctrl+C and run this (using the host name of your machine or `localhost`): - -```sh -cat2bro --http broker.example.org:3104 --statedir ./cat2-bro -``` - -(The ``./`` is not needed, but it shows that the `--statedir` option allows both relative and absolute paths, not necessarily under the current directory.) - -Let's put those options in the `caterva2.toml` configuration file: - -```toml -[broker] -http = "broker.example.org:3104" -statedir = "./cat2-bro" -``` - -You may now stop the broker and run it with just: - -```sh -cat2bro -``` - -## Publishers - -Here we will setup at the `pub.lab.example.org` host two publishers, each serving one of the roots which we shall name `foo` and `bar`. We'll create their respective Caterva2 directories with the (arbitrary but meaningful) names `foo-root` and `bar-root`, with simple text files inside: +Here we will setup at the `pub.lab.example.org` host two clients, each serving one of the roots which we shall name `foo` and `bar`. We'll create their respective Caterva2 directories with the (arbitrary but meaningful) names `foo-root` and `bar-root`, with simple text files inside: ```sh mkdir foo-root diff --git a/doc/tutorials/launching-services.md b/doc/tutorials/launching-services.md index e93cbdfc..8d04ec70 100644 --- a/doc/tutorials/launching-services.md +++ b/doc/tutorials/launching-services.md @@ -1,7 +1,7 @@ (Launching-Caterva2-services)= # Launching Caterva2 services -To do anything useful with Caterva2, you need at least a running broker, publisher (with some datasets) and subscriber. For the following tutorials we'll run our own services in the local machine, with the publisher serving some example datasets included in the Caterva2 package (the `root-example` directory). +To do anything useful with Caterva2, you need at least a running client (with some datasets) and server. For the following tutorials we'll run our own services in the local machine, with the client serving some example datasets included in the Caterva2 package (the `root-example` directory). First of all, you need to install Caterva2 with the `services` extra: @@ -15,9 +15,9 @@ The easiest way to run a set of services with example datasets is to launch the python -m caterva2.tests.services ``` -This will run a broker, a publisher and a subscriber listening for HTTP requests on `localhost:8000`, `localhost:8001` and `localhost:8002` respectively. They will put their private files under the `_caterva2` directory, respectively in `broker`, `publisher.foo` and `subscriber`. The publisher will be serving a root called `foo`, whose datasets (copied from `root-example`) sit in `_caterva2/data.foo`. You may want to browse that directory. +This will run a client and a server listening for HTTP requests on `localhost:8001` and `localhost:8002` respectively. They will put their private files under the `_caterva2` directory, respectively in `client.foo` and `server`. The client will be serving a root called `foo`, whose datasets (copied from `root-example`) sit in `_caterva2/data.foo`. You may want to browse that directory. -**Note:** You may provide command-line arguments to `caterva2.tests.services` to use a different state directory, root name and dataset source instead of `_caterva2` and `foo=root-example`. Multiple `ROOT_NAME=DATASET_SOURCE` arguments may be given, each to be served by a different publisher (run `caterva2.tests.services` with `--help` for more information). Moreover, you may customize some service settings (except those set by the previous arguments) via a `caterva2.toml` configuration file in the current directory (see [](caterva2.toml) and [](Running-independent-Caterva2-services) for more information). +**Note:** You may provide command-line arguments to `caterva2.tests.services` to use a different state directory, root name and dataset source instead of `_caterva2` and `foo=root-example`. Multiple `ROOT_NAME=DATASET_SOURCE` arguments may be given, each to be served by a different client (run `caterva2.tests.services` with `--help` for more information). Moreover, you may customize some service settings (except those set by the previous arguments) via a `caterva2.toml` configuration file in the current directory (see [](caterva2.toml) and [](Running-independent-Caterva2-services) for more information). **Note:** If you want to test user authentication, you may run the previous command with some secret value in the `CATERVA2_SECRET` environment variable, e.g. `env CATERVA2_SECRET=c2sikrit python -m caterva2.tests.services`. This will also create a sample user named `user@example.com` with password `foobar`. diff --git a/examples/query_server.py b/examples/query_server.py index e8b8b31f..17393750 100644 --- a/examples/query_server.py +++ b/examples/query_server.py @@ -10,7 +10,7 @@ import blosc2 -# Small example of how to query a dataset on a subscriber +# Small example of how to query a dataset on a server import httpx import numpy as np @@ -22,7 +22,7 @@ user_auth = None # Uncomment the following line and use your username and password -# if the subscriber requires authentication. +# if the server requires authentication. # user_auth = {'username': 'user@example.com', 'password': 'foobar11'} client = cat2.Client(URLBASE, auth=user_auth) @@ -50,7 +50,7 @@ mem_array = blosc2.ndarray_from_cframe(data.content) print(mem_array.info) -# 2. Fetch the data from the subscriber +# 2. Fetch the data from the server t0 = time() mem_array2 = array[:] t = time() - t0 diff --git a/pyproject.toml b/pyproject.toml index 1c6926d6..89abe98c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -77,7 +77,7 @@ dev = [ "pytest-cov", "ruff", ] -# Currently used to enable HDF5 support at the publisher. +# Currently used to enable HDF5 support at the client. hdf5 = [ "h5py", "b2h5py>=0.5.1",