diff --git a/.gitignore b/.gitignore index 76c1974f..3e31ec57 100644 --- a/.gitignore +++ b/.gitignore @@ -8,3 +8,4 @@ auto-save-list tramp .\#* generate/generator/generator +.DS_Store diff --git a/enterprise/enterprise-analytics/2.2.0/README.md b/enterprise/enterprise-analytics/2.2.0/README.md index d6ab9468..3e35593a 100644 --- a/enterprise/enterprise-analytics/2.2.0/README.md +++ b/enterprise/enterprise-analytics/2.2.0/README.md @@ -12,9 +12,14 @@ Traditionally, analyzing JSON data in NoSQL databases requires complex transform * An enhanced MPP-based query engine enables scalable, real-time analytical query computation. * A cost-based optimizer improves query execution without requiring user intervention. Using a sample-based approach, it quickly estimates data statistics from a small subset of the data, enabling it to identify the lowest-cost query plan without scanning the entire dataset. * Zero ETL for incoming data, with real-time ingestion capabilities powered by Confluent Kafka, that provide the ability to connect, capture, and extract data from nearly any database or application. One can optionally modify the target JSON structure of the incoming data while in transit, for example, to omit or modify its fields. -* Data Lakehouse capabilities that enable direct querying from Amazon S3 and S3-compatible storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, and Delta tables, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* Data Lakehouse capabilities that enable direct querying from supported object storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, Delta tables, and Apache Iceberg tables via Iceberg catalogs, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* A choice of certified object stores for the underlying storage layer: Amazon S3, S3-compatible storage, [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) (2.1 and later) and [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html) (2.2 and later). +* Continuous, row-level change data capture from Oracle and SQL Server, powered by Kafka links and Debezium connectors (2.2 and later). * A SQL++ based path for writing the results of a query back to the Couchbase Operational data service to support adaptive applications. * A tabular view facility that provides native SQL-based support for Tableau, PowerBI and Apache Superset for building business reports, visualizations, and dashboards. +* An Index Advisor that recommends secondary indexes for your queries, plus index-only query plans that serve requests entirely from covered indexes (2.2 and later). +* JWT authentication for REST API and query requests, including tokens issued by external identity providers (2.2 and later). +* An asynchronous REST API for submitting long-running queries without blocking application workflows (2.2 and later). This Docker image is designed to make it easy to run Enterprise Analytics for development, testing, and proof-of-concept environments. @@ -25,8 +30,10 @@ To quickly get started with Enterprise Analytics, you can run an instance using ### Prerequisites These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No services running on ports `8091` or `8095` 1. No existing containers named `ea` (or `s3mock` if using S3Mock) +1. No existing Docker network named `ea-net` ### 1. Create a Docker network Create a user-defined network so the container can communicate with other services if needed. @@ -36,23 +43,29 @@ docker network create ea-net ``` ### 2. Configure S3Mock (optional) -If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use a different blob storage backend. +If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use one of the other supported blob storage backends — Amazon S3, S3-compatible storage, Azure Blob Storage, or Google Cloud Storage. See [Supported Object Storage Solutions](https://docs.couchbase.com/enterprise-analytics/current/install/supported-platform.html#supported-object-storage-solutions). ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the Enterprise Analytics container Run the Enterprise Analytics container with host and port mappings for the Couchbase Web Console and Enterprise Analytics service, exposed on ports `8091` and `8095` on the host. ```bash -docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` +To connect to this container from an SDK, also publish the Data Service ports, e.g. `-p 18091:18091 -p 18095:18095 -p 11207:11207 -p 11210:11210`. See "Ports" below. + ### 4. Initialize the cluster Next, visit http://localhost:8091 on the host machine to see the Web Console to start Enterprise Analytics setup. +> **Note:** The storage scheme and the number of storage partitions are fixed at cluster setup and cannot be changed afterwards. The remaining blob storage settings — bucket, region, path prefix, endpoint, authentication and TLS options — can be changed later under *Settings > Cluster*. + ![Setup splash screen](https://d774lla4im6mk.cloudfront.net/ea/setup-initial.png) Walk through the Setup wizard @@ -74,9 +87,9 @@ After completing the setup, the console will load. ![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/install-samples.png) ### 6. Execute a sample query -You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airports in the `travel-sample`. +You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airlines in the `travel-sample`. -![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) +![Sample query](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) ### 7. Next steps @@ -94,7 +107,9 @@ The following example shows how to start a two-node Enterprise Analytics cluster These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No existing containers named `s3mock`, `ea1`, or `ea2` +1. No existing Docker network named `ea-net` 1. No services running on ports `8091`, `8095`, `9091`, or `9095` ### 1. Create a Docker network @@ -110,7 +125,9 @@ docker network create ea-net Start the S3Mock container with an initial bucket called `ea-storage`. ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the first Enterprise Analytics node @@ -119,7 +136,7 @@ Run the first node (`ea1`) with host and port mappings for the Couchbase Web Con ```bash docker run -d --name ea1 --network ea-net --hostname ea1.example.com --network-alias ea1.example.com \ - -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 + -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` ### 4. Start the second Enterprise Analytics node @@ -128,7 +145,7 @@ Run the second node (`ea2`) with its own mapped ports so you can access it separ ```bash docker run -d --name ea2 --network ea-net --hostname ea2.example.com --network-alias ea2.example.com \ - -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics:2.0.0 + -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics ``` ### 5. Wait for the nodes to be ready @@ -172,9 +189,16 @@ docker exec ea1 couchbase-cli setting-enterprise-analytics --cluster http://loca --region us-east-1 \ --endpoint http://s3mock:9090 \ --anonymous-auth 1 \ - --path-style-addressing 1 + --path-style-addressing 1 \ + --partitions 16 ``` +`--partitions 16` keeps the number of storage partitions low; the default of 128 adds unnecessary overhead for a small development cluster. + +> **Note:** The storage scheme (`--scheme`) and the number of storage partitions (`--partitions`) are fixed once the cluster is initialized, so this step must come before `cluster-init` below. The remaining settings can be changed later with `--set`, or under *Settings > Cluster* in the Web Console. + +To use a backend other than S3 or S3-compatible storage, set `--scheme azblob` for [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) or `--scheme gs` for [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html). + ### 8. Initialize the cluster Initialize the Enterprise Analytics cluster. @@ -212,25 +236,68 @@ Once rebalanced, the cluster is ready to be used. Access the UI at: - **ea1:** [http://localhost:8091](http://localhost:8091) - **ea2:** [http://localhost:9091](http://localhost:9091) +## Container Requirements + +| Requirement | Details | +|-------------|---------| +| CPU | 2 GHz dual core x86_64 **supporting AVX2 or later**, or 2 GHz dual core 64-bit ARM v8. The container aborts at startup if the host CPU does not meet the required microarchitecture. | +| RAM | 4 GB minimum; 32 GB or more recommended | +| Disk | 8 GB minimum, block-based (HDD, SSD, EBS, iSCSI). Network file systems such as CIFS and NFS are not supported. | + +For development and testing you can go below these minimums — as little as 1 GB of free RAM and a single core — but production deployments must meet them. See [System Resource Requirements](https://docs.couchbase.com/enterprise-analytics/current/install/sys-resource-req.html). + +The image is published for both `linux/amd64` and `linux/arm64`. + +For production deployments, raise the container's ulimits: + +```bash +docker run -d --name ea \ + --ulimit nofile=40960:40960 \ + --ulimit core=100000000:100000000 \ + --ulimit memlock=100000000:100000000 \ + couchbase/enterprise-analytics +``` + +`unlimited` is not a supported value, so `core` and `memlock` are set to 100 GB. If your host has more than 100 GB of RAM, raise these to match. See [Deployment Considerations for Virtual Machines and Containers](https://docs.couchbase.com/enterprise-analytics/current/install/vm-container-guidelines.html). + ## Ports -| Port | Description | -|-------|-------------------------------| -| 8091 | Web console / REST API (HTTP) | -| 8095 | Analytics HTTP API | -| 18091 | Web console / REST API (HTTPS)| -| 18095 | Analytics HTTPS API | +The image exposes the following ports: + +| Port | Description | +|-------|-----------------------------------------------------------------------------| +| 8091 | Cluster administration REST/HTTP traffic, including the Web Console | +| 8095 | Enterprise Analytics Service REST/HTTP traffic | +| 9123 | Prometheus metrics | +| 11207 | Data Service (SDK cluster topology and auditing, TLS) | +| 11210 | Data Service (SDK cluster topology and auditing) | +| 11280 | Data Service Prometheus metrics | +| 18091 | Cluster administration REST/HTTP traffic, including the Web Console (TLS) | +| 18095 | Enterprise Analytics Service REST/HTTP traffic (TLS) | + +A multi-node cluster also requires node-to-node connectivity on a number of additional ports. When each node runs in its own container on a shared Docker network, this works without further configuration. For the full list, see [Enterprise Analytics Ports](https://docs.couchbase.com/enterprise-analytics/current/install/cb-enterprise-analytics-ports.html). ## Volumes -Data in Enterprise Analytics is stored under `/opt/enterprise-analytics/var/lib/couchbase/data`. For persistent deployments, mount a Docker volume or host directory to this path. +Enterprise Analytics writes all persistent, node-specific data — including cluster configuration, local storage, and logs — under `/opt/enterprise-analytics/var`, which the image declares as a volume. For persistent deployments, mount a Docker volume or host directory to this path so you can delete and recreate the container, or upgrade to a later version, without losing data. Example: ```bash -docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var/lib/couchbase/data couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var couchbase/enterprise-analytics ``` +Leaving `/opt/enterprise-analytics/var` inside the container's union filesystem also results in some performance degradation. + +> **Note:** If SELinux is enabled and you are bind-mounting a host directory, run `chcon -Rt svirt_sandbox_file_t ` once before starting the first container on that host. + +## Additional References + +- [Enterprise Analytics documentation](https://docs.couchbase.com/enterprise-analytics/current/index.html) +- [Install Enterprise Analytics Using Docker](https://docs.couchbase.com/enterprise-analytics/current/install/getting-started-docker.html) +- [Do a Quick Install](https://docs.couchbase.com/enterprise-analytics/current/intro/do-a-quick-install.html) +- [Release Notes](https://docs.couchbase.com/enterprise-analytics/current/release-notes/release-notes.html) + ## License -Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement. +Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement, available on the [Legal Agreements](https://www.couchbase.com/legal/agreements) page. The license is free for development and testing; a paid subscription is required for production deployment. See the [pricing](https://www.couchbase.com/pricing) page for details. diff --git a/enterprise/enterprise-analytics/2.2.1/README.md b/enterprise/enterprise-analytics/2.2.1/README.md index d6ab9468..3e35593a 100644 --- a/enterprise/enterprise-analytics/2.2.1/README.md +++ b/enterprise/enterprise-analytics/2.2.1/README.md @@ -12,9 +12,14 @@ Traditionally, analyzing JSON data in NoSQL databases requires complex transform * An enhanced MPP-based query engine enables scalable, real-time analytical query computation. * A cost-based optimizer improves query execution without requiring user intervention. Using a sample-based approach, it quickly estimates data statistics from a small subset of the data, enabling it to identify the lowest-cost query plan without scanning the entire dataset. * Zero ETL for incoming data, with real-time ingestion capabilities powered by Confluent Kafka, that provide the ability to connect, capture, and extract data from nearly any database or application. One can optionally modify the target JSON structure of the incoming data while in transit, for example, to omit or modify its fields. -* Data Lakehouse capabilities that enable direct querying from Amazon S3 and S3-compatible storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, and Delta tables, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* Data Lakehouse capabilities that enable direct querying from supported object storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, Delta tables, and Apache Iceberg tables via Iceberg catalogs, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* A choice of certified object stores for the underlying storage layer: Amazon S3, S3-compatible storage, [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) (2.1 and later) and [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html) (2.2 and later). +* Continuous, row-level change data capture from Oracle and SQL Server, powered by Kafka links and Debezium connectors (2.2 and later). * A SQL++ based path for writing the results of a query back to the Couchbase Operational data service to support adaptive applications. * A tabular view facility that provides native SQL-based support for Tableau, PowerBI and Apache Superset for building business reports, visualizations, and dashboards. +* An Index Advisor that recommends secondary indexes for your queries, plus index-only query plans that serve requests entirely from covered indexes (2.2 and later). +* JWT authentication for REST API and query requests, including tokens issued by external identity providers (2.2 and later). +* An asynchronous REST API for submitting long-running queries without blocking application workflows (2.2 and later). This Docker image is designed to make it easy to run Enterprise Analytics for development, testing, and proof-of-concept environments. @@ -25,8 +30,10 @@ To quickly get started with Enterprise Analytics, you can run an instance using ### Prerequisites These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No services running on ports `8091` or `8095` 1. No existing containers named `ea` (or `s3mock` if using S3Mock) +1. No existing Docker network named `ea-net` ### 1. Create a Docker network Create a user-defined network so the container can communicate with other services if needed. @@ -36,23 +43,29 @@ docker network create ea-net ``` ### 2. Configure S3Mock (optional) -If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use a different blob storage backend. +If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use one of the other supported blob storage backends — Amazon S3, S3-compatible storage, Azure Blob Storage, or Google Cloud Storage. See [Supported Object Storage Solutions](https://docs.couchbase.com/enterprise-analytics/current/install/supported-platform.html#supported-object-storage-solutions). ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the Enterprise Analytics container Run the Enterprise Analytics container with host and port mappings for the Couchbase Web Console and Enterprise Analytics service, exposed on ports `8091` and `8095` on the host. ```bash -docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` +To connect to this container from an SDK, also publish the Data Service ports, e.g. `-p 18091:18091 -p 18095:18095 -p 11207:11207 -p 11210:11210`. See "Ports" below. + ### 4. Initialize the cluster Next, visit http://localhost:8091 on the host machine to see the Web Console to start Enterprise Analytics setup. +> **Note:** The storage scheme and the number of storage partitions are fixed at cluster setup and cannot be changed afterwards. The remaining blob storage settings — bucket, region, path prefix, endpoint, authentication and TLS options — can be changed later under *Settings > Cluster*. + ![Setup splash screen](https://d774lla4im6mk.cloudfront.net/ea/setup-initial.png) Walk through the Setup wizard @@ -74,9 +87,9 @@ After completing the setup, the console will load. ![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/install-samples.png) ### 6. Execute a sample query -You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airports in the `travel-sample`. +You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airlines in the `travel-sample`. -![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) +![Sample query](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) ### 7. Next steps @@ -94,7 +107,9 @@ The following example shows how to start a two-node Enterprise Analytics cluster These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No existing containers named `s3mock`, `ea1`, or `ea2` +1. No existing Docker network named `ea-net` 1. No services running on ports `8091`, `8095`, `9091`, or `9095` ### 1. Create a Docker network @@ -110,7 +125,9 @@ docker network create ea-net Start the S3Mock container with an initial bucket called `ea-storage`. ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the first Enterprise Analytics node @@ -119,7 +136,7 @@ Run the first node (`ea1`) with host and port mappings for the Couchbase Web Con ```bash docker run -d --name ea1 --network ea-net --hostname ea1.example.com --network-alias ea1.example.com \ - -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 + -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` ### 4. Start the second Enterprise Analytics node @@ -128,7 +145,7 @@ Run the second node (`ea2`) with its own mapped ports so you can access it separ ```bash docker run -d --name ea2 --network ea-net --hostname ea2.example.com --network-alias ea2.example.com \ - -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics:2.0.0 + -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics ``` ### 5. Wait for the nodes to be ready @@ -172,9 +189,16 @@ docker exec ea1 couchbase-cli setting-enterprise-analytics --cluster http://loca --region us-east-1 \ --endpoint http://s3mock:9090 \ --anonymous-auth 1 \ - --path-style-addressing 1 + --path-style-addressing 1 \ + --partitions 16 ``` +`--partitions 16` keeps the number of storage partitions low; the default of 128 adds unnecessary overhead for a small development cluster. + +> **Note:** The storage scheme (`--scheme`) and the number of storage partitions (`--partitions`) are fixed once the cluster is initialized, so this step must come before `cluster-init` below. The remaining settings can be changed later with `--set`, or under *Settings > Cluster* in the Web Console. + +To use a backend other than S3 or S3-compatible storage, set `--scheme azblob` for [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) or `--scheme gs` for [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html). + ### 8. Initialize the cluster Initialize the Enterprise Analytics cluster. @@ -212,25 +236,68 @@ Once rebalanced, the cluster is ready to be used. Access the UI at: - **ea1:** [http://localhost:8091](http://localhost:8091) - **ea2:** [http://localhost:9091](http://localhost:9091) +## Container Requirements + +| Requirement | Details | +|-------------|---------| +| CPU | 2 GHz dual core x86_64 **supporting AVX2 or later**, or 2 GHz dual core 64-bit ARM v8. The container aborts at startup if the host CPU does not meet the required microarchitecture. | +| RAM | 4 GB minimum; 32 GB or more recommended | +| Disk | 8 GB minimum, block-based (HDD, SSD, EBS, iSCSI). Network file systems such as CIFS and NFS are not supported. | + +For development and testing you can go below these minimums — as little as 1 GB of free RAM and a single core — but production deployments must meet them. See [System Resource Requirements](https://docs.couchbase.com/enterprise-analytics/current/install/sys-resource-req.html). + +The image is published for both `linux/amd64` and `linux/arm64`. + +For production deployments, raise the container's ulimits: + +```bash +docker run -d --name ea \ + --ulimit nofile=40960:40960 \ + --ulimit core=100000000:100000000 \ + --ulimit memlock=100000000:100000000 \ + couchbase/enterprise-analytics +``` + +`unlimited` is not a supported value, so `core` and `memlock` are set to 100 GB. If your host has more than 100 GB of RAM, raise these to match. See [Deployment Considerations for Virtual Machines and Containers](https://docs.couchbase.com/enterprise-analytics/current/install/vm-container-guidelines.html). + ## Ports -| Port | Description | -|-------|-------------------------------| -| 8091 | Web console / REST API (HTTP) | -| 8095 | Analytics HTTP API | -| 18091 | Web console / REST API (HTTPS)| -| 18095 | Analytics HTTPS API | +The image exposes the following ports: + +| Port | Description | +|-------|-----------------------------------------------------------------------------| +| 8091 | Cluster administration REST/HTTP traffic, including the Web Console | +| 8095 | Enterprise Analytics Service REST/HTTP traffic | +| 9123 | Prometheus metrics | +| 11207 | Data Service (SDK cluster topology and auditing, TLS) | +| 11210 | Data Service (SDK cluster topology and auditing) | +| 11280 | Data Service Prometheus metrics | +| 18091 | Cluster administration REST/HTTP traffic, including the Web Console (TLS) | +| 18095 | Enterprise Analytics Service REST/HTTP traffic (TLS) | + +A multi-node cluster also requires node-to-node connectivity on a number of additional ports. When each node runs in its own container on a shared Docker network, this works without further configuration. For the full list, see [Enterprise Analytics Ports](https://docs.couchbase.com/enterprise-analytics/current/install/cb-enterprise-analytics-ports.html). ## Volumes -Data in Enterprise Analytics is stored under `/opt/enterprise-analytics/var/lib/couchbase/data`. For persistent deployments, mount a Docker volume or host directory to this path. +Enterprise Analytics writes all persistent, node-specific data — including cluster configuration, local storage, and logs — under `/opt/enterprise-analytics/var`, which the image declares as a volume. For persistent deployments, mount a Docker volume or host directory to this path so you can delete and recreate the container, or upgrade to a later version, without losing data. Example: ```bash -docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var/lib/couchbase/data couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var couchbase/enterprise-analytics ``` +Leaving `/opt/enterprise-analytics/var` inside the container's union filesystem also results in some performance degradation. + +> **Note:** If SELinux is enabled and you are bind-mounting a host directory, run `chcon -Rt svirt_sandbox_file_t ` once before starting the first container on that host. + +## Additional References + +- [Enterprise Analytics documentation](https://docs.couchbase.com/enterprise-analytics/current/index.html) +- [Install Enterprise Analytics Using Docker](https://docs.couchbase.com/enterprise-analytics/current/install/getting-started-docker.html) +- [Do a Quick Install](https://docs.couchbase.com/enterprise-analytics/current/intro/do-a-quick-install.html) +- [Release Notes](https://docs.couchbase.com/enterprise-analytics/current/release-notes/release-notes.html) + ## License -Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement. +Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement, available on the [Legal Agreements](https://www.couchbase.com/legal/agreements) page. The license is free for development and testing; a paid subscription is required for production deployment. See the [pricing](https://www.couchbase.com/pricing) page for details. diff --git a/generate/resources/enterprise-analytics/README.md b/generate/resources/enterprise-analytics/README.md index d6ab9468..3e35593a 100644 --- a/generate/resources/enterprise-analytics/README.md +++ b/generate/resources/enterprise-analytics/README.md @@ -12,9 +12,14 @@ Traditionally, analyzing JSON data in NoSQL databases requires complex transform * An enhanced MPP-based query engine enables scalable, real-time analytical query computation. * A cost-based optimizer improves query execution without requiring user intervention. Using a sample-based approach, it quickly estimates data statistics from a small subset of the data, enabling it to identify the lowest-cost query plan without scanning the entire dataset. * Zero ETL for incoming data, with real-time ingestion capabilities powered by Confluent Kafka, that provide the ability to connect, capture, and extract data from nearly any database or application. One can optionally modify the target JSON structure of the incoming data while in transit, for example, to omit or modify its fields. -* Data Lakehouse capabilities that enable direct querying from Amazon S3 and S3-compatible storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, and Delta tables, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* Data Lakehouse capabilities that enable direct querying from supported object storage, with support for formats including JSON, Parquet, Avro, CSV, TSV, Delta tables, and Apache Iceberg tables via Iceberg catalogs, providing the ability for queries to combine external data with other data in Enterprise Analytics. +* A choice of certified object stores for the underlying storage layer: Amazon S3, S3-compatible storage, [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) (2.1 and later) and [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html) (2.2 and later). +* Continuous, row-level change data capture from Oracle and SQL Server, powered by Kafka links and Debezium connectors (2.2 and later). * A SQL++ based path for writing the results of a query back to the Couchbase Operational data service to support adaptive applications. * A tabular view facility that provides native SQL-based support for Tableau, PowerBI and Apache Superset for building business reports, visualizations, and dashboards. +* An Index Advisor that recommends secondary indexes for your queries, plus index-only query plans that serve requests entirely from covered indexes (2.2 and later). +* JWT authentication for REST API and query requests, including tokens issued by external identity providers (2.2 and later). +* An asynchronous REST API for submitting long-running queries without blocking application workflows (2.2 and later). This Docker image is designed to make it easy to run Enterprise Analytics for development, testing, and proof-of-concept environments. @@ -25,8 +30,10 @@ To quickly get started with Enterprise Analytics, you can run an instance using ### Prerequisites These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No services running on ports `8091` or `8095` 1. No existing containers named `ea` (or `s3mock` if using S3Mock) +1. No existing Docker network named `ea-net` ### 1. Create a Docker network Create a user-defined network so the container can communicate with other services if needed. @@ -36,23 +43,29 @@ docker network create ea-net ``` ### 2. Configure S3Mock (optional) -If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use a different blob storage backend. +If you want to use S3Mock as the blob storage backend, you can start the S3Mock container first. Otherwise, you need to configure Enterprise Analytics to use one of the other supported blob storage backends — Amazon S3, S3-compatible storage, Azure Blob Storage, or Google Cloud Storage. See [Supported Object Storage Solutions](https://docs.couchbase.com/enterprise-analytics/current/install/supported-platform.html#supported-object-storage-solutions). ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the Enterprise Analytics container Run the Enterprise Analytics container with host and port mappings for the Couchbase Web Console and Enterprise Analytics service, exposed on ports `8091` and `8095` on the host. ```bash -docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea --network ea-net -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` +To connect to this container from an SDK, also publish the Data Service ports, e.g. `-p 18091:18091 -p 18095:18095 -p 11207:11207 -p 11210:11210`. See "Ports" below. + ### 4. Initialize the cluster Next, visit http://localhost:8091 on the host machine to see the Web Console to start Enterprise Analytics setup. +> **Note:** The storage scheme and the number of storage partitions are fixed at cluster setup and cannot be changed afterwards. The remaining blob storage settings — bucket, region, path prefix, endpoint, authentication and TLS options — can be changed later under *Settings > Cluster*. + ![Setup splash screen](https://d774lla4im6mk.cloudfront.net/ea/setup-initial.png) Walk through the Setup wizard @@ -74,9 +87,9 @@ After completing the setup, the console will load. ![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/install-samples.png) ### 6. Execute a sample query -You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airports in the `travel-sample`. +You can now run a sample query to verify that everything is working correctly. For example, you can run the following SQL++ query to get a count of airlines in the `travel-sample`. -![Install Samples](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) +![Sample query](https://d774lla4im6mk.cloudfront.net/ea/sample-query.png) ### 7. Next steps @@ -94,7 +107,9 @@ The following example shows how to start a two-node Enterprise Analytics cluster These instructions assume the following: 1. Docker installed and running +1. A host that meets the container requirements described under "Container Requirements" below 1. No existing containers named `s3mock`, `ea1`, or `ea2` +1. No existing Docker network named `ea-net` 1. No services running on ports `8091`, `8095`, `9091`, or `9095` ### 1. Create a Docker network @@ -110,7 +125,9 @@ docker network create ea-net Start the S3Mock container with an initial bucket called `ea-storage`. ```bash -docker run -d --name s3mock --network ea-net -e initialBuckets=ea-storage adobe/s3mock +docker run -d --name s3mock --network ea-net \ + -e COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage \ + adobe/s3mock ``` ### 3. Start the first Enterprise Analytics node @@ -119,7 +136,7 @@ Run the first node (`ea1`) with host and port mappings for the Couchbase Web Con ```bash docker run -d --name ea1 --network ea-net --hostname ea1.example.com --network-alias ea1.example.com \ - -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics:2.0.0 + -p 8091:8091 -p 8095:8095 couchbase/enterprise-analytics ``` ### 4. Start the second Enterprise Analytics node @@ -128,7 +145,7 @@ Run the second node (`ea2`) with its own mapped ports so you can access it separ ```bash docker run -d --name ea2 --network ea-net --hostname ea2.example.com --network-alias ea2.example.com \ - -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics:2.0.0 + -p 9091:8091 -p 9095:8095 couchbase/enterprise-analytics ``` ### 5. Wait for the nodes to be ready @@ -172,9 +189,16 @@ docker exec ea1 couchbase-cli setting-enterprise-analytics --cluster http://loca --region us-east-1 \ --endpoint http://s3mock:9090 \ --anonymous-auth 1 \ - --path-style-addressing 1 + --path-style-addressing 1 \ + --partitions 16 ``` +`--partitions 16` keeps the number of storage partitions low; the default of 128 adds unnecessary overhead for a small development cluster. + +> **Note:** The storage scheme (`--scheme`) and the number of storage partitions (`--partitions`) are fixed once the cluster is initialized, so this step must come before `cluster-init` below. The remaining settings can be changed later with `--set`, or under *Settings > Cluster* in the Web Console. + +To use a backend other than S3 or S3-compatible storage, set `--scheme azblob` for [Azure Blob Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/azure-blob-storage.html) or `--scheme gs` for [Google Cloud Storage](https://docs.couchbase.com/enterprise-analytics/current/manage/manage-nodes/google-cloud-storage.html). + ### 8. Initialize the cluster Initialize the Enterprise Analytics cluster. @@ -212,25 +236,68 @@ Once rebalanced, the cluster is ready to be used. Access the UI at: - **ea1:** [http://localhost:8091](http://localhost:8091) - **ea2:** [http://localhost:9091](http://localhost:9091) +## Container Requirements + +| Requirement | Details | +|-------------|---------| +| CPU | 2 GHz dual core x86_64 **supporting AVX2 or later**, or 2 GHz dual core 64-bit ARM v8. The container aborts at startup if the host CPU does not meet the required microarchitecture. | +| RAM | 4 GB minimum; 32 GB or more recommended | +| Disk | 8 GB minimum, block-based (HDD, SSD, EBS, iSCSI). Network file systems such as CIFS and NFS are not supported. | + +For development and testing you can go below these minimums — as little as 1 GB of free RAM and a single core — but production deployments must meet them. See [System Resource Requirements](https://docs.couchbase.com/enterprise-analytics/current/install/sys-resource-req.html). + +The image is published for both `linux/amd64` and `linux/arm64`. + +For production deployments, raise the container's ulimits: + +```bash +docker run -d --name ea \ + --ulimit nofile=40960:40960 \ + --ulimit core=100000000:100000000 \ + --ulimit memlock=100000000:100000000 \ + couchbase/enterprise-analytics +``` + +`unlimited` is not a supported value, so `core` and `memlock` are set to 100 GB. If your host has more than 100 GB of RAM, raise these to match. See [Deployment Considerations for Virtual Machines and Containers](https://docs.couchbase.com/enterprise-analytics/current/install/vm-container-guidelines.html). + ## Ports -| Port | Description | -|-------|-------------------------------| -| 8091 | Web console / REST API (HTTP) | -| 8095 | Analytics HTTP API | -| 18091 | Web console / REST API (HTTPS)| -| 18095 | Analytics HTTPS API | +The image exposes the following ports: + +| Port | Description | +|-------|-----------------------------------------------------------------------------| +| 8091 | Cluster administration REST/HTTP traffic, including the Web Console | +| 8095 | Enterprise Analytics Service REST/HTTP traffic | +| 9123 | Prometheus metrics | +| 11207 | Data Service (SDK cluster topology and auditing, TLS) | +| 11210 | Data Service (SDK cluster topology and auditing) | +| 11280 | Data Service Prometheus metrics | +| 18091 | Cluster administration REST/HTTP traffic, including the Web Console (TLS) | +| 18095 | Enterprise Analytics Service REST/HTTP traffic (TLS) | + +A multi-node cluster also requires node-to-node connectivity on a number of additional ports. When each node runs in its own container on a shared Docker network, this works without further configuration. For the full list, see [Enterprise Analytics Ports](https://docs.couchbase.com/enterprise-analytics/current/install/cb-enterprise-analytics-ports.html). ## Volumes -Data in Enterprise Analytics is stored under `/opt/enterprise-analytics/var/lib/couchbase/data`. For persistent deployments, mount a Docker volume or host directory to this path. +Enterprise Analytics writes all persistent, node-specific data — including cluster configuration, local storage, and logs — under `/opt/enterprise-analytics/var`, which the image declares as a volume. For persistent deployments, mount a Docker volume or host directory to this path so you can delete and recreate the container, or upgrade to a later version, without losing data. Example: ```bash -docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var/lib/couchbase/data couchbase/enterprise-analytics:2.0.0 +docker run -d --name ea1 -v ea1-data:/opt/enterprise-analytics/var couchbase/enterprise-analytics ``` +Leaving `/opt/enterprise-analytics/var` inside the container's union filesystem also results in some performance degradation. + +> **Note:** If SELinux is enabled and you are bind-mounting a host directory, run `chcon -Rt svirt_sandbox_file_t ` once before starting the first container on that host. + +## Additional References + +- [Enterprise Analytics documentation](https://docs.couchbase.com/enterprise-analytics/current/index.html) +- [Install Enterprise Analytics Using Docker](https://docs.couchbase.com/enterprise-analytics/current/install/getting-started-docker.html) +- [Do a Quick Install](https://docs.couchbase.com/enterprise-analytics/current/intro/do-a-quick-install.html) +- [Release Notes](https://docs.couchbase.com/enterprise-analytics/current/release-notes/release-notes.html) + ## License -Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement. +Enterprise Analytics is licensed under the Couchbase Enterprise License Agreement, available on the [Legal Agreements](https://www.couchbase.com/legal/agreements) page. The license is free for development and testing; a paid subscription is required for production deployment. See the [pricing](https://www.couchbase.com/pricing) page for details. diff --git a/generate/screenshots/README.md b/generate/screenshots/README.md index e9a91074..41ec335d 100644 --- a/generate/screenshots/README.md +++ b/generate/screenshots/README.md @@ -1,21 +1,59 @@ -# Screenshot Robot +# Screenshot Robots -This directory contains tooling used to automatically generate screenshots for the couchbase/server docker hub repository. It needs enough free memory for docker to bring up a node, and will probably need tweaked over time as UI elements evolve. +This directory contains tooling used to automatically generate the screenshots embedded in the Docker Hub overview pages. It needs enough free memory for docker to bring up a node, and will probably need tweaked over time as UI elements evolve. + +There are two robots: + +| Robot | Product | Compose file | +|-------|---------|--------------| +| `robot` | `couchbase/server` | `docker-compose.yml` | +| `robot-ea` | `couchbase/enterprise-analytics` | `docker-compose.ea.yml` | ## How it works -A docker-compose stack creates a couchbase/server:${TAG} service, and a robot service (playwright/node/chromium), the robot service opens the couchbase UI, walks through the initial setup wizard resizing the viewport and taking screenshots as it goes. +A docker-compose stack creates the product service, and a robot service (playwright/node/chromium), the robot service opens the product UI, walks through the initial setup wizard resizing the viewport and taking screenshots as it goes. Screenshots are stored in the `output` directory (./output) when complete. + +Output filenames must stay in sync with the image URLs in the corresponding `generate/resources//README.md`, since that README is what gets published as the Docker Hub description. + +Note: we use `--renew-anon-volumes` to ensure the product container is coming up on clean volumes and not bringing up an initialised cluster on subsequent runs. With `--exit-code-from robot` we ensure the robot is responsible for the lifecycle of the stack. -## Example Usage +## Example Usage: Couchbase Server `docker-compose build && TAG=6.6.2 docker-compose up --renew-anon-volumes --exit-code-from robot` -This will build the robot image and bring up the stack, screenshots will be stored in the `output` directory (./output) when complete. +## Example Usage: Enterprise Analytics + +The Enterprise Analytics stack also brings up [Adobe S3Mock](https://github.com/adobe/S3Mock) to stand in for S3-compatible blob storage, and the robot configures the cluster against it, mirroring the quickstart in the Docker Hub overview. + +``` +TAG=2.2.1 docker compose -f docker-compose.ea.yml build +TAG=2.2.1 docker compose -f docker-compose.ea.yml up --renew-anon-volumes --abort-on-container-exit --exit-code-from robot +``` + +The run takes several minutes: after initializing the cluster the robot waits for the Analytics service to warm up, then loads `travel-sample` and waits for ingestion to finish before running the sample query. + +Notes: -Note: we use `--renew-anon-volumes` to ensure the couchbase container is coming up on clean volumes and not bringing up an initialised cluster on subsequent runs. With `--exit-code-from robot` we ensure the robot is responsible for the lifecycle of the stack. +* `robot-ea` builds on `mcr.microsoft.com/playwright`, which is multi-arch, so it runs natively on both amd64 and arm64. The older `robot` image is amd64-only and runs under emulation on Apple Silicon. +* On failure the robot writes `output/FAILURE.png` and dumps the page text, so you can see which step broke when the UI changes. +* If a step times out on an element that does exist, check whether the input is visually hidden behind a styled label. Several of them are, which is why the robots click those via the DOM and wait for `state: 'attached'` rather than for visibility. ## Uploading the output -Once the images have been generated and you have visually confirmed they are correct, upload the contents of the output folder with: +Once the images have been generated and you have visually confirmed they are correct, upload them to the origin bucket behind `https://d774lla4im6mk.cloudfront.net`. + +Couchbase Server images live at the root of the bucket: `aws s3 cp --recursive output/ s3://cb-dockerhub-screenshots-origin` + +Enterprise Analytics images live under the `ea/` prefix: + +`aws s3 cp --recursive output/ s3://cb-dockerhub-screenshots-origin/ea/` + +The objects carry no `Cache-Control` header, so CloudFront keeps serving the previous images until its default TTL expires. Because the filenames do not change between releases, invalidate the prefix after uploading: + +`aws cloudfront create-invalidation --distribution-id --paths '/ea/*'` + +Look up `` with: + +`aws cloudfront list-distributions --query "DistributionList.Items[?contains(DomainName, 'd774lla4im6mk')].Id" --output text` diff --git a/generate/screenshots/docker-compose.ea.yml b/generate/screenshots/docker-compose.ea.yml new file mode 100644 index 00000000..f8972a45 --- /dev/null +++ b/generate/screenshots/docker-compose.ea.yml @@ -0,0 +1,21 @@ +services: + s3mock: + image: adobe/s3mock + environment: + - COM_ADOBE_TESTING_S3MOCK_STORE_INITIAL_BUCKETS=ea-storage + enterprise-analytics: + image: couchbase/enterprise-analytics:${TAG} + robot: + build: robot-ea + environment: + - EA_HOST=enterprise-analytics + - S3_ENDPOINT=http://s3mock:9090 + - S3_BUCKET=ea-storage + - CLUSTER_NAME=ea-cluster + - EA_USER=Administrator + - EA_PASS=password + volumes: + - ./output:/output + depends_on: + - enterprise-analytics + - s3mock diff --git a/generate/screenshots/robot-ea/Dockerfile b/generate/screenshots/robot-ea/Dockerfile new file mode 100644 index 00000000..9861688f --- /dev/null +++ b/generate/screenshots/robot-ea/Dockerfile @@ -0,0 +1,18 @@ +# Multi-arch (amd64 + arm64) Playwright image with browsers preinstalled, so +# the Enterprise Analytics robot runs natively on both Intel and Apple Silicon. +# The npm package version must match the image tag so the client and the +# preinstalled browser revisions agree. +FROM mcr.microsoft.com/playwright:v1.55.0-noble + +# Uncomment this to show playwright debug output +# ENV DEBUG=pw:api + +WORKDIR /src + +# Install the client at build time so runs need no network access +RUN npm install --no-save playwright@1.55.0 + +COPY --chown=pwuser:pwuser src /src +COPY entrypoint.sh / + +CMD [ "/entrypoint.sh" ] diff --git a/generate/screenshots/robot-ea/entrypoint.sh b/generate/screenshots/robot-ea/entrypoint.sh new file mode 100755 index 00000000..4886078b --- /dev/null +++ b/generate/screenshots/robot-ea/entrypoint.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +EA_HOST=${EA_HOST:-enterprise-analytics} + +while ! curl --fail "http://${EA_HOST}:8091" &>/dev/null +do + echo "Waiting for http://${EA_HOST}:8091" + sleep 1 +done + +set -e +node ea.js diff --git a/generate/screenshots/robot-ea/src/ea.js b/generate/screenshots/robot-ea/src/ea.js new file mode 100644 index 00000000..40c513b4 --- /dev/null +++ b/generate/screenshots/robot-ea/src/ea.js @@ -0,0 +1,308 @@ +// Walks the Enterprise Analytics setup wizard and console, capturing the +// screenshots embedded in the couchbase/enterprise-analytics Docker Hub +// overview (generate/resources/enterprise-analytics/README.md). +// +// Output filenames must stay in sync with the image URLs in that README. +const playwright = require('playwright'); + +const EA_HOST = process.env.EA_HOST || 'enterprise-analytics'; +const S3_ENDPOINT = process.env.S3_ENDPOINT || 'http://s3mock:9090'; +const S3_BUCKET = process.env.S3_BUCKET || 'ea-storage'; +const CLUSTER_NAME = process.env.CLUSTER_NAME || 'ea-cluster'; +const EA_USER = process.env.EA_USER || 'Administrator'; +const EA_PASS = process.env.EA_PASS || 'password'; +const OUT = process.env.OUTPUT_DIR || '/output'; + +const BASE = `http://${EA_HOST}:8091`; +const QUERY_URL = `http://${EA_HOST}:8095/analytics/service`; + +// Keep the number of storage partitions low; the default of 128 is needless +// overhead for a single-node screenshot cluster. +const STORAGE_PARTITIONS = 16; + +// Dimensions of the published images. The wizard pages are captured at a +// narrow width so they stay legible inline on Docker Hub; the console pages +// are captured wider so the left nav and results pane both fit. +const WIZARD_WIDTH = 640; +const CONSOLE = { width: 1024, height: 646 }; + +// Console navigation. Both the left nav and the section tab bar contain a link +// labelled "Workbench", so target them by route instead of by text. +const NAV_WORKBENCH = 'a[href^="#/cbas?"]'; +const TAB_WORKBENCH = 'a[href^="#/cbas/workbench"]'; +const TAB_SAMPLES = 'a[href^="#/cbas/samples"]'; + +const log = (msg) => console.log(`[ea-robot] ${msg}`); + +// Set once the page exists, so a failure can be diagnosed from the output dir. +let failurePage = null; + +async function shoot(page, name, size) { + if (size) { + await page.setViewportSize(size); + // let the UI reflow at the new size before capturing + await page.waitForTimeout(750); + } + await page.screenshot({ path: `${OUT}/${name}.png` }); + log(`captured ${name}.png`); +} + +// Set an input's value the way Angular's change detection expects. +async function setField(page, id, value) { + await page.evaluate(([id, value]) => { + const el = document.getElementById(id); + if (!el) throw new Error(`no element #${id}`); + el.focus(); + el.value = value; + el.dispatchEvent(new Event('input', { bubbles: true })); + el.dispatchEvent(new Event('change', { bubbles: true })); + el.blur(); + }, [id, value]); +} + +async function clickById(page, id) { + await page.evaluate((id) => { + const el = document.getElementById(id); + if (!el) throw new Error(`no element #${id}`); + el.click(); + }, id); +} + +// The wizard form scrolls inside its own container rather than the window, so +// position it by scrolling that container. Pass an element id to bring to the +// top of the form, or null for the very top. +async function scrollFormTo(page, anchorId) { + await page.evaluate((anchorId) => { + const findScroller = (el) => { + let p = el && el.parentElement; + while (p) { + const style = getComputedStyle(p); + if ( + (style.overflowY === 'auto' || style.overflowY === 'scroll') && + p.scrollHeight > p.clientHeight + ) { + return p; + } + p = p.parentElement; + } + return document.scrollingElement; + }; + + const probe = document.getElementById('setup_hostname'); + const scroller = findScroller(probe); + if (!scroller) return; + + if (!anchorId) { + scroller.scrollTop = 0; + return; + } + const anchor = document.getElementById(anchorId); + if (!anchor) return; + // Offset by the anchor's label so it is not clipped at the top edge. + const label = document.querySelector(`label[for="${anchorId}"]`); + const top = (label || anchor).getBoundingClientRect().top; + scroller.scrollTop += top - scroller.getBoundingClientRect().top; + }, anchorId); +} + +async function setCheckbox(page, id, checked) { + await page.evaluate(([id, checked]) => { + const el = document.getElementById(id); + if (!el) throw new Error(`no element #${id}`); + if (el.checked !== checked) el.click(); + }, [id, checked]); +} + +// Run a SQL++ statement against the Analytics service. +async function query(page, statement) { + const res = await page.request.post(QUERY_URL, { + form: { statement }, + headers: { + Authorization: + 'Basic ' + Buffer.from(`${EA_USER}:${EA_PASS}`).toString('base64'), + }, + timeout: 120000, + failOnStatusCode: false, + }); + if (!res.ok()) throw new Error(`query failed (${res.status()}): ${statement}`); + return res.json(); +} + +// The Analytics service warms up for a while after the cluster is initialized; +// until it does, the Workbench renders an error banner instead of the editor. +async function waitForAnalytics(page, timeoutMs = 300000) { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + try { + await query(page, 'SELECT 1'); + return; + } catch (e) { + await page.waitForTimeout(5000); + } + } + throw new Error('Analytics service did not become ready'); +} + +(async () => { + const browser = await playwright.chromium.launch(); + const context = await browser.newContext(); + const page = await context.newPage(); + page.setDefaultTimeout(60000); + failurePage = page; + + // ---------------------------------------------------------------- splash + await page.setViewportSize({ width: WIZARD_WIDTH, height: 476 }); + await page.goto(`${BASE}/ui/index.html`); + await page.waitForSelector('text=Setup New Cluster'); + await page.waitForTimeout(1000); // let the logo finish painting + await shoot(page, 'setup-initial'); + + // ------------------------------------------------- cluster + credentials + await page.click('text=Setup New Cluster'); + await page.waitForSelector('#for-cluster-name-field'); + await setField(page, 'for-cluster-name-field', CLUSTER_NAME); + await setField(page, 'secure-username', EA_USER); + await setField(page, 'secure-password', EA_PASS); + await setField(page, 'secure-password-verify', EA_PASS); + await shoot(page, 'setup-wizard', { width: WIZARD_WIDTH, height: 448 }); + + // ----------------------------------------------------------------- terms + log('accepting terms'); + await page.click('button[type=submit]'); // Next: Accept Terms + // The real checkbox input is visually hidden behind a styled label, so wait + // for it to be attached rather than visible, and click it via the DOM. + await page.waitForSelector('#for-accept-terms', { state: 'attached' }); + await clickById(page, 'for-accept-terms'); + await page.waitForTimeout(500); + + // ------------------------------------------------- disk / memory / blob + log('opening disk/memory/blob configuration'); + await page.click('button[type=submit]'); // Configure Disk, Memory, Blob Storage + await page.waitForSelector('#num_storage_partitions', { state: 'attached' }); + + // S3-compatible, pointed at the S3Mock service, with anonymous auth and + // path-style addressing (S3Mock supports neither IAM nor virtual-host URLs). + await clickById(page, 's3-compat'); + await page.waitForSelector('#bucket_endpoint', { state: 'attached' }); + await setField(page, 'bucket_endpoint', S3_ENDPOINT); + await setField(page, 'bucket_name', S3_BUCKET); + await setField(page, 'bucket_path_prefix', `${CLUSTER_NAME}/`); + await setField(page, 'bucket_region', 'us-east-1'); + await clickById(page, 'cred-mode-anonymous'); + await setCheckbox(page, 'for-force-path-style', true); + await setField(page, 'num_storage_partitions', String(STORAGE_PARTITIONS)); + + // The config form is taller than one screen, so it is published as two + // images: the top of the form, then the remainder scrolled into view. + await page.setViewportSize({ width: WIZARD_WIDTH, height: 784 }); + await page.waitForTimeout(750); + await scrollFormTo(page, null); + await page.waitForTimeout(500); + await shoot(page, 'blob-storage-config-1'); + + // Second image picks up from the bucket fields, so the authentication mode + // and the local storage paths are both visible. Taller than the equivalent + // 2.0 image because 2.2 added the endpoint certificate field and the + // Advanced section to this form. + await page.setViewportSize({ width: WIZARD_WIDTH, height: 960 }); + await page.waitForTimeout(750); + await scrollFormTo(page, 'bucket_name'); + await page.waitForTimeout(500); + await shoot(page, 'blob-storage-config-2'); + + // -------------------------------------------------------------- finish up + log('submitting cluster configuration'); + await page.click('text=Save & Finish'); + + // The console loads (on the Dashboard) once the cluster is initialized. + await page.waitForSelector(`${NAV_WORKBENCH}`, { timeout: 180000 }); + + log('waiting for the Analytics service to warm up'); + await waitForAnalytics(page); + await page.reload(); + await page.waitForSelector(`${NAV_WORKBENCH}`); + await page.waitForTimeout(3000); + + // ------------------------------------------------------------- workbench + // Captured before travel-sample is loaded, so the Databases pane shows only + // the Default database. + log('opening workbench'); + await page.setViewportSize(CONSOLE); + await page.click(NAV_WORKBENCH); + await page.waitForSelector('.wb-ace-editor', { state: 'attached' }); + await page.waitForTimeout(3000); + await shoot(page, 'workbench'); + + // ------------------------------------------------------- sample datasets + log('opening samples'); + await page.click(TAB_SAMPLES); + await page.waitForSelector('text=Load Sample Data'); + await page.waitForTimeout(1000); + // Same hidden-input treatment as the terms checkbox. + await page.evaluate(() => { + const box = document.querySelector('input[type=checkbox]'); + if (box && !box.checked) box.click(); + }); + await page.waitForTimeout(500); + await shoot(page, 'install-samples'); + + log('loading travel-sample'); + await page.click('text=Load Sample Data'); + + // Wait for ingestion to finish rather than guessing at a sleep. + const deadline = Date.now() + 300000; + let airlines = 0; + while (Date.now() < deadline) { + try { + const r = await query( + page, + 'SELECT VALUE COUNT(*) FROM `travel-sample`.`inventory`.`airline`' + ); + airlines = (r.results && r.results[0]) || 0; + if (airlines > 0) break; + } catch (e) { + // dataset not queryable yet + } + await page.waitForTimeout(5000); + } + if (!airlines) throw new Error('travel-sample did not finish loading'); + log(`travel-sample ready (${airlines} airlines)`); + + // ---------------------------------------------------------- sample query + // Must stay in step with the query described in the README text. + const statement = 'SELECT COUNT(*) FROM `travel-sample`.`inventory`.`airline`'; + + log('running sample query'); + await page.click(TAB_WORKBENCH); + await page.waitForSelector('.wb-ace-editor', { state: 'attached' }); + await page.waitForTimeout(2000); + + // The query editor is an ACE instance; drive it through ACE's own API rather + // than synthesising keystrokes. + await page.evaluate((statement) => { + const el = document.querySelector('.wb-ace-editor'); + window.ace.edit(el).setValue(statement); + }, statement); + await page.waitForTimeout(500); + + await page.click('button:has-text("Execute")'); + await page.waitForSelector('text=success', { timeout: 120000 }); + await page.waitForTimeout(2500); + await shoot(page, 'sample-query'); + + await browser.close(); + log('done'); +})().catch(async (err) => { + console.error(`[ea-robot] FAILED: ${err.stack || err}`); + if (failurePage) { + try { + await failurePage.screenshot({ path: `${OUT}/FAILURE.png`, fullPage: true }); + const text = await failurePage.evaluate(() => document.body.innerText); + console.error(`[ea-robot] page text at failure:\n${text.slice(0, 1500)}`); + console.error(`[ea-robot] wrote ${OUT}/FAILURE.png`); + } catch (e) { + console.error(`[ea-robot] could not capture failure state: ${e}`); + } + } + process.exit(1); +});