From 02bb3527b9525201ca5aaa9c3850c578b9178393 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:00:56 -0700 Subject: [PATCH 1/9] docs: update Testing on cloud section to use mage targets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The make -C dev-tools/cloud targets (cloud-deploy, cloud-clean, build-and-push-cloud-image) no longer exist — dev-tools/cloud contains only a terraform/ directory. Replace with the current mage-based workflow (test:cloudE2E and the individual docker:/test: targets) that CI actually uses. --- docs/developers-guide.md | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index b1d85768ec..08e0041086 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -400,23 +400,30 @@ cp build/binaries/fleet-server-8.7.0-SNAPSHOT-linux-x86_64/fleet-server ./data/e ## Testing on cloud -Elastic employees can create an Elastic Cloud deployment with a locally built Fleet Server. +Elastic employees can create an Elastic Cloud (ECH) deployment with a locally built Fleet Server. -To deploy it you can use the following commands: +The full end-to-end flow — build, push, deploy, test, and teardown — can be run with a single mage target: ```bash -EC_API_KEY=yourapikey make -C dev-tools/cloud cloud-deploy +EC_API_KEY=yourapikey mage test:cloudE2E ``` -And then to clean the deployment +This runs the following steps in sequence: -```bash -EC_API_KEY=yourapikey make -C dev-tools/cloud cloud-clean -``` +1. **`mage docker:cover`** — builds a coverage-instrumented `fleet-server` binary inside Docker +2. **`mage docker:customAgentImage`** — creates a custom `elastic-agent-cloud` image with the locally built `fleet-server` binary swapped in (base image: `docker.elastic.co/cloud-release/elastic-agent-cloud`) +3. **`mage docker:push`** — pushes the custom image to the registry (`docker.elastic.co/beats-ci/elastic-agent-cloud-fleet`) +4. **`mage test:cloudE2EUp`** — provisions an ECH deployment via Terraform using the custom image +5. **`mage test:cloudE2ERun`** — runs the cloud E2E tests against the deployment +6. **`mage test:cloudE2EDown`** — destroys the ECH deployment -For more advanced scenario you can build a custom docker image that you could use in your own terraform. +You can also run each step individually. For example, to provision and tear down a deployment without running tests: +```bash +EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp +# ... manual testing ... +EC_API_KEY=yourapikey mage test:cloudE2EDown ``` -make -C dev-tools/cloud build-and-push-cloud-image -``` + +The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag used for the ECH deployment. Run `mage -h test:cloudE2EUp` for all available options. From 4815bea78891c36fe3023f5cfb9771dd7ccbf8e9 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:01:49 -0700 Subject: [PATCH 2/9] docs: put manual deploy workflow first, cloudE2E is the CI target --- docs/developers-guide.md | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 08e0041086..3533684295 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -402,28 +402,27 @@ cp build/binaries/fleet-server-8.7.0-SNAPSHOT-linux-x86_64/fleet-server ./data/e Elastic employees can create an Elastic Cloud (ECH) deployment with a locally built Fleet Server. -The full end-to-end flow — build, push, deploy, test, and teardown — can be run with a single mage target: +To build a custom image and deploy it to ECH for manual testing: ```bash -EC_API_KEY=yourapikey mage test:cloudE2E +EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp +# ... manual testing ... +EC_API_KEY=yourapikey mage test:cloudE2EDown ``` -This runs the following steps in sequence: +These steps do the following: 1. **`mage docker:cover`** — builds a coverage-instrumented `fleet-server` binary inside Docker 2. **`mage docker:customAgentImage`** — creates a custom `elastic-agent-cloud` image with the locally built `fleet-server` binary swapped in (base image: `docker.elastic.co/cloud-release/elastic-agent-cloud`) 3. **`mage docker:push`** — pushes the custom image to the registry (`docker.elastic.co/beats-ci/elastic-agent-cloud-fleet`) 4. **`mage test:cloudE2EUp`** — provisions an ECH deployment via Terraform using the custom image -5. **`mage test:cloudE2ERun`** — runs the cloud E2E tests against the deployment -6. **`mage test:cloudE2EDown`** — destroys the ECH deployment +5. **`mage test:cloudE2EDown`** — destroys the ECH deployment when done + +The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag. Run `mage -h test:cloudE2EUp` for all available options. -You can also run each step individually. For example, to provision and tear down a deployment without running tests: +To run the full automated cloud E2E test suite (build, deploy, test, and teardown in one shot), use: ```bash -EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp -# ... manual testing ... -EC_API_KEY=yourapikey mage test:cloudE2EDown +EC_API_KEY=yourapikey mage test:cloudE2E ``` -The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag used for the ECH deployment. Run `mage -h test:cloudE2EUp` for all available options. - From d26732ab8c0ab76156a37cf302864c8707d23660 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:04:05 -0700 Subject: [PATCH 3/9] docs: clarify that test:cloudE2E adds test:cloudE2ERun to the manual flow --- docs/developers-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 3533684295..41e5583755 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -420,7 +420,7 @@ These steps do the following: The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag. Run `mage -h test:cloudE2EUp` for all available options. -To run the full automated cloud E2E test suite (build, deploy, test, and teardown in one shot), use: +To also run the automated cloud E2E test suite after deploying, use the all-in-one target (equivalent to the above steps plus `mage test:cloudE2ERun`): ```bash EC_API_KEY=yourapikey mage test:cloudE2E From 581df19bb25a460d2d9702898be124b3b555ebf4 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:04:30 -0700 Subject: [PATCH 4/9] docs: clarify when test:cloudE2ERun runs relative to up/down --- docs/developers-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 41e5583755..a2d612daaa 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -420,7 +420,7 @@ These steps do the following: The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag. Run `mage -h test:cloudE2EUp` for all available options. -To also run the automated cloud E2E test suite after deploying, use the all-in-one target (equivalent to the above steps plus `mage test:cloudE2ERun`): +To also run the automated cloud E2E test suite against the deployment (and tear it down afterwards), use the all-in-one target — equivalent to the above steps with `mage test:cloudE2ERun` inserted between `test:cloudE2EUp` and `test:cloudE2EDown`: ```bash EC_API_KEY=yourapikey mage test:cloudE2E From aa8e9994436336897218b76c20ade075cacf0ba5 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:07:32 -0700 Subject: [PATCH 5/9] docs: note CFT region is used for cloud E2E deployment --- docs/developers-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index a2d612daaa..3918341b3c 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -415,7 +415,7 @@ These steps do the following: 1. **`mage docker:cover`** — builds a coverage-instrumented `fleet-server` binary inside Docker 2. **`mage docker:customAgentImage`** — creates a custom `elastic-agent-cloud` image with the locally built `fleet-server` binary swapped in (base image: `docker.elastic.co/cloud-release/elastic-agent-cloud`) 3. **`mage docker:push`** — pushes the custom image to the registry (`docker.elastic.co/beats-ci/elastic-agent-cloud-fleet`) -4. **`mage test:cloudE2EUp`** — provisions an ECH deployment via Terraform using the custom image +4. **`mage test:cloudE2EUp`** — provisions an ECH deployment in the Cloud-First Testing (CFT) region via Terraform using the custom image 5. **`mage test:cloudE2EDown`** — destroys the ECH deployment when done The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag. Run `mage -h test:cloudE2EUp` for all available options. From 4a541ec3a9860e9fe3b998fcaae3f2e0214d79e0 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:10:51 -0700 Subject: [PATCH 6/9] docs: clarify DOCKER_IMAGE/TAG overrides only apply to individual targets --- docs/developers-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 3918341b3c..5af8526f78 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -418,7 +418,7 @@ These steps do the following: 4. **`mage test:cloudE2EUp`** — provisions an ECH deployment in the Cloud-First Testing (CFT) region via Terraform using the custom image 5. **`mage test:cloudE2EDown`** — destroys the ECH deployment when done -The `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` environment variables can be used to override the image name and tag. Run `mage -h test:cloudE2EUp` for all available options. +When running the individual targets, `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` can be used to override the image name and tag used for the ECH deployment. Note that `mage test:cloudE2E` sets these variables internally, so external overrides have no effect on the all-in-one target. Run `mage -h test:cloudE2EUp` for all available options. To also run the automated cloud E2E test suite against the deployment (and tear it down afterwards), use the all-in-one target — equivalent to the above steps with `mage test:cloudE2ERun` inserted between `test:cloudE2EUp` and `test:cloudE2EDown`: From 840701d2728ef5f138f791f9a822ca06108f1d80 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Fri, 7 Aug 2026 13:15:33 -0700 Subject: [PATCH 7/9] docs: address copilot feedback on cloud testing section - Fix ECH expansion: Elastic Cloud Hosted (ECH) - Add required SNAPSHOT, PLATFORMS, and DOCKER_IMAGE_TAG env vars to manual deploy example; explain why each is needed - Note that mage test:cloudE2E may leave a deployment running on failure and how to clean up --- docs/developers-guide.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 5af8526f78..c60fa5446c 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -400,16 +400,19 @@ cp build/binaries/fleet-server-8.7.0-SNAPSHOT-linux-x86_64/fleet-server ./data/e ## Testing on cloud -Elastic employees can create an Elastic Cloud (ECH) deployment with a locally built Fleet Server. +Elastic employees can create an Elastic Cloud Hosted (ECH) deployment with a locally built Fleet Server. To build a custom image and deploy it to ECH for manual testing: ```bash -EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp +SNAPSHOT=true PLATFORMS=linux/amd64 DOCKER_IMAGE_TAG=my-custom-tag \ + EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp # ... manual testing ... -EC_API_KEY=yourapikey mage test:cloudE2EDown +DOCKER_IMAGE_TAG=my-custom-tag EC_API_KEY=yourapikey mage test:cloudE2EDown ``` +`SNAPSHOT=true` and `PLATFORMS=linux/amd64` are required — ECH runs on `linux/amd64`, so omitting `PLATFORMS` on Apple Silicon will produce an `arm64` image that won't run in the deployment. Setting a consistent `DOCKER_IMAGE_TAG` ensures that `docker:customAgentImage`, `docker:push`, and `test:cloudE2EUp`/`test:cloudE2EDown` all refer to the same image. + These steps do the following: 1. **`mage docker:cover`** — builds a coverage-instrumented `fleet-server` binary inside Docker @@ -418,7 +421,7 @@ These steps do the following: 4. **`mage test:cloudE2EUp`** — provisions an ECH deployment in the Cloud-First Testing (CFT) region via Terraform using the custom image 5. **`mage test:cloudE2EDown`** — destroys the ECH deployment when done -When running the individual targets, `DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` can be used to override the image name and tag used for the ECH deployment. Note that `mage test:cloudE2E` sets these variables internally, so external overrides have no effect on the all-in-one target. Run `mage -h test:cloudE2EUp` for all available options. +`DOCKER_IMAGE` and `DOCKER_IMAGE_TAG` can be used to override the image name and tag. Note that `mage test:cloudE2E` sets these variables internally, so external overrides have no effect on the all-in-one target. Run `mage -h test:cloudE2EUp` for all available options. To also run the automated cloud E2E test suite against the deployment (and tear it down afterwards), use the all-in-one target — equivalent to the above steps with `mage test:cloudE2ERun` inserted between `test:cloudE2EUp` and `test:cloudE2EDown`: @@ -426,3 +429,5 @@ To also run the automated cloud E2E test suite against the deployment (and tear EC_API_KEY=yourapikey mage test:cloudE2E ``` +If `mage test:cloudE2E` fails partway through, the deployment may be left running. Run `mage test:cloudE2EDown` to clean it up. + From 3c4e5b8ac90b73658e0a3701338d09dc7b6da6a4 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Mon, 10 Aug 2026 16:29:10 -0700 Subject: [PATCH 8/9] docs: use a unique cloud test image tag --- docs/developers-guide.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index c60fa5446c..ce4ca4f4c3 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -405,13 +405,15 @@ Elastic employees can create an Elastic Cloud Hosted (ECH) deployment with a loc To build a custom image and deploy it to ECH for manual testing: ```bash -SNAPSHOT=true PLATFORMS=linux/amd64 DOCKER_IMAGE_TAG=my-custom-tag \ +export DOCKER_IMAGE=docker.elastic.co/beats-ci/elastic-agent-cloud-fleet +export DOCKER_IMAGE_TAG=my-username-$(date +%s) +SNAPSHOT=true PLATFORMS=linux/amd64 \ EC_API_KEY=yourapikey mage docker:cover docker:customAgentImage docker:push test:cloudE2EUp # ... manual testing ... -DOCKER_IMAGE_TAG=my-custom-tag EC_API_KEY=yourapikey mage test:cloudE2EDown +EC_API_KEY=yourapikey mage test:cloudE2EDown ``` -`SNAPSHOT=true` and `PLATFORMS=linux/amd64` are required — ECH runs on `linux/amd64`, so omitting `PLATFORMS` on Apple Silicon will produce an `arm64` image that won't run in the deployment. Setting a consistent `DOCKER_IMAGE_TAG` ensures that `docker:customAgentImage`, `docker:push`, and `test:cloudE2EUp`/`test:cloudE2EDown` all refer to the same image. +`SNAPSHOT=true` and `PLATFORMS=linux/amd64` build the required snapshot binary. The registry is shared, so choose a globally unique `DOCKER_IMAGE_TAG`; keeping the same `DOCKER_IMAGE` and tag in the shell session ensures that `docker:customAgentImage`, `docker:push`, `test:cloudE2EUp`, and `test:cloudE2EDown` all refer to the same image. These steps do the following: @@ -430,4 +432,3 @@ EC_API_KEY=yourapikey mage test:cloudE2E ``` If `mage test:cloudE2E` fails partway through, the deployment may be left running. Run `mage test:cloudE2EDown` to clean it up. - From 3eec6995a7a525df4662a9378d4c744c5f9ad188 Mon Sep 17 00:00:00 2001 From: Shaunak Kashyap Date: Mon, 10 Aug 2026 16:29:23 -0700 Subject: [PATCH 9/9] docs: clarify cloud test host architecture --- docs/developers-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developers-guide.md b/docs/developers-guide.md index ce4ca4f4c3..514ff4da9a 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -413,7 +413,7 @@ SNAPSHOT=true PLATFORMS=linux/amd64 \ EC_API_KEY=yourapikey mage test:cloudE2EDown ``` -`SNAPSHOT=true` and `PLATFORMS=linux/amd64` build the required snapshot binary. The registry is shared, so choose a globally unique `DOCKER_IMAGE_TAG`; keeping the same `DOCKER_IMAGE` and tag in the shell session ensures that `docker:customAgentImage`, `docker:push`, `test:cloudE2EUp`, and `test:cloudE2EDown` all refer to the same image. +Run this workflow from an `amd64` host: `docker:customAgentImage` creates an image for the host architecture, while ECH runs on `linux/amd64`. `SNAPSHOT=true` and `PLATFORMS=linux/amd64` build the required snapshot binary. The registry is shared, so choose a globally unique `DOCKER_IMAGE_TAG`; keeping the same `DOCKER_IMAGE` and tag in the shell session ensures that `docker:customAgentImage`, `docker:push`, `test:cloudE2EUp`, and `test:cloudE2EDown` all refer to the same image. These steps do the following: