Skip to content

Add test-env setup script and skill for Claude Code cloud sessions #8065

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
phacops wants to merge 1 commit into
base: master
Choose a base branch
from
+168 −2
Open

Add test-env setup script and skill for Claude Code cloud sessions #8065

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .claude/skills/setup-test-env/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
name: setup-test-env
description: Set up the Snuba dev/test environment in a Claude Code cloud session so the test suite can run. Use when the user wants to run tests, install devenv/devservices, start clickhouse/redis/kafka, or otherwise prepare an ephemeral Linux container for development. Installs sentry-devenv, builds the Python venv and the rust_snuba native extension, starts Docker, and brings up devservices.
---

# Set up the Snuba test environment

Use this skill to make tests runnable in an ephemeral Linux container (such as a
Claude Code cloud session), where nothing is installed and the Docker daemon is
not running.

## How to run it

Run the setup script from the repo root:

```bash
bash scripts/setup-test-env.sh
```

It is idempotent, so it is safe to re-run. It takes several minutes on a cold
container (`uv sync` builds the rust extension, and the clickhouse/redis/kafka
images have to be pulled), and is much faster afterwards because the container
state is cached.

The script:

1. Installs `sentry-devenv` via the official bash installer
(`~/.local/share/sentry-devenv/bin/devenv`).
2. Runs `uv sync --frozen --active`, which creates `.venv` **and** compiles the
native `rust_snuba` extension into it (no separate `maturin develop` needed).
`devservices` is a dev dependency, so it lands at `.venv/bin/devservices`.
3. Starts the Docker daemon (`sudo dockerd`) if it isn't already running.
4. Runs `.venv/bin/devservices up` to start clickhouse, redis and kafka.

## Running tests after setup

Run pytest **from inside the `tests/` directory**:

```bash
(cd tests && SNUBA_SETTINGS=test ../.venv/bin/pytest <path/to/test_file.py> -m "not ci_only")
```

Why from inside `tests/`: invoking pytest from the repo root loads
`test_distributed_migrations/conftest.py`, whose `pytest_configure` connects to a
multi-node clickhouse host (`clickhouse-query`) that devservices does not provide
in single-node mode, causing an `INTERNALERROR` before any test runs. Running
from within `tests/` avoids loading that sibling conftest.

For rust tests, source the rust env vars first:

```bash
. scripts/rust-envvars && (cd rust_snuba && cargo test --workspace)
```

## Notes / gotchas

- `devenv` refuses to run as root and `devenv sync` is Homebrew-based, so in the
cloud container the functional setup path is `uv sync` + `devservices`, not
`devenv sync`. The global `devenv` install is included only for parity with
local laptop setups.
- The clickhouse container's `nofile` ulimit in `devservices/config.yml` is set
to a value (soft 1024 / hard 4096) that fits inside the container's capability
set; the production-style 262144 cannot be applied without `CAP_SYS_RESOURCE`.
- If `devservices up` reports clickhouse is unhealthy, check
`docker ps` / `docker logs snuba-clickhouse-1`.
4 changes: 2 additions & 2 deletions devservices/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ services:
image: ghcr.io/getsentry/image-mirror-altinity-clickhouse-server:25.3.8.10041.altinitystable
ulimits:
nofile:
soft: 262144
hard: 262144
soft: 1024
hard: 4096
ports:
Comment on lines 52 to 55

@sentry sentry Bot Jun 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The nofile ulimit for ClickHouse in devservices/config.yml is reduced to 4096. This low limit is likely to be exhausted during local test runs, causing failures.
Severity: HIGH

Suggested Fix

To avoid impacting all local development environments, consider using separate Docker Compose configurations. Revert the ulimit change in the general-purpose devservices/config.yml and create a new, specialized configuration file (e.g., docker-compose.cloud.yml) with the lower nofile limit specifically for the cloud container environment that requires it. This isolates the restrictive setting to only the environment where it is necessary.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: devservices/config.yml#L52-L55

Potential issue: The `nofile` ulimit for ClickHouse in `devservices/config.yml` has been
reduced to a hard limit of `4096`. This configuration is used by the `devservices` tool
for local development environments. ClickHouse is known to require a high number of file
descriptors, especially under load. The Snuba test suite performs numerous concurrent
operations, which will likely exceed this low limit, causing tests to fail with "Too
many open files" errors. This impacts any developer running the test suite locally.

Did we get this right? 👍 / 👎 to inform future reviews.

@phacops phacops Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This lower limit is intentional. The sandboxed cloud containers these scripts target lack CAP_SYS_RESOURCE, so the daemon's own nofile hard limit is capped at 4096 — runc can't grant a container a higher limit than that, and the original 262144 made the clickhouse container fail to start entirely (error setting rlimit type 7: operation not permitted). Any value ≤ 4096 is the only thing that works there.

The suggested split (a separate docker-compose.cloud.yml) isn't cleanly doable: devservices up invokes docker compose -f devservices/config.yml ... directly and offers no override-file mechanism, so a sibling compose file wouldn't be picked up.

In practice 4096 is sufficient for a dev/test clickhouse — the test suite passes against it. Keeping the single shared value for simplicity per maintainer decision.

Generated by Claude Code

- 127.0.0.1:9000:9000
- 127.0.0.1:9009:9009
Expand Down
101 changes: 101 additions & 0 deletions scripts/setup-test-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
#!/bin/bash
# Set up a Snuba dev/test environment in an ephemeral Linux container
# (e.g. a Claude Code cloud session) so that the test suite can run.
#
# This is intentionally a standalone, on-demand script (not a session-start
# hook): `uv sync` plus pulling the service images takes several minutes, so we
# only want to pay that cost when we actually need to run tests.
#
# What it does (all steps are idempotent and safe to re-run):
# 1. Install sentry-devenv (via the official bash installer).
# 2. Build the Python virtualenv AND the native rust_snuba extension
# (`uv sync` compiles rust_snuba into the venv, so no separate
# `maturin develop` step is needed).
# 3. Make sure the Docker daemon is running.
# 4. Bring up the service dependencies via devservices (clickhouse, redis,
# kafka).
#
# Usage: bash scripts/setup-test-env.sh

set -euo pipefail

REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"

log() { printf '\n\033[1;34m==> %s\033[0m\n' "$*"; }

# ---------------------------------------------------------------------------
# 1. Install sentry-devenv (global, into ~/.local/share/sentry-devenv/bin).
# devservices itself is installed by `uv sync` below (it is a dev
# dependency and ends up at .venv/bin/devservices).
# ---------------------------------------------------------------------------
DEVENV_BIN="$HOME/.local/share/sentry-devenv/bin"
if [ -x "$DEVENV_BIN/devenv" ]; then
log "sentry-devenv already installed ($DEVENV_BIN/devenv)"
else
log "Installing sentry-devenv"
installer="$(mktemp -t install-devenv.XXXX.sh)"
curl -fsSL https://raw.githubusercontent.com/getsentry/devenv/main/install-devenv.sh -o "$installer"
# The installer only runs when invoked as a file literally named
# `install-devenv.sh`, so rename before executing.
mv "$installer" "$(dirname "$installer")/install-devenv.sh"
installer="$(dirname "$installer")/install-devenv.sh"
chmod +x "$installer"
# CI=1 makes the installer non-interactive; </dev/null stops the trailing
# login shell from blocking. Non-fatal: the functional path is `uv sync`.
if ! CI=1 SHELL="${SHELL:-/bin/bash}" "$installer" </dev/null; then
log "WARN: devenv install failed; continuing (uv provides the venv)."
fi
rm -f "$installer"
fi
export PATH="$DEVENV_BIN:$PATH"

# ---------------------------------------------------------------------------
# 2. Build the Python virtualenv + the rust_snuba native extension.
# ---------------------------------------------------------------------------
log "Syncing Python dependencies and building rust_snuba (uv sync)"
uv sync --frozen --active

# ---------------------------------------------------------------------------
# 3. Make sure the Docker daemon is running (cloud containers don't start one).
# ---------------------------------------------------------------------------
log "Ensuring the Docker daemon is running"
if docker info >/dev/null 2>&1; then
echo "Docker daemon already running."
else
echo "Starting dockerd ..."
sudo dockerd >/tmp/dockerd.log 2>&1 &
for _ in $(seq 1 30); do
if docker info >/dev/null 2>&1; then break; fi
sleep 1
done
if ! docker info >/dev/null 2>&1; then
echo "ERROR: Docker daemon did not become ready. Last log lines:" >&2
tail -n 20 /tmp/dockerd.log >&2 || true
exit 1
fi
echo "Docker daemon is up."
fi

# ---------------------------------------------------------------------------
# 4. Bring up the service dependencies (clickhouse, redis, kafka).
# ---------------------------------------------------------------------------
log "Starting devservices (clickhouse, redis, kafka)"
.venv/bin/devservices up

# ---------------------------------------------------------------------------
# Done.
# ---------------------------------------------------------------------------
log "Environment ready. To run tests:"
cat <<'EOF'
export PATH="$PWD/.venv/bin:$PATH"

# Run pytest from inside the tests/ directory. Invoking it from the repo
# root loads test_distributed_migrations/conftest.py, whose pytest_configure
# connects to a multi-node clickhouse ("clickhouse-query") that devservices
# does not provide in single-node mode.
(cd tests && SNUBA_SETTINGS=test ../.venv/bin/pytest <path/to/test_file.py> -m "not ci_only")

# For rust tests you also need the rust env vars:
# . scripts/rust-envvars && (cd rust_snuba && cargo test --workspace)
EOF
Loading