Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions deploy/release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,77 @@ if [ ! -s /opt/crypto-sentiment/data/orchestrator_state.json ]; then
exit 1
fi

TRENDING_SCOUT_IMAGE="${TRENDING_SCOUT_IMAGE:-panicradar-trending-scout:latest}"
TRENDING_SCOUT_PIN_NAME="${TRENDING_SCOUT_PIN_NAME:-panicradar-trending-scout-image-pin}"
TRENDING_SCOUT_PIN_NEXT_NAME="${TRENDING_SCOUT_PIN_NAME}-next"
TRENDING_SCOUT_IMAGE_ID=""

# The scout runs from root's hourly cron and is not built by this release. Pin
# the exact image ID used by its mutable tag before broad Docker cleanup so an
# unrelated application release cannot remove the scheduler's local image.
refresh_trending_scout_pin() {
if ! docker image inspect "$TRENDING_SCOUT_IMAGE" >/dev/null 2>&1; then
echo "ERROR: Required trending scout image is missing: $TRENDING_SCOUT_IMAGE"
echo "Restore the image before deploying; proceeding would leave hourly trends stale."
exit 1
fi

TRENDING_SCOUT_IMAGE_ID=$(docker image inspect --format '{{.Id}}' "$TRENDING_SCOUT_IMAGE")

# Create and verify the replacement first so the current image stays
# protected even while an obsolete pin is replaced.
docker rm -f "$TRENDING_SCOUT_PIN_NEXT_NAME" >/dev/null 2>&1 || true
docker run -d --name "$TRENDING_SCOUT_PIN_NEXT_NAME" --restart unless-stopped \

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pin container is created with no resource constraints. Since it's a long-running --restart unless-stopped container living in the root daemon, add --memory and --cpus limits (e.g. --memory 8m --cpus 0.01) for defense-in-depth against accidental resource consumption / a misconfigured entrypoint. The sleep 3600 loop is negligible in normal operation, but limits make the intent explicit.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in ed05892: the placeholder pin is now capped at 8 MiB and 0.01 CPU; its running state is also verified.

--memory 8m --cpus 0.01 \
--entrypoint /bin/sh "$TRENDING_SCOUT_IMAGE_ID" \

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You set --label panicradar.role=trending-scout-image-pin but this label is never consumed anywhere (the prune commands don't filter by it, and verify uses container/image IDs, not labels). If the goal is to make the pin discoverable or prune-exclusive, that filtering isn't wired up; otherwise the label is dead metadata. Consider documenting its purpose or dropping it.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in ed05892: removed the unused label. The stable container name is sufficient for operator discovery.

-c 'while :; do sleep 3600; done' >/dev/null

if ! docker inspect "$TRENDING_SCOUT_PIN_NEXT_NAME" >/dev/null 2>&1; then
echo "ERROR: Replacement trending scout pin container was not created"
exit 1
fi
PINNED_NEXT_IMAGE_ID=$(docker inspect --format '{{.Image}}' "$TRENDING_SCOUT_PIN_NEXT_NAME")
PINNED_NEXT_RUNNING=$(docker inspect --format '{{.State.Running}}' "$TRENDING_SCOUT_PIN_NEXT_NAME")
if [ "$PINNED_NEXT_IMAGE_ID" != "$TRENDING_SCOUT_IMAGE_ID" ] || \
[ "$PINNED_NEXT_RUNNING" != "true" ]; then
docker rm -f "$TRENDING_SCOUT_PIN_NEXT_NAME" >/dev/null 2>&1 || true
echo "ERROR: Replacement trending scout pin is not running on the expected image ID"
exit 1
fi

docker rm -f "$TRENDING_SCOUT_PIN_NAME" >/dev/null 2>&1 || true
docker rename "$TRENDING_SCOUT_PIN_NEXT_NAME" "$TRENDING_SCOUT_PIN_NAME"
echo "Pinned trending scout image: $TRENDING_SCOUT_IMAGE_ID"
}

verify_trending_scout_pin() {
if ! docker image inspect "$TRENDING_SCOUT_IMAGE" >/dev/null 2>&1; then
echo "ERROR: Trending scout image was removed during cleanup: $TRENDING_SCOUT_IMAGE"
exit 1
fi

CURRENT_SCOUT_IMAGE_ID=$(docker image inspect --format '{{.Id}}' "$TRENDING_SCOUT_IMAGE")
if ! docker inspect "$TRENDING_SCOUT_PIN_NAME" >/dev/null 2>&1; then
echo "ERROR: Trending scout pin container is missing: $TRENDING_SCOUT_PIN_NAME"
exit 1
fi
PINNED_SCOUT_IMAGE_ID=$(docker inspect --format '{{.Image}}' "$TRENDING_SCOUT_PIN_NAME")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker inspect --format '{{.Image}}' "$TRENDING_SCOUT_PIN_NAME" will return a non-zero exit with a cryptic docker error if the pin container does not exist (e.g. it was evicted or never created), rather than your intended 'no longer protects' message. Since set -e (if enabled) would abort here, consider a docker inspect ... >/dev/null 2>&1 || { echo 'ERROR: pin container missing'; exit 1; } guard for a clearer failure.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied in ed05892: verification now explicitly reports a missing pin container before inspecting its image, and checks that the pin remains running.

PINNED_SCOUT_RUNNING=$(docker inspect --format '{{.State.Running}}' "$TRENDING_SCOUT_PIN_NAME")
if [ "$CURRENT_SCOUT_IMAGE_ID" != "$TRENDING_SCOUT_IMAGE_ID" ] || \
[ "$PINNED_SCOUT_IMAGE_ID" != "$TRENDING_SCOUT_IMAGE_ID" ] || \
[ "$PINNED_SCOUT_RUNNING" != "true" ]; then
echo "ERROR: Trending scout pin no longer protects the scheduled image"
exit 1
fi
}

# ========== DISK SPACE CHECK & CLEANUP ==========
echo "Pre-cleanup disk usage:"
df -h /
refresh_trending_scout_pin
docker system prune -af || true

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker system prune -af || true swallows all errors from the prune. While the subsequent verify_trending_scout_pin (line 103) provides the real safety check, a prune failure here (e.g. daemon error) is masked and could hide genuine issues unrelated to the scout pin. This is pre-existing behavior, but worth noting that verify only checks the scout image, not general prune health.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed in this focused fix: the best-effort prune behavior predates this PR and changing it would broaden deployment semantics. The new guard still fails closed if the scout image or pin is affected.

docker builder prune -af || true
verify_trending_scout_pin

echo "Post-cleanup disk usage:"
AVAIL_KB=$(df -kP / | awk 'NR == 2 {print $4}')
Expand Down Expand Up @@ -489,6 +555,7 @@ fi
# Stopped rollback containers still protect their known-good images here.
echo "Cleaning up unused images..."
docker image prune -af
verify_trending_scout_pin
df -h /

# ========== VERIFY DEPLOYMENT ==========
Expand Down
26 changes: 26 additions & 0 deletions tests/test_deploy_workflow.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,29 @@ def test_release_script_is_shell_parseable_and_expression_free() -> None:
capture_output=True,
text=True,
)


def test_release_refreshes_and_verifies_the_trending_scout_pin() -> None:
release_script = RELEASE_SCRIPT_PATH.read_text()

assert "TRENDING_SCOUT_IMAGE" in release_script
assert "TRENDING_SCOUT_IMAGE_ID=$(docker image inspect" in release_script
assert 'docker run -d --name "$TRENDING_SCOUT_PIN_NEXT_NAME"' in release_script
assert "--memory 8m --cpus 0.01" in release_script
assert "--label panicradar.role=trending-scout-image-pin" not in release_script
assert 'docker inspect --format \'{{.Image}}\' "$TRENDING_SCOUT_PIN_NAME"' in release_script
assert "Trending scout pin container is missing" in release_script

refresh_call = release_script.index("\nrefresh_trending_scout_pin\n")
system_prune = release_script.index("docker system prune -af")
post_deploy_prune = release_script.index("docker image prune -af")
verify_after_system_prune = release_script.index(
"\nverify_trending_scout_pin\n", system_prune
)
verify_after_post_deploy_prune = release_script.index(
"\nverify_trending_scout_pin\n", post_deploy_prune
)

assert refresh_call < system_prune
assert verify_after_system_prune < post_deploy_prune
assert verify_after_post_deploy_prune > post_deploy_prune
Loading