Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.git
.github
.venv
__pycache__
*.py[cod]
.pytest_cache
.ruff_cache
.mypy_cache
.env
.env.*
!.env.example
data
tests
*.md
.snyk
docker-compose*.yml
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ GOOGLE_CX_ID=
# ScraperAPI — redirect/fetch helper only; do not scrape full articles
SCRAPER_API_KEY=

# Container user; must own ./data on the host (check with `id -u` / `id -g`)
APP_UID=1000
APP_GID=1000

# Schedule in Australia/Brisbane
TZ=Australia/Brisbane
DIGEST_HOUR=7
Expand Down
5 changes: 5 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,8 @@ updates:
directory: "/"
schedule:
interval: "monthly"

- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "monthly"
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ Add or update tests for code you change. Fix failures before finishing.
- Do **not** add full-page article scraping; ScraperAPI is only for exceptional fetch helpers (e.g. redirect resolution), never as a content source
- Own sites (`rummerlab.com`, `jodierummer.com`, `physioshark.org`) stay excluded
- Seen URLs live in `data/seen.json` — do not wipe production state casually
- The container runs non-root (`APP_UID`/`APP_GID` in compose, default 1000). The `./data` bind mount must be writable by that uid, or `seen.json` can't be saved and items are re-emailed

## README and docs

Expand Down
36 changes: 26 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,20 +1,36 @@
FROM python:3.12-slim
# syntax=docker/dockerfile:1

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM: Non-root user (uid 1001) cannot write to the bind-mounted ./data directory

The image now runs as appuser (uid 1001) and only chowns /app/data inside the image. docker-compose.yml bind-mounts ./data:/app/data, and that mount hides the chowned directory. On a Linux host, ./data is owned by the host user (commonly uid 1000) or by root, so uid 1001 gets no write access. data/seen.json then can't be saved, so seen-URL state is lost or the run errors, and the digest re-emails the same items every day. Fix: make the container uid configurable and match the host, for example user: "${UID:-1000}:${GID:-1000}" in compose. Alternatively, document a required chown 1001:1001 data step, or use a named volume.

⚠️ Could not locate exact line in the diff. The bug is at Dockerfile:70.

Additional Locations

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in bb80a15. The image user is now uid/gid 1000, and compose runs as ${APP_UID:-1000}:${APP_GID:-1000} (documented in .env.example). The README has a one-time chown step for existing root-owned data/ directories left by the old root image. I checked locally that the container runs as 1000 and that save_seen/load_seen round-trip under /app/data.


FROM python:3.12.16-alpine3.24 AS build

WORKDIR /src

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

RUN --mount=type=bind,source=requirements.txt,target=requirements.txt \
pip install --no-cache-dir --no-compile -r requirements.txt

FROM python:3.12.16-alpine3.24 AS runtime

WORKDIR /app

ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
TZ=Australia/Brisbane

RUN apt-get update \
&& apt-get install -y --no-install-recommends tzdata \
&& rm -rf /var/lib/apt/lists/*
TZ=Australia/Brisbane \
PATH="/opt/venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# uid/gid 1000 matches the usual first host user, so the ./data bind mount stays
# writable; compose can override it via APP_UID/APP_GID when the host differs.
RUN --mount=type=cache,target=/etc/apk/cache,sharing=locked \
apk add tzdata \
&& addgroup -S -g 1000 appgroup \
&& adduser -S -u 1000 -G appgroup appuser \
&& mkdir -p /app/data \
&& chown appuser:appgroup /app/data

COPY src ./src
COPY --from=build --chown=appuser:appgroup /opt/venv /opt/venv
COPY --chown=appuser:appgroup src ./src

RUN mkdir -p /app/data
USER appuser

CMD ["python", "-m", "src", "--schedule"]
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ docker compose run --rm digest python -m src --dry-run

Seen URLs are stored in `data/seen.json` so repeats are not emailed again.

The container runs as a non-root user (`APP_UID`/`APP_GID`, default `1000:1000`), and that user must be able to write `./data`. If your host user isn't 1000, set `APP_UID`/`APP_GID` in `.env` to the output of `id -u` / `id -g`. Images up to v0.0.2 ran as root, so an existing `data/` may be root-owned. Fix it once before upgrading:

```bash
sudo chown -R "${APP_UID:-1000}:${APP_GID:-1000}" data
```

## Local tests

```bash
Expand Down
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ services:
digest:
build: .
container_name: rummerlab-media-alerts
# Must own ./data on the host, or seen.json can't be saved and items repeat.
user: "${APP_UID:-1000}:${APP_GID:-1000}"
env_file:
- path: .env
required: false
Expand Down
11 changes: 9 additions & 2 deletions src/normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,19 @@
}


def _is_google_host(host: str) -> bool:
# Require an exact google.com label (or subdomain). Plain endswith("google.com")
# would also match attacker hosts like "evilgoogle.com".
return host == "google.com" or host.endswith(".google.com")


def unwrap_google_url(url: str) -> str:
parsed = urlparse(url)
host = (parsed.hostname or "").lower()
if host.endswith("google.com") and parsed.path == "/url":
if _is_google_host(host) and parsed.path == "/url":
target = parse_qs(parsed.query).get("q", [""])[0]
if target.startswith("http"):
# Require a real http(s) scheme; startswith("http") accepts "httpevil://…".
if target.startswith(("http://", "https://")):
return target
return url

Expand Down
11 changes: 11 additions & 0 deletions tests/test_normalize.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ def test_unwraps_google_redirect():
assert unwrap_google_url(wrapped) == "https://www.abc.net.au/news/story"


def test_rejects_lookalike_google_host():
# endswith("google.com") alone would treat this as Google.
spoofed = "https://evilgoogle.com/url?q=https://www.abc.net.au/news/story"
assert unwrap_google_url(spoofed) == spoofed


def test_rejects_non_http_google_target():
wrapped = "https://www.google.com/url?q=httpevil://example.com/payload"
assert unwrap_google_url(wrapped) == wrapped


def test_strips_utm():
url = "https://www.theguardian.com/environment/story?utm_source=rss&utm_medium=feed"
assert canonicalize_url(url) == "https://www.theguardian.com/environment/story"
Expand Down
Loading