From 8161b91db3b0f494026a8662f1a330f3ced05a7e Mon Sep 17 00:00:00 2001 From: Fran BlakIA Date: Mon, 10 Aug 2026 20:00:01 +0200 Subject: [PATCH] docs(readme): lead with why the system exists The intro described a hypothetical user watching 'a few thousand sources'. The real driver is stronger and more specific: a Hagalink project monitoring more than 10,000 RSS sources with keyword alerting, which is why the capacity benchmark tops out at exactly 10,000 and why the runtimes are split the way they are. Adds a Deployment section stating what Kubernetes buys over Compose here -- queue-depth autoscaling, which Compose cannot express -- and links the CI that applies the manifests to a real cluster. --- README.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 53 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 1b3f6e2..2b6f496 100644 --- a/README.md +++ b/README.md @@ -3,16 +3,34 @@ [![CI](https://github.com/FullFran/feedpulse/actions/workflows/ci.yml/badge.svg)](https://github.com/FullFran/feedpulse/actions/workflows/ci.yml) [![Smoke](https://github.com/FullFran/feedpulse/actions/workflows/smoke.yml/badge.svg)](https://github.com/FullFran/feedpulse/actions/workflows/smoke.yml) [![CodeQL](https://github.com/FullFran/feedpulse/actions/workflows/codeql.yml/badge.svg)](https://github.com/FullFran/feedpulse/actions/workflows/codeql.yml) +[![Kubernetes](https://github.com/FullFran/feedpulse/actions/workflows/k8s.yml/badge.svg)](https://github.com/FullFran/feedpulse/actions/workflows/k8s.yml) ![Node](https://img.shields.io/badge/node-24-3c873a) ![License](https://img.shields.io/badge/license-MIT-blue) FeedPulse is a multi-tenant RSS/Atom monitoring service. You register feeds and keyword rules; it polls each feed on its own schedule, deduplicates entries, matches them against -your rules, and delivers **one alert per article** over webhook, email and Telegram. It is -built for the person who has to watch a few thousand sources for a handful of specific -phrases and wants that to be a background service with an API, not a browser tab — a news -desk, a compliance or market-intelligence team, or anyone self-hosting a monitoring stack -on a single VPS. +your rules, and delivers **one alert per article** over webhook, email and Telegram. + +## Why it exists + +It was built for a Hagalink project that had to monitor **more than 10,000 RSS sources** +and alert on keyword matches across all of them. + +Hosted readers do not solve that problem. They are priced per seat, they are built for a +human reading a timeline, and keyword rules with multi-channel alerting are not what they +are for. At that source count the cost stops tracking the value, and you still do not have +an API to plug the matches into anything else. + +So the requirement was specific: **10,000 feeds, on infrastructure we control, with +alerting as a first-class output rather than a browser tab.** That number is why the +[capacity benchmark](#performance) tops out at exactly 10,000 feeds — it is the acceptance +criterion, not a round figure chosen to look good. It is also why the architecture looks +the way it does: three runtimes because the API, the scheduler and the fetching have +unrelated load profiles, and queues because ten thousand fetches do not fit inside a +request cycle. + +The service was later folded into a wider platform. This repository is the standalone +system. It is also a portfolio codebase, so it is written to be read: ports and adapters where a seam earns its keep, raw SQL you can review, decisions recorded in @@ -33,6 +51,7 @@ seam earns its keep, raw SQL you can review, decisions recorded in - [Rules](#rules) - [API](#api) - [Performance](#performance) +- [Deployment](#deployment) - [Engineering notes](#engineering-notes) - [Testing](#testing) - [Quality gates](#quality-gates) @@ -398,6 +417,35 @@ Read those numbers honestly: --- +## Deployment + +Two supported shapes, and the choice is a real one rather than a preference. + +**Docker Compose** (`compose.dokploy.yml`) is the simpler and, at low feed counts, the +cheaper option. It is what the project ran on first. + +**Kubernetes** ([`k8s/`](./k8s/)) exists because of one property Compose cannot express: +the worker's load is a function of **queue depth**, not of HTTP traffic. It blocks on +network I/O against slow feeds, so CPU stays low exactly while the backlog grows — a +CPU-based autoscaler would never fire. A [KEDA](./k8s/keda/) `ScaledObject` reads the +BullMQ waiting list in Redis and scales the worker on the signal that actually tracks the +work. At the 10,000-feed scale this was built for, that is the difference between a +backlog that drains and one that does not. + +The manifests are not decorative. `.github/workflows/k8s.yml` schema-checks them with +`kubeconform` and then applies them to a real cluster on every change: it builds the image, +side-loads it into kind, runs the migration Job, waits for all three runtimes, curls +`/health` and `/ready`, restarts the worker to exercise graceful shutdown, installs KEDA, +and waits for the `ScaledObject` to report `Ready=True`. + +The honest counterpart is tracked in [#24](https://github.com/FullFran/feedpulse/issues/24): +measuring the crossover point against a scale-to-zero PaaS and publishing the range where +Kubernetes is the **wrong** choice. + +See [k8s/README.md](./k8s/README.md) for the decisions and their reasoning. + +--- + ## Engineering notes Four places where the obvious implementation is wrong, and the repository says why.