Open-source self-hosted comments

Comparison table for open-source self-hosted commenting servers (lisakov.com/projects/open-source-comments/). Inspired by staticsitegenerators.net.

Workflow

• The data are stored in data.yaml. This file is edited manually.

• get_data.py reads the GitHub and GitLab repositories from data.yaml, fetches the current repository metadata and latest commit through their APIs, then updates data.yaml for the following:

  • displayed stars from the repository with the higher star count,
  • combined star growth over the last 30 days,
  • latest commit date,
  • creation date,
  • license.

• Historical repository-derived data are stored in apigh/history.json. It keeps only value changes for the fields used by the page (stars, stars_total, stars_github, stars_gitlab, open_issues, created, license, last_commit) instead of storing raw API responses for every repository every day.

• yaml_2_js.py converts data.yaml to data.js (it defines three variables — osc_data, cols and col_keys).

• history_2_js.py reduces apigh/history.json to star-history.js, the star series the interactive chart reads: only the stars field, without the samples plot-stars.py also rejects, downsampled to the points that are visible at chart resolution.

• index.html is the page structure; index.md holds all of its prose. md_to_html.py renders each <!--osc:NAME--> section of index.md into the matching slot in index.html, and computes the system and attribute counts from the data. The table, the column popover, the project record and the chart are drawn by js/osc.js (with js/osc-lib.js) from data.js and star-history.js — plain scripts, no framework and no build step.

• plot-stars.py reads apigh/history.json, plots stars vs. time for selected projects, and writes stars-v-date.svg. The page shows the interactive chart instead and falls back to this SVG inside <noscript>.

• The webpage is updated daily via cron. updater.sh runs get_data.py, md_to_html.py, yaml_2_js.py, plot-stars.py, and history_2_js.py, then deploys the updated files and pushes the repository. Repository redirects are followed automatically and their canonical URLs are saved back to data.yaml.

• The chain from cron to the webroot is: the crontab entry names ~/.local/bin/open-source-comments-update, which is cron_wrapper.sh installed under that name; it sets OSC_SCRIPT_DIR (~/.local/share/open-source-comments) and hands over to the updater.sh installed there; that operates on the checkout in ~/open-source-comments and deploys to /var/www/lisakov.com/projects/open-source-comments.

• The scripts are executed from OSC_SCRIPT_DIR rather than from the checkout so that the git pull at the start of a run cannot replace code while it is executing. updater.sh reinstalls the pulled Python there on every run, so a new build step takes effect immediately. It cannot do that for itself — bash is already reading the installed copy — so if updater.sh changed it stops and asks for ./install_scripts.sh to be run once.

• install_scripts.sh installs every tracked script into OSC_SCRIPT_DIR and the wrapper into OSC_BIN_DIR (~/.local/bin). It is idempotent and reports what it changed. Run it after changing updater.sh, cron_wrapper.sh or itself; the Python looks after itself.

• Any non-routine event for an individual repository — including an API retry, redirect, invalid response, suspicious identity change, or persistent request failure — sends an email immediately. Affected repositories retain their last trusted values and get an asterisk next to the star count; its tooltip contains the warning date and details. Other repositories continue to update and the page is still published. A daily star decrease is treated as non-routine only when it reaches 20 stars; smaller decreases do not produce an asterisk. The marker is removed after the next completely clean update for that repository.

• updater.sh exits on a failed step and writes the failure to stderr and the system log. It also sends a direct email to lisakov57@gmail.com; set OSC_ALERT_EMAIL in the cron environment to override that address. Cron's standard MAILTO remains supported as an alternative. Concurrent runs are rejected, generated files are backed up during the transaction, all outputs are validated before commit/push, and deployment starts only after a successful push. A global failure rolls generated files back and prevents deployment.

• Generated YAML history, JavaScript, HTML, and SVG files use atomic replacement so an interrupted write does not leave a truncated production artifact.

• If the server's system DNS is unavailable, the updater starts a loopback-only CONNECT proxy that resolves hosts through a fallback DNS server. Git and API HTTPS traffic keep normal TLS hostname verification. The first activation during a continuous DNS outage sends an email; later runs only log it. The notification is armed again after system DNS recovers.

Dependencies

Install Python dependencies with:

python3 -m pip install -r requirements.txt

How to view the page locally

Clone the repo and open index.html in your browser — the page has no build step and no module imports, so it works straight from the filesystem. Some browsers refuse to load fonts over file://; the page then falls back to a system sans and is otherwise unchanged.

The page loads nothing from a third party. Everything but the comment thread is served from this directory; the thread comes from comments.lisakov.com, the site's own Isso instance.

css/fonts/archivo.woff2 is Archivo subset to the characters the page and data.yaml actually render, with the variable weight axis clamped to the 400-800 the design uses — 18 kB rather than the 35 kB of the stock Latin subset. css/fonts/archivo-subset.txt lists exactly what it covers, and validate_outputs.py reports any character in data.js that falls outside it (a fallback glyph, not a failure). Text outside that set — Cyrillic in comments, for instance — renders in the reader's system sans, as it did before. To widen the subset, request a new file from Google Fonts with the extra characters in the text= parameter and update the .txt alongside it.

To preview it at the URL it has in production, run the lisakov.com Hexo site with hexo s. The middleware that does it is tools/hexo-osc-preview.js in this repository; the Hexo site keeps a short scripts/osc-preview.js that requires it and passes in serve-static, so the behaviour is versioned here rather than in that site, which is not a git repo. Point it at a different checkout with OSC_DIR=/path/to/open-source-comments hexo s.

A Hexo scripts/ file is dev-server tooling: it never becomes part of the generated site, so hexo generate / hexo deploy has nothing to publish for it. Pulling this repository is what updates the preview.

That script also proxies the Isso API, so the real comment thread loads in the local preview — Isso answers CORS only for the lisakov.com origin, and the proxy makes the call same-origin instead. It is read-only: posting, editing and voting are refused locally so a preview cannot write to the live comment database. Turn the proxy off with OSC_ISSO_PROXY=off.

To change the prose, edit index.md and run python3 md_to_html.py. Sections in index.md are delimited by <!--osc:NAME--> markers and land in the matching <!--osc:NAME-->…<!--/osc:NAME--> slot in index.html, which the script overwrites.

After modifying data.yaml, run python3 yaml_2_js.py to update data.js, and python3 history_2_js.py to update star-history.js.

Run reliability tests with:

python3 -m unittest -v test_get_data.py test_reliability.py

TODO

• Check and add the information to make the table useful. I would appreciate adding a missing demo.

• Where do I find a number of opened and closed issues? For example, https://api.github.com/users/posativ/isso has open_issues_count and open_issues, both equal to 131, whereas there are 110 issues and 21 PR.

• apigh/<date> folders store a lot of information which is never used. Need to extract only needed info from the files and remove the rest.

• Plot stars vs. time for several top commenting systems. Update the plot daily automatically.

• Get rid of yaml, convert data.yaml to data.json

Contribution

Contributions are welcome. Fork the repo and send PR, submit an issue, or leave a comment at the website.