AlgoJudge is open-source, self-hosted software for programming contests and courses, with automatic evaluation of submitted solutions.
This is the component that does the evaluating: isolated execution and marking, for AlgoJudge.
docs.algojudge.pl is written for somebody who does not have this source open. This README is the other half: what the repository is, and how to build, run and change it.
/en/runner/ |
what a Runner is, isolation and limits, the four languages, problem types, routing by tags, and every configuration key |
/en/protocol/ |
the contract this Runner and the Server share, written for a reader rather than as a specification |
Both are English. Polish covers /client/ and /install/.
It judges. This Runner registers, is approved, authenticates, claims jobs, holds and renews a lease, downloads and verifies packages, unpacks them, compiles and runs a submission in isolated containers, scores it by groups, and reports the mark with the compiler's log and a per-test table attached.
Two problem types, and eighteen toolchains. standard-io@1 in C, C++ and
Python — the table is crates/aj-standard-io/src/language.rs — and
output-only@1, where the participant uploads answers rather than a program.
The Server holds nothing about either type. A problem type costs one match
arm in crates/aj-runner/src/run.rs and a crate; claiming, leasing, the package
cache, integrity and reporting are shared.
The Runner is the component that actually runs untrusted code. It is deliberately interchangeable: the Server must never depend on any particular Runner implementation, and several may coexist.
The Server–Runner contract is at v1.1, amended seven times. Thirty
conformance cases hold the Server to it, in
AlgoJudge.Server.Tests/RunnerConformanceTests.cs; the protocol is written up
for a reader at
/en/protocol/.
Three properties of it shape everything here:
- The Runner opens an outbound connection. The Server never calls a Runner, which is what lets one sit behind a domestic router with no public address.
- There is no socket for a Runner. The queue is polled, and an empty one
answers
204— a normal state, not an error. This is simpler than a socket, survives a dropped connection with no reconnection logic, and cannot deliver a job twice. - The Runner is stateless apart from a package cache. One that dies mid-evaluation resumes nothing and nobody comes back for that work. The Server's lease is the whole recovery story: it expires, the job returns to the queue, and the Runner that woke up late is refused rather than allowed to overwrite whoever holds it now.
Pushing a v* tag publishes five images under one version, to GitHub's
container registry:
docker pull ghcr.io/algojudge/algojudge-runner:0.1.0
docker pull ghcr.io/algojudge/lang-gcc:0.1.0
docker pull ghcr.io/algojudge/lang-clang:0.1.0
docker pull ghcr.io/algojudge/lang-python:0.1.0
docker pull ghcr.io/algojudge/lang-pypy:0.1.0All five, because a Runner without the language images judges nothing, and because a release is tested as one set: the images and the binary are built, checked and pushed together, or not at all.
0.1.0, 0.1, 0 and latest point at the same image; a prerelease
(v0.1.0-rc.1) publishes only its own tag, so nothing moving ever points at a
release candidate.
The Runner is linux/amd64 only. That is not an oversight: cgroup v2 on amd64 is
what the measurement rests on, and a submission's container has to match the
architecture of the host that runs it.
A deployment names the language images explicitly, because the built-in defaults
(algojudge/lang-gcc:local) are what the development stack builds locally:
AJ_Sandbox__Image__Gcc=ghcr.io/algojudge/lang-gcc:0.1.0
AJ_Sandbox__Image__Clang=ghcr.io/algojudge/lang-clang:0.1.0
AJ_Sandbox__Image__Python=ghcr.io/algojudge/lang-python:0.1.0
AJ_Sandbox__Image__Pypy=ghcr.io/algojudge/lang-pypy:0.1.0Each is independent: anything left unset keeps its compiled-in default, so an
operator republishing one image says so in one line.
AJ_Sandbox__Image__Cpp is the old name for …__Gcc and is still read.
Pin the same version the Runner is, unless there is a reason not to: that pairing is what the release was tested as.
.env.example lists every variable this Runner reads, with what each one
defaults to and which of them cannot be turned off by writing false. A test
compares it against the source, so a key added to the code and not to that file
reddens the gate.
Rust, one static x86_64-unknown-linux-musl binary with every backend
compiled in and chosen by configuration, shipped in a minimal image.
linux/amd64; cgroup v2 is required and is checked at start.
A .deb with a systemd unit is supported and not preferred in production:
the Runner needs a container runtime anyway, so a package that suggests
otherwise invites installing it where it cannot work.
Rust does not have to be installed to work on this. cargo runs in a pinned
container:
./x build
./x test
./x fmt
./x clippyNothing here is claimed without a test that runs it.
| Suite | What it proves |
|---|---|
cargo test |
the pure parts — the checker contract, comparison, scoring, the archive defences |
--test conformance |
the wire protocol, against a real Server, with this Runner as the client |
--test adversarial |
the isolation, against real containers, one case per attack |
--test judging |
a real submission compiled, run and marked, against the committed package |
--test end_to_end |
the whole product: a manager publishes, a participant submits, this Runner judges, and what is asserted is what the participant reads |
The last four need a container runtime and are #[ignore]d so an ordinary
./x test stays fast. All of them run in CI.
Sibling containers. The Runner is trusted and holds the container runtime's
socket; the containers that run submissions never do. Each step of the pipeline
— compile, run, check — has its own profile, and every one of them drops all
capabilities, disables the network, runs as an unprivileged user, and caps
memory, processes, CPU, wall time and output. The read-only root filesystem is
the one that is not universal: the two build steps ask for a writable root,
because a compiler has to put the program it made somewhere collect can read
it back from. They still get no writable host path, and the layer dies with
the container. Every step that runs a submission is read-only.
One container per test, never reused.
docs/SECURITY.md is written for the person deploying
this: what contains a submission, where the boundary is, and what the evaluation
host is therefore assumed to be — no secrets on it, reproducible, nothing else
running.
A host the Runner cannot measure on is refused at start, and there are three
ways to be one: cgroup v1, a cgroup driver the Runner knows neither of, and a
cgroup tree it cannot use. The limits are enforced on all three — what is
missing is the reading, and a time limit is decided on processor time read
from cpu.stat in a cgroup the sandbox is started under.
Both cgroup drivers are supported, cgroupfs and systemd, and the Runner
chooses from what the daemon reports. Under cgroupfs it makes a cgroup per run
and removes it; under systemd — the default on virtually every Linux server —
systemd owns the cgroup, so the Runner keeps one slice for its whole life and
takes each run's numbers as the change across it. Either way the container needs
the host's cgroup tree mounted writable, --cgroupns=host, and to run as root.
docs/CGROUP_V2.md compares them.
AJ_Sandbox__AllowUnmeasured starts anyway and says so at ERROR on every
start. It makes the process come up; it does not make it judge.
Every submission is untrusted and assumed hostile: attempts to read system files and secrets, write outside the working directory, spawn processes, fork-bomb, exhaust memory or CPU, produce unbounded output, reach the network, survive past the end of a test, or interfere with another job.
These are held to by an adversarial suite that runs in CI, one case per attack with the outcome it must produce. A suite that is not a gate is not a gate.
- AlgoJudge-Server — jobs, packages, results
- AlgoJudge-Client — the web frontend
- AlgoJudge-External-Runner —
a second Runner, forwarding submissions to external judging systems. It
consumes this repository's
aj-protocolcrate over Git, pinned to a revision in itsCargo.toml, so a breaking change here is a change two repositories have to agree on - AlgoJudge-Ops — the production Compose stack, which is what runs this image at an installation
- AlgoJudge-Docs — the source of the documentation site linked under Documentation above
Open an issue saying what you expected, what happened, and how to reproduce it.
Or open a pull request against main: one subject per pull request, with a note
on what changes and why.
By contributing you agree that your work is licensed under the terms below.
This project is licensed under the MIT License. See LICENSE.
Authors are listed in AUTHORS.txt.