A peer-evaluation toolkit for Canvas group projects. Teams keep one shared grade on every deliverable, and individual contribution is handled separately: each student's teammates distribute a fixed pool of points, that becomes a multiplier on the team's earned credit, and the difference is posted as a signed number on a zero-point assignment. The mechanism is consequential, the arithmetic is auditable, and the syllabus can still honestly say "all teammates receive the same grade on the deliverable."
It is four things: a student-facing calculator, a set of Canvas templates, a scoring library with 62 tests, and one command that runs a whole evaluation round end to end.
Group work rewards the wrong people. The usual fixes fail in specific, familiar ways:
- A peer-evaluation slice of the course grade (worth a few percent) is not consequential enough to change behaviour. Doerry & Palmer measured this and abandoned it.
- Splitting the team grade per student breaks the promise that the team shares a grade, and turns every deliverable into a negotiation.
- A term-long "teamwork" score produces negative carry-over: one bad fortnight follows a student to the final grade with no way to recover.
- Free-form 1-5 ratings are not zero-sum, so raters inflate everyone and the signal disappears.
This toolkit implements the alternative: a fixed budget per rater, attached to a specific deliverable window, applied as a direct multiplier. On the origin course a student who was persistently rated at the floor lost 13.41 percentage points of final grade — enough to matter, bounded enough to survive an appeal.
Yes, if you run team projects of 2-4 students, with multiple graded deliverables across several weeks, and you already grade the team as a unit. Two or more evaluation rounds per project is where this earns its keep: students see a low factor early, and can act on it before the round that carries the most points.
No, if it is one group assignment, once. The setup cost (a group set, a survey, a zero-point assignment, a hosted calculator, a config file) is not worth it for a single deliverable — use a rubric line item and your judgement.
Also no, if your teams are larger than about six. The distribution tables, the anonymity guarantees, and the flag thresholds are all tuned for small teams; nothing breaks mathematically, but the guidance stops fitting.
You need Canvas Classic Quizzes (graded surveys). New Quizzes has no API that exposes per-question responses this way, so the round runner cannot read them.
-
A calculator.
allocator/index.html— one self-contained HTML file, no dependencies, no backend, nothing transmitted. The student picks a team size (2/3/4), can start from a named pattern ("One did clearly less", "One barely appeared" — every pattern already totals correctly), and types one row per teammate including themselves. It refuses to produce output until the points total exactly team size x 100, every address matches the institutional email form, and no address appears twice. Then it emits the block to paste. -
A survey, six free-text/numeric questions (
canvas/quiz-peer-evaluation-questions.json): the allocation, strengths per teammate, improvements per teammate, a required justification for any rating below 80 or above 120, a private note to the instructor, and a numeric self-check that accepts only 200, 300, or 400. Two attempts, so a bad total is fixable. -
A zero-point adjustment assignment (
canvas/assignment-0pt-adjustment.html) whose description states the arithmetic, the 0.60-1.10 bounds, and five worked outcomes for the same team grade. Most students score 0 here, and the page says so in the first sentence. -
A private report as a submission comment: how many teammates rated them, their mean, their factor, the multiplication, the posted adjustment, and their teammates' written comments with names stripped.
Per round, after the team deliverables are graded:
cd scripts
python3 peer_report.py tangible_draft # dry run: reads Canvas, writes nothing
That pulls the roster, the group set, the team scores on the covered deliverables, and every
ballot; reconciles mistyped email addresses against the roster; validates each ballot; computes a
factor per student; and writes a report plus a JSON row set to scripts/reports/. It prints a
summary: who did not submit, every flagged student, every allocation problem, every auto-corrected
address, every cross-team address it refused to merge, and the full table of adjustments it intends
to post.
You read that. You decide. Then:
python3 peer_report.py tangible_draft --post # writes the deltas and the comments
Grades land under a manual post policy, so nothing is visible until you release it in the
gradebook. The judgement calls the tool deliberately leaves to you: held ballots, UNANIMOUS_LOW
(every teammate independently rated someone below 80 — that is a conversation, not a number), teams
that came in undersized, and anything a student appeals.
| Component | File | What it is |
|---|---|---|
| Point allocator | allocator/index.html |
Student-facing zero-sum calculator. Vanilla JS, single file, no network calls. |
| Scoring core | scripts/ags/scoring.py |
Ballot validation, factor arithmetic, safety rails, instructor flags. Pure functions. |
| Reconciliation & reporting | scripts/ags/peerreview.py |
Email repair, free-text allocation parsing, budget/coverage checks, report rendering. Pure functions. |
| Canvas client | scripts/ags/canvas.py |
Minimal REST wrapper: pagination, throttle retries, token scrubbing in errors. |
| Round runner | scripts/peer_report.py |
The one command. Dry run by default; --post writes. |
| Allocator embedder | scripts/embed_allocator.py |
Injects the hosted calculator into the page and quizzes, with a preflight that refuses a URL that will not run scripts. |
| Canvas templates | canvas/*.html, canvas/*.json |
Assignment description, quiz description, the six questions, the fallback distribution tables. |
| Configuration | scripts/provision.example.yaml |
Course, teams, rounds, deliverables. Drives provision.py. |
| Tuning | scripts/config.example.yaml |
Factor bounds, pair policy, flag thresholds. Scoring knobs only. |
| Tests | scripts/test_scoring.py, scripts/test_peerreview.py |
62 tests (36 + 26). Readable as the spec. |
The scoring model in one line:
factor = clamp(mean of all ratings a student received / 100, floor 0.60, ceiling 1.10)
adjustment posted = round(team_earned x factor - team_earned, 2)
| Cost | Reality |
|---|---|
| Programming skill | None. Fill in scripts/.env and scripts/provision.yaml, then run three commands in Terminal. If you can edit a YAML file you can run this. |
| Software | Python 3, plus requests and python-dotenv. The tests need neither. |
| Accounts | A Canvas API access token you generate yourself (Account -> Settings -> New Access Token) — no admin rights needed. A place to serve one static HTML file; GitHub Pages is free and is what the toolkit assumes. |
| First-course setup | Budget an afternoon. Per project arc: one self-signup group set, one graded survey per round, one zero-point assignment per round, one guide page, assignment-group weights, and the round table in the config. |
| Per round, ongoing | Roughly ten minutes of machine time and reading, plus however long the flagged conversations take. The conversations are the real cost and the toolkit cannot remove them. |
| Student time | A few minutes with the calculator; the survey itself is short and carries small completion credit. |
The one structural decision that costs nothing to make and a lot to get wrong: the adjustment assignment must be worth zero points, and it should cover the whole project arc rather than just the final deliverable. Both claims are measured, not asserted:
| Choice | Measured effect on the origin course |
|---|---|
| Adjustment carries the deliverable's own point value | A team at 85% with a factor of exactly 1.00 fell to 54.24% in its assignment group — the delta's points entered the denominator (67 -> 105). A 0.60 factor then costs only 1.85 points of final grade. |
| Adjustment worth 0 points | The delta lands in the numerator only. Denominator untouched. |
| Covers the final two deliverables only (78 of 153 group points) | A persistent 0.60 student loses 6.76 percentage points of final grade. |
| Covers the whole arc (all 153 points) | The same student loses 13.41. Same number of Canvas objects, roughly double the consequence. |
Fifteen minutes is enough to decide whether this fits your course. It is not enough to deploy it — that is an afternoon, and it is docs/03-canvas-setup.md.
git clone <this repo> && cd peer-evaluation-toolkit
# 1. Run the spec. No Canvas, no token, no dependencies.
cd scripts && python3 -m unittest discover -p 'test_*.py' # 62 tests
# 2. Be a student for two minutes.
open ../allocator/index.html
- Read examples/worked-round.md — one real round, from what three students typed (messy on purpose: mixed separators, a bare user id with no domain) through to the comment each of them received.
- Read docs/01-pedagogical-model.md for why the model is shaped this way, then docs/02-canvas-architecture.md for why the zero-point assignment is load-bearing.
- If you are still in, docs/00-quick-start.md is the ordered build path.
allocator/index.html student-facing calculator (host this)
canvas/
assignment-0pt-adjustment.html adjustment assignment description
quiz-peer-evaluation-description.html survey description
quiz-peer-evaluation-questions.json the six questions, as authored
page-distribution-guide.html fallback distribution tables (no-JS path)
examples/assignment-group-weights.json the origin course's group weights
docs/ see below
examples/worked-round.md input -> output for one real round
scripts/
ags/scoring.py factor math + safety rails
ags/peerreview.py reconciliation, parsing, reports
ags/canvas.py REST client
peer_report.py run a round
embed_allocator.py embed the hosted allocator
provision.example.yaml course, teams, rounds -> provision.py
config.example.yaml scoring knobs (bounds, flags)
test_scoring.py test_peerreview.py 62 tests
LICENSE CONTRIBUTING.md
| Doc | Read it when |
|---|---|
| docs/00-quick-start.md | You have decided to adopt and want the ordered build path. |
| docs/01-pedagogical-model.md | You want to know why zero-sum, why per-deliverable, why a multiplier. |
| docs/02-canvas-architecture.md | Before you create a single assignment. Explains the zero-point trick and the weighting arithmetic. |
| docs/03-canvas-setup.md | You are building the group sets, surveys, assignments, and pages. |
| docs/04-running-a-round.md | A round has closed and you want to score it. |
| docs/05-scoring-reference.md | You need to explain a specific number to a specific student, or tune a threshold. |
| docs/06-canvas-field-notes.md | Canvas is lying to you. Empirically verified failures, several of which return HTTP success while doing nothing. |
| docs/07-adapting-to-your-course.md | Your course is not IMG 186 — which is to say, always. |
Stated plainly, because finding these out in week ten is worse.
- Teams of two are not anonymous. With no averaging, each partner can derive the other's rating
from their own posted result. This is arithmetic, not a policy choice. The default policy narrows
the factor range to 0.80-1.10 rather than pretending otherwise;
neutralize(force 1.00) andfull(normal clamp) are also selectable per round. peer_report.pyis not yet config-driven. The course id and the round table are constants at the top of the file, andconfig.example.yamldocuments the intended shape rather than being read by the runner. Adopting means editing that block.- Institution-specific assumptions. The email pattern (
letters + digits @nau.edu) is hard-coded in bothallocator/index.htmlandags/peerreview.py. The Canvas templates use DesignPLUS markup that degrades to plain HTML elsewhere but will not look the same. embed_allocator.pyreads a wiring file (scripts/backups/peer_eval_wiring.json) produced by the origin course's build scripts and not shipped here.- Classic Quizzes only, and Canvas will not accept grades in an unpublished course (403), so negative-score behaviour cannot be smoke-tested before you publish.
- The tool does not decide anything that matters. It holds suspicious ballots, raises flags, and refuses to guess. A held ballot, a unanimous low rating, or an appeal is yours to resolve.
The model is from Doerry and Palmer, Improving Efficacy of Peer-Evaluation in Team Project
Scenarios, ASEE Annual Conference, 2011 — the peer-evaluation schema developed for the
Design4Practice program at Northern Arizona University. The zero-sum
budget, the requirement that ratings be anonymous, the per-deliverable scope, the direct-multiplier
application (their "Version 4", after a percentage-of-grade "Version 3" proved insufficiently
consequential), the justification band, and the flat-ballot dysfunction profile are all theirs.
The tests in test_scoring.py pin the arithmetic to that paper's worked examples.
What is added here is the Canvas implementation: the zero-point adjustment mechanism, the safety rails (rescale limits, coverage checks, quorum bounds, pair policies), the reconciliation of messy student input, and the field notes on Canvas behaviour.
Built for one course — IMG 186, Game History and Theory, Northern Arizona University — and generalised afterwards. Every number in the examples is that course's: the 153 group points, the 40% group weight, the 85% team, the assignment ids, the arc names, the NAU email pattern. They are included because a measured number is more useful than an invented one, not because they will match your course. Recompute the consequence for your own weighting before you tell students what a factor costs; docs/02-canvas-architecture.md shows the arithmetic.
MIT licensed — see LICENSE. Contributions, especially new Canvas field notes and real examples of what students actually typed, are welcome: CONTRIBUTING.md.