Can a model estimate software effort from public data? A pre-registered blind test, with an honest negative result.
Reproducible backtest of the Metis software effort-estimation engine on open datasets. This repository is the Phase 0 gate (G0) of the Metis MVP by NaCode Studios: if the engine does not beat the standard baselines, and the human expert where a dataset records one, the product does not get built.
Read the report: Can a model estimate software effort from public data? A pre-registered blind test across nine datasets, and an honest negative result.
Two tracks, mirroring the engine's two channels:
| Track | Level | Channel | Datasets |
|---|---|---|---|
| A | Project | Tabular regression (GP / gradient boosting, quantile + CQR) | PROMISE (Desharnais, COCOMO81, China, Kitchenham, Maxwell, Albrecht), SEERA |
| B | Task (with text) | Semantic similarity (embeddings, k-NN, reranking) | Deep-SE, JOSSE, SiP; TAWOS for retrieval scale |
Metrics: PRED(25), MdAPE, and the empirical coverage of conformalized quantile intervals (CQR). Temporal splits only, never random. Full rules in reports/protocol.md, frozen before experiments run.
Mandatory baselines the engine must beat:
- median effort by category;
- linear regression of log(effort) on log(size);
- the human expert estimate recorded in the dataset (JOSSE, SiP).
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[models,dev]" -c constraints-g0.txt
make test
make status # which datasets are present in data/raw/ (status-a / status-b per track)
make reproduce-g0 # full G0 chain (REPORT.md §8): download → both channels → ceilings → testsconstraints-g0.txt pins the exact package versions the G0 numbers were
produced with; install with -c to reproduce them bit-for-bit.
Datasets are not committed: each has its own license and citation requirements. Sources are listed in src/metis_benchmark/datasets/registry.py.
- Metrics, baselines, temporal split (tested)
- Week 1: dataset download (10/10 gate datasets), EDA, data dictionary, baseline smoke test, protocol frozen
- Week 2: Track A GP + GBM quantile + CQR; rolling-origin estimator (v1.2); partial Track A verdict
- Week 3: Track A feature expansion (cocomo81, seera), mean-function GP, hierarchical pooling, honest feasibility ceiling → Track A final verdict. Not passing, 55% proven unreachable on public tabular data; closed best-effort; proceeding to Track B
- Track B, the semantic channel (JOSSE, SiP): bge embeddings + k-NN Nadaraya-Watson + cross-encoder + learned-regressor ceiling → Track B verdict. Not passing. Text does not predict logged effort cross-project (ceiling 15-20%)
- Full G0 verdict: NO-GO on the public gate. Both channels proven below 55%; the intervals are honest; expert-assist and a proprietary-data pilot are the evidence-based next steps
- Post-G0, not gate-carrying — the same question asked of three corpora: does correcting an estimator's own estimate, from its own history, beat leaving it alone? All three say no. - SiP: one company, ten years, 12,299 estimates. A measured tie at n=4,533 once the exact ties are dropped, and a single global recalibration factor is significantly worse (−0.8% PRED(25), McNemar p=2.6e-05). The expert's headline 41.6% is also 26% logging convention — drop the rows where the actual equals the estimate to the digit and it falls to 20.7% - Apache JIRA: ~2,100 estimates across ~180 independent teams, mined fresh. Here the estimator is heavily biased (mean log-ratio −1.05), and the centre of the correction decides everything: the mean-based factor costs −7.2% (p=3e-07), the median-based one is a clean tie (+0.4%, p=0.66) - TAWOS: 43,108 story-pointed issues across 28 projects. A team's own point-to-outcome rate beats the estimate-blind floor by +2.2% (p=3e-15), and a model given the same points plus context does not beat the plain conversion. On the small correctly-labelled effort subset the points stop working entirely
The benchmark's one honest gap is the regime it could not test: a single organization's own delivery history. Every gate split here was a cross-organization cold start on public data, the hardest case, and precisely not where an estimation tool is deployed. Whether calibrated estimation clears the bar on one team's own consistent history is an open, testable question, and the obvious next experiment.
Slices of it have since been measured on public data, and all came back negative. SiP is one company's decade of logged tasks; Apache JIRA is ~180 independent teams; TAWOS is 28 projects with a story-point scale each. In every one of them the estimator that already exists is the one to beat, and none of the model classes tried beats it. That is task and issue granularity, not the project-level history a pilot would bring — but it is evidence, and it is the reason the offer below leads with the calibrated interval rather than with an expected accuracy gain.
If your organization has that history, we would like to run a pilot.
What qualifies: completed projects (or a task backlog) with recorded actual effort (developer hours or cost), logged consistently by one team. Enough to train and validate on a time-ordered split, so realistically some dozens of finished projects, or a few hundred logged tasks.
What we would do: apply the same discipline you can read here (pre-registered thresholds, temporal splits, leakage audit, honest feasibility ceiling, calibrated intervals) to your data, under NDA, and report straight whether it passes on your history. No cherry-picked demo.
Reach out: info@nacodestudios.it.
This benchmark builds on public datasets by their respective authors: the
PROMISE repository (CC-BY, Zenodo mirrors), SEERA (PROMISE/ACM 2020),
JOSSE (Alhamed & Storer 2022), TAWOS (MSR 2022, DOI 10.5522/04/21308124,
Apache-2.0, research use only), Deep-SE (Choetkiertikul et al. 2019), and
SiP (Jones & Cullum 2019). The post-G0 Apache corpus is mined from the Apache
Software Foundation's public issue tracker by scripts/mine_apache_jira.py. It also builds on
Conformalized Quantile Regression (Romano, Patterson, Candès 2019) via the
MAPIE library. Cite the original sources when reusing the data.
Code: MIT. Datasets: see their respective licenses.
Brand assets (the tetradrachm mark, the wordmark, and the colour and type tokens) are in docs/brand.