Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
f42d959
Add the conformance suite, its manifest, and the adapter protocol
johnml1135 Sep 1, 2026
4f5aeeb
conformance: tolerate CS0117 in the single-file source census
johnml1135 Sep 2, 2026
25ddf91
conformance: pin two behaviours a port diverged on
johnml1135 Sep 2, 2026
853168d
docs: specify memoized conformance oracle
johnml1135 Sep 3, 2026
be851e3
docs: plan memoized conformance default
johnml1135 Sep 3, 2026
100d7be
conformance: pin cross-table root respelling as its own construct
johnml1135 Sep 3, 2026
4823a05
test: make memoization conformance default
johnml1135 Sep 4, 2026
43af40e
conformance: mark every fixture's FieldWorks producibility in words.y…
johnml1135 Sep 5, 2026
e2688c6
conformance: feature-gating-breadth uses an ordinary rule for past te…
johnml1135 Sep 5, 2026
2a2a874
test: update pinned ledger counts for the feature-gating-breadth conv…
johnml1135 Sep 5, 2026
7a4ec94
conformance: regenerate coverage.csv for the feature-gating-breadth c…
johnml1135 Sep 5, 2026
bd13eba
conformance: add a FieldWorks source witness and phoneme-mutation man…
johnml1135 Sep 5, 2026
2b520cc
conformance: correct the marks that a rule-order measurement and a lo…
johnml1135 Sep 5, 2026
f730e29
fix: four fieldworks_producible marks were wrong, verified against HC…
johnml1135 Sep 6, 2026
806b3be
fix: simultaneous-epenthesis-cascade header comment was stale, not th…
johnml1135 Sep 6, 2026
09f001a
docs: record and reproduce the morphologicalRuleOrder dependence meas…
johnml1135 Sep 6, 2026
cf14cad
feat: three gaps in the FieldWorks producibility table itself
johnml1135 Sep 6, 2026
e3751c6
fix: strip the coverage profiler from conformance child processes
johnml1135 Sep 6, 2026
2c87903
fix: make the coverage-profiler strip the only way to spawn a child p…
johnml1135 Sep 6, 2026
30216cd
convert morphotactic-attribute-breadth to fieldworks_producible: true
johnml1135 Sep 6, 2026
baa401f
convert fusional-realizational-morphology family-blocking to a produc…
johnml1135 Sep 6, 2026
435e6c9
docs: engine-only fixture retirement analysis (no deletions)
johnml1135 Sep 6, 2026
17f15d7
WIP: post-conversion derived-artifact regeneration, unreviewed
johnml1135 Sep 6, 2026
ea3b211
fix: restore morphotactic-attribute-breadth's silently-dropped covera…
johnml1135 Sep 6, 2026
b30540b
test: update pinned coverage counts after the family-blocking conversion
johnml1135 Sep 6, 2026
6ff5400
docs: keep suffixing-extension-slot-ordering family-blocking engine-only
johnml1135 Sep 6, 2026
85926f8
conformance: gate every fixture contributes a coverage.csv row
johnml1135 Sep 6, 2026
0f97024
docs: HCLoader.cs is 2,510 non-blank lines of 2,837 -- both figures w…
johnml1135 Sep 6, 2026
cf085f8
conformance: convert polysynthetic-stratal-derivation-chain to fieldw…
johnml1135 Sep 6, 2026
38a1649
conformance: convert compounding-breadth to fieldworks_producible
johnml1135 Sep 6, 2026
c107d55
conformance: convert mpr-gated-exception to fieldworks_producible
johnml1135 Sep 6, 2026
e27bca1
conformance: regenerate derived ledgers and update pinned counts
johnml1135 Sep 6, 2026
30b99dc
conformance: formalize KEEP-ENGINE-ONLY as a permanent, honest verdict
johnml1135 Sep 6, 2026
4b14a93
conformance: re-verify bistratal-overlapping-segment-representation a…
johnml1135 Sep 6, 2026
1e39927
docs: resolve mpr-group-overwrite-without-realizational's DELETE-pend…
johnml1135 Sep 6, 2026
7d02c93
conformance: the two-table premise is a loader gap, not a data-model …
johnml1135 Sep 8, 2026
95a8b69
conformance: correct two more verdicts that read exporter silence as …
johnml1135 Sep 8, 2026
bf16ff2
docs: the irregular-variant substitute over-generates, and the author…
johnml1135 Sep 8, 2026
d3b7643
conformance: classify the five unresolved producibility verdicts as l…
johnml1135 Sep 8, 2026
b15073b
Fix a dropped morpheme ID when a zero-width rule wraps one ending at …
johnml1135 Sep 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
18 changes: 18 additions & 0 deletions .claude/skills/author-coverage-cell/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
name: author-coverage-cell
description: >-
Use when asked to cover, satisfy, witness, or close an obligation cell in the HermitCrab
conformance suite — anything of the form "cover cell X", "satisfy this dataflow obligation",
"add words so this chain is witnessed", "close a coverage gap", or when handed a cell id from
conformance/dataflow-obligations.tsv. Authors the minimal-pair words in an existing language
grammar and the claimed_cells entry that records the evidence, then verifies by severance rather
than by inspection. Does NOT author new grammars, new fixtures, or grammar.xml content — if the
construct is absent from every language grammar, this skill reports that and stops.
---

The authoritative instructions for this skill live with the fixtures, because they ship to whoever
receives `conformance/`. Read and follow:

`conformance/skills/author-coverage-cell/SKILL.md`

Do not follow a summary of it from memory; open the file.
16 changes: 16 additions & 0 deletions .claude/skills/measure-authoring-quality/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: measure-authoring-quality
description: >-
Use when asked to measure, score, or benchmark the conformance authoring harness — "how good is
the author skill", "what's our first-pass yield", "run the harness eval", "measure before and
after a skill revision". Runs a batch of obligation cells through author-coverage-cell without
intervention, scores first-pass yield against the production gate, and attributes every failure to
a class. Produces a run record; changes no skill and fixes no fixture.
---

The authoritative instructions for this skill live with the fixtures, because they ship to whoever
receives `conformance/`. Read and follow:

`conformance/skills/measure-authoring-quality/SKILL.md`

Do not follow a summary of it from memory; open the file.
16 changes: 16 additions & 0 deletions .claude/skills/review-coverage-claim/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: review-coverage-claim
description: >-
Use when asked to review, verify, certify, or check a coverage claim in the HermitCrab conformance
suite — "review this claimed_cells entry", "does this word really demonstrate that cell", "check
the role attribution", or when handed an evidence card from conformance/evidence-cards/. Judges
ONLY role attribution: whether the word occupies the cell it claims. Never sets a status, never
edits a claim, never edits a grammar or an expectation. Declining is a valid outcome.
---

The authoritative instructions for this skill live with the fixtures, because they ship to whoever
receives `conformance/`. Read and follow:

`conformance/skills/review-coverage-claim/SKILL.md`

Do not follow a summary of it from memory; open the file.
16 changes: 16 additions & 0 deletions .claude/skills/revise-coverage-harness/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: revise-coverage-harness
description: >-
Use when asked to improve, revise, or fix the conformance authoring harness after a measurement —
"first-pass yield is low, fix the skill", "most failures are not-minimal-pair, revise the
instructions", "improve the harness based on the run record". Consumes a measurement run record,
diagnoses which instruction produced the dominant failure class, and revises exactly one thing.
Never revises without a measurement, and never touches the golden set.
---

The authoritative instructions for this skill live with the fixtures, because they ship to whoever
receives `conformance/`. Read and follow:

`conformance/skills/revise-coverage-harness/SKILL.md`

Do not follow a summary of it from memory; open the file.
10 changes: 10 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ jobs:
uses: actions/setup-dotnet@v5
with:
dotnet-version: |
10.0.303
10.0.x

- name: Restore dotnet tools
Expand Down Expand Up @@ -64,6 +65,14 @@ jobs:
run: dotnet build --no-restore -c Release
- name: Test
run: dotnet test --verbosity normal --collect:"Xplat Code Coverage"
# The semantic-coverage audit is gated by TheCheckedInCatalogMapsEveryRealSurfaceExactlyOnce
# under dotnet test, not here. That test can express what this CLI cannot: the catalog is
# deliberately an incomplete proposal backlog, so unclassified-mapping is the expected state
# rather than a regression, and the test pins it as the only diagnostic class allowed.
- name: Run conformance fixtures
run: dotnet run --no-build -c Release --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance
- name: Check coarse construct parity
run: python conformance/parity-check.py
- name: Upload coverage reports to Codecov
uses: codecov/codecov-action@v5
env:
Expand All @@ -81,6 +90,7 @@ jobs:
uses: actions/setup-dotnet@v5
with:
dotnet-version: |
10.0.303
10.0.x

- uses: actions/download-artifact@v7
Expand Down
71 changes: 71 additions & 0 deletions .github/workflows/conformance.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Conformance

on:
push:
branches:
- "**"
paths:
- "conformance/**"
- "src/SIL.Machine.Morphology.HermitCrab*/**"
pull_request:
paths:
- "conformance/**"
- "src/SIL.Machine.Morphology.HermitCrab*/**"

jobs:
conformance:
name: Conformance suite
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v6

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: |
10.0.303
10.0.x

- name: Restore dependencies
run: dotnet restore src/SIL.Machine.Morphology.HermitCrab.Conformance/SIL.Machine.Morphology.HermitCrab.Conformance.csproj

# Gate (a): every fixture's declared parses/rejections must match what the C# oracle
# actually produces (self-check mode runs the reference engine in-process against every
# words.yaml, including budget_ms pathological fixtures).
- name: Run conformance self-check
run: >-
dotnet run --no-restore --project src/SIL.Machine.Morphology.HermitCrab.Conformance --
--fixtures conformance --include-pathological

# Gate (b): --coverage-report (re)writes coverage.csv/rules.csv, which the next step checks
# for staleness, and prints any dead rules (rules exercised by zero words) to stdout. NOTE:
# on this branch, RunCoverageReport's caller in Program.cs unconditionally returns 0 --
# it does not fail the build when dead rules are found, unlike the upstream branch this
# workflow was ported from. Making this step's own exit code the dead-rule gate would
# require changing Program.cs's dispatch, which is out of scope for this port; until that
# lands, dead rules are visible in this step's log but do not fail CI. `if: always()` so this
# still runs (and reports its own true status) even if the self-check step above failed.
- name: Run coverage report
if: always()
run: >-
dotnet run --no-restore --project src/SIL.Machine.Morphology.HermitCrab.Conformance --
--fixtures conformance --coverage-report

# Gate (c): the checked-in coverage.csv/rules.csv must already reflect the fixtures as they
# stand -- generated so nobody needs to remember to refresh them, but that only holds if
# forgetting to refresh them is a CI failure. Runs after the coverage-report step (which just
# regenerated both) and `if: always()` for the same reason as above: a dead rule and a stale
# index are two different failures and must not collapse into one.
#
# conformance/fixtures.csv is intentionally NOT checked here: on this branch nothing
# regenerates it (--coverage-report only writes coverage.csv/rules.csv), so a freshness check
# against it would never catch drift -- it is a one-time snapshot from when it was ported, not
# a generated artifact.
- name: Verify generated coverage files are fresh
if: always()
run: |
if ! git diff --exit-code -- conformance/coverage.csv conformance/rules.csv; then
echo "::error::conformance/coverage.csv or rules.csv is stale relative to the fixtures (diff above). Run 'dotnet run --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --fixtures conformance --coverage-report' locally and commit the result."
exit 1
fi
36 changes: 36 additions & 0 deletions .github/workflows/counterfactual-coverage.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# The counterfactual sweep re-parses every conformance fixture once per grammar-observable surface with
# that surface neutralized, so it costs minutes rather than seconds. It runs on demand and weekly rather
# than per push; the cheap coverage ledger and fixture gates stay in the main CI job.
name: Counterfactual coverage

on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1"

jobs:
counterfactual:
name: Recompute counterfactual evidence
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v6

- name: Setup .NET
uses: actions/setup-dotnet@v5
with:
dotnet-version: |
10.0.303
10.0.x

- name: Restore dependencies
run: dotnet restore

- name: Build
run: dotnet build --no-restore -c Release

- name: Recompute and compare the evidence ledger
run: dotnet run --no-build -c Release --project src/SIL.Machine.Morphology.HermitCrab.Conformance -- --counterfactual --repository-root .

- name: Run the explicit counterfactual tests
run: dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests/SIL.Machine.Morphology.HermitCrab.Tests.csproj -c Release --filter "TestCategory=Counterfactual"
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,12 @@ tests/SIL.Machine.Tests/Corpora/TestData/usfm/target/*
tests/SIL.Machine.Tests/Corpora/TestData/project/*
tests/SIL.Machine.Tests/Corpora/TestData/pretranslations.json
.idea

# Claude Code conversation exports
20??-??-??-*.txt

# Agent scratch: test result dumps, captured logs, throwaway SDK/CLI state
.tmp/

# Local pre-PR review artifacts, never committed
.review/
7 changes: 7 additions & 0 deletions Machine.sln
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SIL.Machine.Tokenization.Se
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SIL.Machine.Tokenization.SentencePiece.Tests", "tests\SIL.Machine.Tokenization.SentencePiece.Tests\SIL.Machine.Tokenization.SentencePiece.Tests.csproj", "{AB5F75C1-64B7-4E0F-A4B5-B14EB16E6DDC}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "SIL.Machine.Morphology.HermitCrab.Conformance", "src\SIL.Machine.Morphology.HermitCrab.Conformance\SIL.Machine.Morphology.HermitCrab.Conformance.csproj", "{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -99,6 +101,10 @@ Global
{AB5F75C1-64B7-4E0F-A4B5-B14EB16E6DDC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{AB5F75C1-64B7-4E0F-A4B5-B14EB16E6DDC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{AB5F75C1-64B7-4E0F-A4B5-B14EB16E6DDC}.Release|Any CPU.Build.0 = Release|Any CPU
{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3}.Debug|Any CPU.Build.0 = Debug|Any CPU
{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3}.Release|Any CPU.ActiveCfg = Release|Any CPU
{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand All @@ -116,6 +122,7 @@ Global
{0BBEED18-3CC1-4BA6-8E11-603A5D2DEDC9} = {31B4F5EB-E148-4A51-84B5-B90BB8F38AD3}
{09135223-6285-4268-BFA3-147BDB5B663B} = {31B4F5EB-E148-4A51-84B5-B90BB8F38AD3}
{AB5F75C1-64B7-4E0F-A4B5-B14EB16E6DDC} = {66246A1C-8D45-40FB-A660-C58577122CA7}
{8E40C2C5-F4C5-4A41-B78A-10A43FCF0CE3} = {31B4F5EB-E148-4A51-84B5-B90BB8F38AD3}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {9F18C25E-E140-43C3-B177-D562E1628370}
Expand Down
Loading
Loading