Feature/basic score

[laurejt]

Associated Issue(s): resolves #10, #11, #12, #13

Changes in this PR

Include all key changes in this pull request

• Moves span methods to spans/match to span_utils
• Moves custom span-level function types to span_utils
• Removes ignore_label functionality from span utility methods
• Adds a custom composite match score (uses both label and boundary scoring)
• Add methods for computing evaluation metrics for individual SpanAlignments (eval)
• Add a meta-function construct_aligner for building a span alignment method
• Update DocSpans so they are hashable (by converting internal _spans field to a tuple)
• Two test JSONL files ref.jsonl and sys.jsonl to make local testing easier (tests/test_data)
• Add compute_metrics script for computing document- and entity-level evaluation metrics (precision, recall, F1) for two span annotations JSONL files.

Notes

• Apologies in advance for the very large PR
• I've used meta-functions to help simplify computing aggregated evaluation metrics. Both of these are used to set the optional parameters that should be fixed during evaluation.
• With this PR, does spanerr look ready for an alpha release?

Notes for Testing

Case 1. Entity-level metrics with "select_first" alignment & "overlap_factor" scoring

Command:

uv run python src/spanerr/compute_metrics.py tests/test_data/ref.jsonl tests/test_data/sys.jsonl entity select_first overlap_factor

Final Output

Macro entity-level metrics for 9 documents:
- Precision = 0.348
- Recall = 0.3797
- F1 = 0.3631

Case 2. Document-level metrics with "corppa" alignment & "jaccard" scoring

Command:

uv run python src/spanerr/compute_metrics.py tests/test_data/ref.jsonl tests/test_data/sys.jsonl document corppa jaccard

Final Output

Macro document-level metrics for 9 documents:
- Precision = 0.4276
- Recall = 0.4649
- F1 = 0.43

Reviewer Checklist

• Can successfully run compute_metrics script locally with expected output (see Notes for Testing)
• Output to command line looks reasonable
• Changelog is legible and sufficient

[codecov]

Codecov Report

❌ Patch coverage is 98.78788% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.30%. Comparing base (d005215) to head (0039d7e).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop      #14      +/-   ##
===========================================
- Coverage   100.00%   99.30%   -0.70%     
===========================================
  Files            7       11       +4     
  Lines          545     1159     +614     
  Branches        36       77      +41     
===========================================
+ Hits           545     1151     +606     
- Misses           0        4       +4     
- Partials         0        4       +4     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rlskoeser]

I've reviewed and made a few comments and small suggestions, but this looks solid to me. I was also able to run the scripts with the parameters you provided and get the expected output.

does spanerr look ready for an alpha release?

It looks great - the only major missing item I see is that you need a readme to provide some context and instructions for installation basic usage.

[rlskoeser]

Somewhere (probably not here) it would be helpful to have a note explaining why the lambda approach - once I finished reading through the function I saw how many variations there are, which I presume is the motivation. It would be helpful to have that explained at a higher level first - maybe a file- level docstring? (Or maybe a design doc if you add some version of that to the code)

[laurejt]

Good idea. I'll open an issue for tracking this.

[rlskoeser]

this is pretty slick

[laurejt]

I've reviewed and made a few comments and small suggestions, but this looks solid to me. I was also able to run the scripts with the parameters you provided and get the expected output.

does spanerr look ready for an alpha release?

It looks great - the only major missing item I see is that you need a readme to provide some context and instructions for installation basic usage.

Thanks for the reminder about the README!