WIP: Add eval command - #98
Draft
osma wants to merge 4 commits into
Draft
Conversation
…lementation) Introduces a new `eval` command to the CLI, allowing users to evaluate metadata extraction results against provided ground truth JSONL files. The actual implementation of evaluation is still a non-functional stub.
Introduces fuzzy string comparison using Levenshtein distance and a configurable threshold (0.95) to improve robustness in evaluation. Also refactors `bibra/cli.py` to use `aggregate_results` instead of raw JSON dumping, providing a cleaner output format for evaluation results.
Implements calculation and inclusion of an overall mean score in the TSV output of `aggregate_results`. This change adds a summary row to the aggregated results, providing a global mean score across all fields and languages. Corresponding tests in `test_eval.py` have been updated to validate the new summary row and edge cases like empty results.
Adds a timeout mechanism to the extraction process to prevent indefinite blocking during evaluation. Also refactors `run_evaluation` to wrap the extraction call in a try/except block, ensuring that evaluation continues for other fields even if a single URL extraction fails.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #98 +/- ##
==========================================
- Coverage 98.68% 96.72% -1.97%
==========================================
Files 13 15 +2
Lines 533 733 +200
==========================================
+ Hits 526 709 +183
- Misses 7 24 +17
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reasons for creating this PR
See #46. WIP.
Link to relevant issue(s), if any
Description of the changes in this PR
Instructions how to test this PR
Known problems or uncertainties in this PR
Checklist
Disclosure of AI Tool Usage
Please indicate AI use by choosing the most suitable TLP:AI category below and removing the irrelevant categories from the list. AI:ORANGE is the minimum level for merging.
Describe the AI tool(s) you used: