Skip to content

feat: add configurable regex_match evaluator - #9

Open
nawaaaaaAaar wants to merge 1 commit into
github-community-gitam:mainfrom
nawaaaaaAaar:feat/regex-match-evaluator
Open

nawaaaaaAaar wants to merge 1 commit into
github-community-gitam:mainfrom
nawaaaaaAaar:feat/regex-match-evaluator

Conversation

@nawaaaaaAaar

Copy link
Copy Markdown

Summary

Closes #1. Adds the requested regex_match evaluator with configurable search/full-match behavior and case sensitivity.

  • Exported and registered through the existing evaluator plugin system.
  • Returns failed evaluation results for invalid patterns and agent errors rather than raising.
  • Includes pattern, flags and matched text in structured details.
  • Does not change core models, runner orchestration or existing evaluators.

Verification

  • Before implementation, the new tests failed to import the missing evaluator.
  • make check: Ruff, strict mypy and the complete suite passed; 49 tests passed, with 90% reported overall coverage.
  • Ten new evaluator cases cover matches, misses, case behavior, full matching, empty matching, invalid syntax, agent-error precedence and registry instantiation.
  • Focused branch coverage: 100% for the new evaluator.
  • uv run harness list-plugins evaluators lists regex_match.
  • git diff --check: passed.

Python 3.13; no model APIs or live agents were called. Like standard-library re, this evaluator does not impose a match timeout; patterns should be trusted benchmark configuration rather than hostile input.

Prepared with Perplexity Computer assistance; no personal account-owner review is claimed.

Copilot AI balanced review requested due to automatic review settings October 5, 2026 05:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(evaluator): add RegexMatchEvaluator

2 participants