Idea: Adding a CI/CD gate to block PRs based on these evaluations?

[mukundzha]

Hey @OpenAgentHQ, love this! A local-first, framework-agnostic evaluation framework is exactly what the AI community needs. Calling it "The pytest of AI evaluation" is perfect.

I was wondering if you've considered how users can use these evaluations to actually block bad code from deploying? Running evals is great, but ensuring they automatically fail a GitHub Action when a metric drops is where the real CI/CD value is.

I actually built an open-source Python CLI called Crilio that acts as the CI/CD gate for this. It uses an LLM-as-a-Judge to grade outputs against YAML rules and blocks GitHub PRs if the model hallucinates or breaks formatting.

Repo: https://github.com/mukundzha/crilio

You could use openagent-eval for the deep metrics, and Crilio as the semantic gate in GitHub Actions. Would love to hear your thoughts on this integration!

[himanshu231204]

Hey @mukundzha — great question, and this is actually something we're already working toward!

We're building RepoHeart — an autonomous, event-driven multi-agent system that runs entirely in GitHub Actions. It already handles PR reviews, issue triage, CI repair, and repo health — no server or database needed.

The natural next step is exactly what you're describing: wiring openagent-eval metrics as a quality gate inside the RepoHeart workflow. Instead of a separate tool, the eval results would feed directly into the same Actions pipeline, and the Safety Gate can block a merge if a metric regresses (hallucination rate, format adherence, latency, etc.).

Crilio looks like a clean approach too — the LLM-as-a-Judge + YAML rules idea is solid. A potential integration could be: openagent-eval runs the deep metrics → results are passed to the gate layer (whether that's Crilio, a custom step, or a RepoHeart agent) → PR is blocked or annotated accordingly.

Would love to explore how these could work together. Feel free to open an issue or discussion in the RepoHeart repo if you want to dig into the integration!