Following this discussion, I think agent critic during execution should be a standalone issue.
The previous two issues on agent critic after generation (#512) and trajectory persistence (#554) work together, as the critic agent can read the persisted trajectory for potential LLM-as-judge.
Critic during generation is a bit different, as it touches aspects such as the frequency of critic and how the feedback can be given to the main agent. Any ideas on this would be great.
I think C for the new event makes sense. Another thing I'm thinking is that rather than critic the entire trajectory, an additional useful thing is for the agent to get feedback on the completion of the task so far to better select subsequent actions, so maybe there are different events for either critic during the execution or after an entire execution is completed.
Whatever the direction to go, I want the implementation to be agnostic to the underlying agent to better support #515. There should be a mechanism for the agent (not only the Sherpa agent, but also Claude, Codex, etc) to access the trajectory data and decide to critique. Maybe it should be exposed as some kind of function
Originally posted by @20001LastOrder in #512
Following this discussion, I think agent critic during execution should be a standalone issue.
The previous two issues on agent critic after generation (#512) and trajectory persistence (#554) work together, as the critic agent can read the persisted trajectory for potential LLM-as-judge.
Critic during generation is a bit different, as it touches aspects such as the frequency of critic and how the feedback can be given to the main agent. Any ideas on this would be great.
Originally posted by @20001LastOrder in #512