Created GRPOTrainerWithEval subclass for different evaluation reward functions

[jamesbraza]

This PR creates a GRPOTrainer subclass GRPOTrainerWithEval that adds support for optional eval_reward_processing_classes.

It should be backwards compatible with GRPOTrainer.

The only caveat here is I didn't comprehensively think about args.reward_weights.

[Copilot]

PR Overview

This PR introduces a new subclass, GRPOTrainerWithEval, which extends the GRPOTrainer functionality to support evaluation reward functions while maintaining backward compatibility.

• New GRPOTrainerWithEval subclass accepts separate evaluation reward functions and processing classes.
• Configuration handling is unified through the use of an instance attribute (_model_init_kwargs) and a dedicated helper method (_make_reward_processing_classes).
• The diff adds strict checking in zip calls to enforce matching lengths of reward functions and processing classes.

Reviewed Changes

File	Description
trl/trainer/grpo_trainer.py	Introduces GRPOTrainerWithEval and refactors reward processing and model init kwargs

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[shirinyamani]

Have you tested this against the new multi-task reward_func setup?
Cuz the restrict=True will not work with the current setup which allows reward_funcs to return None

[jamesbraza]

Have you tested this against the new multi-task reward_func setup? Cuz the restrict=True will not work with the current setup which allows reward_funcs to return None

Hi @shirinyamani thanks for the comment, no we stopped rebasing atop trl's main because we were getting broken too often.

Perhaps if we rebased for newer features in trl, splitting out the eval reward function (what this PR does) would have to change due to this restrict=True thing.