请教一下Qwen2.5-3B-instruct用mix_chord训练，reward很低的原因

[caiyutong-eng]

作者您好，非常感谢您的项目，我尝试了用mix_chord训练Qwen2.5-3B-Instruct，但是reward很低，想请教一下原因，是不是超参的问题，以下是我的配置文件，是在8张40G的A800跑的。

project: "mix_chord"
name: "test_mix_2.5_3B"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  optimizer:
    lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
  kl_loss_fn_args:
    kl_coef: 0.0
  entropy_loss_fn: mix
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 200 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.9 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.05 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: false # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 80   # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 4
    ngpus_trainer: 4
    train_batch_size_expert: 16   # sft
    train_batch_size_usual: 64    # batchsize*repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH, /home/dataset-assist-0/wl/cyt/models/Qwen2.5-3B-Instruct}
  max_response_tokens: 10240    # 控制 模型输出 的最大长度
  max_model_len: 11264          # 控制 模型输入 + 输出的总长度上限
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 2
  batch_size: 8
  train_batch_size: 80    # train_batch_size_usual + train_batch_size_expert
  explorer_input:
    taskset:
      name: openr1_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /home/dataset-assist-0/wl/cyt/Trinity-RFT/examples/mix_chord/openr1_rl_dataset}
      format:
        prompt_key: 'problem'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: true
        #use_base: true   # added
    eval_tasksets:
      - name: AIME2024
        storage_type: file
        path: ${oc.env:TRINITY_TASKSET_PATH, /home/dataset-assist-0/wl/cyt/datasets/aime24}  # e.g. path to AIME2024
        split: 'train'
        repeat_times: 4
        format:
            prompt_key: 'problem'
            response_key: 'solution'
        rollout_args:
            temperature: 1.0
            top_p: 0.7
        default_reward_fn_type: 'math_boxed_reward'
        reward_fn_args: {}
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///test_mix_2.5_3B.db'            #change this
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 25
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH, /home/dataset-assist-0/wl/cyt/datasets/open-r1/mixture-of-thoughts/math}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 20
  runner_per_model: 8
  rollout_model:
    engine_num: 4
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
    gpu_memory_utilization: 0.9
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  grad_clip: 1.0
  use_dynamic_bsz: true
  #ppo_max_token_len_per_gpu: 25600
  ulysses_sequence_parallel_size: 2
monitor:
  monitor_type: wandb
  monitor_args:
    api_key: 14d0d70e75f8d966867f6e0fc77783384993335e

下面是截图

非常感谢

[pan-x-c]

提供的配置文件样例与论文中使用的配置差距较大，包含 batch_size, algorithm 等多项变动，导致难以定位问题，推荐先控制变量，不要一次性修改太多参数。

[caiyutong-eng]

非常感谢您的回复，我把batch_size调小了是因为显存限制，其他应该只改了policy_loss_fn_args的超参，是按照yaml文件注释里的chord-mu改的，想问下yaml注释里的是论文中的配置还是yaml里现在的值就是？

[garyzhang99]

非常感谢您的回复，我把batch_size调小了是因为显存限制，其他应该只改了policy_loss_fn_args的超参，是按照yaml文件注释里的chord-mu改的，想问下yaml注释里的是论文中的配置还是yaml里现在的值就是？

其实batchsize不会影响到对应的GPU memory情况，因为最后算loss时走的minibatch和microbatch，对应的显存峰值取决于设置合适的对应ppo_max_token_len_per_gpu: 12288。调高对应的Batchsize不会过多影响显存峰值占比。

从Performance的角度出发非常建议你把对应的Batchsize调大。因为OpenR1数据集对应其实还比较需要难度过滤，OpenR1数据集不做难度过滤的话训练一段时间后对应会有30%多的样本是8次rollout全对，20%左右样本是8次rollout全错，产生无效梯度导致对应实际生效的Batchsize比你设置的Batchsize还小50%。
那RL一次只能实际学到4个样本的Group确实相对太小，如果你把Batchsize尽量设大一些就会稳定多了（3B的训练可能还更不稳定，可能需要更大的batchsize）。

参数配置上，$Chord-mu$ 当时在Qwen2.5-7B-Instruct上的配置是lr5e-6 + 50 step warmup的cosine decay。当然这也不一定是最优的，不过对于Qwen系列使用SFT时一般需要比纯RL更大的lr，既然 $Chord-mu$ 是先SFT再RL就建议也设置一个warmup。lr warmup的设置方法可以参考docs/sphinx_doc/source_zh/tutorial/trinity_configs.md。

[caiyutong-eng]

感谢感谢，我再试一下

[caiyutong-eng]

想再问您一下,我也跑了Qwen2.5-7B-instruct在mix_chord的实验，用论文里fix mu=0.1的chord phi，reward也不太对劲，看了下rollout_examples发现有一些乱码，配置文件如下

project: "mix_7B_yby"
name: "mix_chord_7B_yby"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./ckpt}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 8 # or 16 for better performance in math related tasks
  optimizer:
    lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
  kl_loss_fn_args:
    kl_coef: 0.0
  entropy_loss_fn: mix
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.1 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.1 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 320 # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 4
    ngpus_trainer: 4
    train_batch_size_expert: 64
    train_batch_size_usual: 256 # 32 batchsize * 8 repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH,/data/cyt/Qwen2.5-7B-Ins}
  max_response_tokens: 10240
  max_model_len: 11264
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 4
  total_steps: 1000
  batch_size: 32
  train_batch_size: 320
  explorer_input:
    taskset:
      name: openr1_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH,/data/cyt/Trinity-RFT/examples/mix_chord/openr1_rl_dataset}
      format:
        prompt_key: 'problem'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
    eval_tasksets:
      - name: MATH-500
        storage_type: file
        path: ${oc.env:TRINITY_TASKSET_PATH,/data/cyt/math-500}
        split: 'test'
        repeat_times: 4
        format:
          prompt_key: 'problem'
          response_key: 'answer'
        rollout_args:
          temperature: 1.0
          top_p: 0.7
        default_reward_fn_type: 'math_boxed_reward'
        reward_fn_args: {}
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///test_7B_yby.db'
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 25
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH,/data/cyt/openr1-mixture/math}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 20
  runner_per_model: 8
  rollout_model:
    engine_num: 4
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  grad_clip: 1.0
  use_dynamic_bsz: true
  max_token_len_per_gpu: 16384
  ulysses_sequence_parallel_size: 1
monitor:
  monitor_type: wandb
  monitor_args:
    api_key: 14d0d70e75f8d966867f6e0fc77783384993335e

在80步的时候报错，但是继续训练好像也没有训起来？

Rollout examples:

80步和100步报错截图：

[garyzhang99]

@caiyutong-eng

这里报错应该就是因为RL训练训崩了，导致LLM都没法正常生成Experience了，所以断点续训练也没用了，不过导致这个问题的对应具体原因不好说。可能和数据没做难度过滤（简单题目比例太多会让有效的batchsize太小）和content过滤（有些数据比较脏也会有影响）有关，也有可能是其他原因。我们这边也会试着对应排查一下。

以及我看你之前的dataset写的是open-r1/mixture-of-thoughts/math这个数据集，如果你用的是 https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts 这个数据集的话，对应SFT的system prompt部分应该是对不上的。

你可以试一下不加上fix mu=0.1，把mu设置为0只训练RL不加SFT会不会训崩（看是不是SFT部分有问题还是啥别的原因）。另一方面如果只是想提升稳定性的话最简单的就是再调更大的batchsize和repeat_nums。

[caiyutong-eng]

试了下纯GRPO还是崩了，batch_size应该也算调的比较大了，是openr1-math-220k这个数据集有问题吗？想问下您有训练效果比较好的配置文件或者数据集推荐吗？或者您有其他debug的建议也非常感谢

project: "3B_mu=0"
name: "test_3B_mu"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./ckpt}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 16 # or 16 for better performance in math related tasks
  optimizer:
    lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
  kl_loss_fn_args:
    kl_coef: 0.0
  entropy_loss_fn: mix
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 0  # 0 for chord-mu and chord-phi
    mu_decay_steps: 0 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 2560   # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 64
    ngpus_trainer: 4
    train_batch_size_expert: 512   # sft
    train_batch_size_usual: 2048    # batchsize*repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH, /home/dataset-assist-0/wl/cyt/models/Qwen2.5-3B-Instruct}
  max_response_tokens: 10240    # original: 10240 控制 模型输出 的最大长度
  max_model_len: 11264          # 控制 模型输入 + 输出的总长度上限
  # max_prompt_tokens: 8192
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 2
  batch_size: 128
  train_batch_size: 2560    # train_batch_size_usual + train_batch_size_expert
  explorer_input:
    taskset:
      name: openr1_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /home/dataset-assist-0/wl/cyt/Trinity-RFT/examples/mix_chord/openr1_rl_dataset}
      format:
        prompt_key: 'problem'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: false
        #use_base: true   # added
    eval_tasksets:
      - name: MATH-500
        storage_type: file
        path: ${oc.env:TRINITY_TASKSET_PATH,/home/dataset-assist-0/wl/cyt/datasets/math-500}
        split: 'test'
        repeat_times: 4
        format:
          prompt_key: 'problem'
          response_key: 'answer'
        rollout_args:
          temperature: 1.0
          top_p: 0.7
        default_reward_fn_type: 'math_boxed_reward'
        reward_fn_args: {}
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///test_3B_mu=0.db'            #change this
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 25
        name: SFT_data
        storage_type: file
        schema_type: sft
        # path: ${oc.env:TRINITY_SFT_DATASET_PATH, /home/dataset-assist-0/wl/cyt/datasets/open-r1}
        path: json
        data_files: "/home/dataset-assist-0/wl/cyt/datasets/open-r1/mixture_filtered.json"
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
          system_prompt: 'Please reason step by step, and put your final answer within \boxed{}.'
explorer:
  eval_interval: 20
  runner_per_model: 8
  rollout_model:
    engine_num: 4
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
    gpu_memory_utilization: 0.9
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  grad_clip: 1.0
  use_dynamic_bsz: true
  #max_token_len_per_gpu: 12288
  ulysses_sequence_parallel_size: 2
monitor:
  monitor_type: wandb
  monitor_args:
    api_key: 14d0d70e75f8d966867f6e0fc77783384993335e

[garyzhang99]

@caiyutong-eng 我之前确实跑过使用GRPO或者其他各种不同的算法用qwen2.5-3b-instruct在openr1数据集的训练，训练的也都还很稳定。所以你这个配置会训崩其实很奇怪。

我当时跑的reward曲线为下图：

对应的batchsize和repeat_nums比你这个还小（甚至都没调更大32 batchsize * 8 repeat nums）。你的配置和我图中grpo_online的配置区别只有：1.我的with_think设置的是true而不是false; 2.model_len我设置的稍微更大一些到了16K。

一方面建议你可以试着改一下不同的这两个参数，同时你可以提供一下除了reward对应的其他的一些曲线和更多的信息吗？比如entropy、response_length、grad_norm的情况。

[garyzhang99]

要换更干净数据集的话，可以试一试DAPO数据集。

  explorer_input:
    taskset:
      name: dapo-math
      storage_type: file
      path: open-r1/DAPO-Math-17k-Processed
      subset_name: all
      format:
        prompt_key: 'prompt'
        response_key: 'solution'

[caiyutong-eng]

好的好的，太感谢了，您这个grpo_online是mix_chord设置mu=0的吗，还是直接algorithm type=grpo的？我感觉可能不是with_think的问题，因为之前with_think也设置的true，训崩了才改的。
我这个配置一直训崩会不会是有些库版本有问题：
PyTorch version: 2.6.0+cu124
Python version: 3.11.11
verl version: 0.5.0
vllm version: 0.8.5

[caiyutong-eng]

但是我在同样的环境还训过Qwen3-4B，刚开始试的时候batch size很小，没有训崩（因为测了分没涨才换成2.5-3B-instruct了

project: "mix_chord"
name: "test_mix_4B"
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
algorithm:
  algorithm_type: mix_chord
  repeat_times: 4 # or 16 for better performance in math related tasks
  optimizer:
    lr: 1e-6 # or 5e-6, larger lr with warm up can result in better performance for SFT training.
  kl_loss_fn_args:
    kl_coef: 0.0
  entropy_loss_fn: mix
  sample_strategy_args:
    expert_data_ratio: 0.20
  policy_loss_fn_args: # feel free to change, we encourage you to try out different hyperparameters
    mu_warmup_steps: 200  # 0 for chord-mu and chord-phi
    mu_decay_steps: 400 # 200 for chord-mu and 0 for chord-phi
    mu_peak: 0.5 # 0.9 for chord-mu and 0.1 for chord-phi
    mu_valley: 0.02 # 0.05 for chord-mu and 0.1 for chord-phi
    enable_phi_function: true # false for chord-mu and true for chord-phi
    clip_range: 0.2
    sft_loss_agg_mode: "token-mean"
    use_dynamic_bsz: true
    ppo_mini_batch_size: 32   # 320 = 256 + 64; if you set repeat times = 16, then it shoudle be 32 * 16 + 64
    ppo_micro_batch_size_per_gpu: 4
    ngpus_trainer: 4
    train_batch_size_expert: 16   # sft
    train_batch_size_usual: 16    # batchsize*repeat times
model:
  model_path: ${oc.env:TRINITY_MODEL_PATH, /home/dataset-assist-0/wl/cyt/models/Qwen3-4B}
  max_response_tokens: 10240    # 控制 模型输出 的最大长度
  max_model_len: 11264          # 控制 模型输入 + 输出的总长度上限
cluster:
  node_num: 1
  gpu_per_node: 8
buffer:
  total_epochs: 2
  batch_size: 4
  train_batch_size: 32    # train_batch_size_usual + train_batch_size_expert
  explorer_input:
    taskset:
      name: openr1_data_filtered_int
      storage_type: file
      path: ${oc.env:TRINITY_TASKSET_PATH, /home/dataset-assist-0/wl/cyt/Trinity-RFT/examples/mix_chord/openr1_rl_dataset}
      format:
        prompt_key: 'problem'
        response_key: 'answer'
      rollout_args:
        temperature: 1.0
        logprobs: 0
      workflow_args:
        with_think: true
        #use_base: true   # added
    eval_tasksets: [] # you can add your own eval tasksets here
    default_workflow_type: 'math_boxed_workflow'
  trainer_input:
    experience_buffer:
      name: math_buffer
      storage_type: queue
      path: 'sqlite:///test_mix_4B.db'         
    auxiliary_buffers:
      sft_dataset:
        total_epochs: 25
        name: SFT_data
        storage_type: file
        schema_type: sft
        path: ${oc.env:TRINITY_SFT_DATASET_PATH, /home/dataset-assist-0/wl/cyt/datasets/open-r1/mixture-of-thoughts/math}
        split: 'train'
        format:
          prompt_type: messages
          messages_key: 'messages'
explorer:
  eval_interval: 10
  runner_per_model: 8
  rollout_model:
    engine_num: 4
    tensor_parallel_size: 1
    enable_prefix_caching: false
    enforce_eager: true
    dtype: bfloat16
    seed: 42
    gpu_memory_utilization: 0.9
synchronizer:
  sync_method: 'nccl'
  sync_interval: 1
  sync_timeout: 1200
trainer:
  save_interval: 50
  grad_clip: 1.0
  use_dynamic_bsz: true
  #ppo_max_token_len_per_gpu: 25600
  ulysses_sequence_parallel_size: 2
monitor:
  monitor_type: wandb
  monitor_args:
    api_key: 14d0d70e75f8d966867f6e0fc77783384993335e

[songyang86]

@caiyutong-eng 请问你换了DAPO数据集后训练效果怎么样

[saisharan0103]

the repro details point to the training config rather than a generic mix_chord failure: max_response_tokens: 10240 with max_model_len: 11264 leaves little room for long prompts, and kl_coef: 0.0 also removes one stabilizing term.

model.max_model_len / response length handling is the part I would inspect first against examples/mix_chord/mix_chord.yaml and the config docs. 你现在使用的 reward 配置和数据集是哪一套？