arxiv:2512.02580

From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks

Published on Dec 2

· Submitted by

Jinyang Wu on Dec 8

#3 Paper of the day

Upvote

Authors:

Abstract

CAPO, a curriculum advantage policy optimization, enhances reinforcement learning for large language models by strategically introducing positive and negative advantage signals, improving reasoning capabilities and generalization.

AI-generated summary

Reinforcement learning has emerged as a paradigm for post-training large language models, boosting their reasoning capabilities. Such approaches compute an advantage value for each sample, reflecting better or worse performance than expected, thereby yielding both positive and negative signals for training. However, the indiscriminate mixing of the two signals in existing methods, especially from the early stages, may lead to ambiguous guidance and limited gains. To address this issue, we propose **CAPO** (**C**urriculum **A**dvantage **P**olicy **O**ptimization), an adaptive curriculum mechanism based on advantage signals. The proposed mechanism bootstraps imitation learning with positive-only advantage samples to establish robust foundations, and subsequently introduces negative signals to cultivate discriminative capabilities, thereby improving generalization across complex scenarios. Compatible with diverse optimization methods including GRPO, PPO, RLOO, and Reinforce++, our method consistently achieves stable and significant improvements in mathematical reasoning tasks, and further generalizes effectively to multimodal Graphical User Interface (GUI) reasoning scenarios, establishing itself as a versatile and robust optimization framework.

View arXiv page View PDF Add to collection

Community

Jinyang23

Paper submitter 1 day ago

🚀 [New Paper] CAPO: From Imitation to Discrimination – Rethinking Advantage in RL

Early RL training often suffers from instability due to "mixed signals" (simultaneous positive & negative feedback). Inspired by child cognitive development, we propose CAPO (Curriculum Advantage Policy Optimization).

✨ The Core Intuition:
Instead of a static curriculum, we leverage Advantage values to create a dynamic, two-phase process:
1️⃣ Imitation Phase: Train on Positive Advantage only. This reduces variance and establishes a stable behavioral foundation (Imitate to learn).
2️⃣ Discrimination Phase: Introduce Negative Signals later. [cite_start]This restores unbiased estimation and refines decision boundaries (Discriminate to generalize).

📈 Highlights:

Plug-and-Play: Delivers consistent gains when compatible with GRPO, PPO, RLOO, & Reinforce++.
Cross-Domain: Not just for Math! CAPO demonstrates impressive generalization on Multimodal GUI Agent tasks.