New Methods Combine Distillation and Reinforcement Learning to Improve Small Language Model Reasoning
Researchers have introduced multiple approaches to enhance reasoning capabilities in smaller language models by combining on-policy distillation (OPD) with reinforcement learning (RL), according to papers published on arXiv.
According to arxiv.org, OPDSearch+ represents “the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning.” The method uses a frozen off-the-shelf model as a teacher to reshape the student model’s policy distribution, enabling subsequent RL to “converge to a superior solution that RL alone cannot reach.” In the first stage, the student interacts with a live search engine and learns via a per-position forward KL objective. Testing on seven QA benchmarks, a 3B model achieved gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA compared to prior 3B RL baselines.
A related approach called OPDVR (On-policy Distillation with Verifiable Reward) addresses limitations in existing methods. According to arxiv.org, while RLVR “suffers from sparse task-level feedback,” OPD “provides dense token-level guidance but ignores trajectory correctness.” OPDVR applies “a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards,” eliminating additional hyperparameters. Experiments on six reasoning benchmarks showed OPDVR “consistently outperforms standard OPD.”
Additionally, arxiv.org describes Contrastive Branch Policy Optimization (CBPO), which uses generation entropy to screen branch positions and introduces Contrastive Branch Value for outcome-based credit assignment. Testing on ten benchmarks showed CBPO “consistently outperforms state-of-the-art policy-optimization and branch-based methods.”