Researchers have introduced OPDSearch+, a new approach for improving search-augmented reasoning in small language models without requiring teacher model fine-tuning, according to a paper published on arXiv.org.
According to the paper, OPDSearch+ addresses key limitations in on-policy distillation (OPD) by using a frozen off-the-shelf instruction model as the teacher. The method operates in two stages: first, the student model interacts with a live search engine and learns through a per-position forward KL objective, transferring “reasoning decomposition and evidence integration skills without any task-specific teacher training.” Second, reinforcement learning refines the distilled student from what the researchers describe as “a richer behavioral foundation.”
The researchers report that OPDSearch+ with a 3B parameter model “consistently outperforms all prior 3B RL baselines,” achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA across seven question-answering benchmarks, according to arXiv.org.
In related work published the same day, researchers proposed On-policy Distillation with Verifiable Reward (OPDVR), which combines OPD with reinforcement learning using verifiable rewards. According to arXiv.org, OPDVR applies “a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards.” The method showed consistent improvements over standard OPD across six reasoning benchmarks.