Agentic-DPO: New Method Improves AI Agent Training Without Online Environment Interaction
Researchers have introduced Agentic-DPO, a new approach to training large language model (LLM) agents that improves upon traditional supervised fine-tuning methods, according to a paper published on arxiv.org.
According to the research, LLM agents are commonly trained from expert trajectories using supervised fine-tuning (SFT), which “treats multi-turn agent behavior as ordinary text imitation.” However, this approach “only learns to imitate the sequence of expert actions, rather than training the agent to choose the right action against plausible mistakes at each state.”
Agentic-DPO addresses this limitation by turning expert trajectories into state-conditioned preference supervision. At each expert action state, the method “samples a one-step action from the current state, treats plausible wrong actions as negatives, and contrasts them with the expert action using a DPO-style preference objective,” according to arxiv.org.
The researchers report that “Agentic-DPO requires no online environment rollout, reward model, or full-trajectory student exploration.” In experiments across StableToolBench, tau-bench retail, and Mind2Web, the method “consistently improves agents at different model scales beyond imitation.”
According to the paper, Agentic-DPO raised tau-bench accuracy from 21.7% with SFT to 41.4% for a 9B model, “matching online GRPO under the same backbone with only step-level rollouts and without environment interaction during gradient steps.” Code for the method has been released on GitHub.