According to arxiv.org, researchers have introduced Policy Aligned Critic Training (PACT), a new reinforcement learning approach that addresses token-level credit assignment in large language model post-training. The paper formulates three regularity conditions—Completeness, Prefix Consistency, and Neutrality—that uniquely determine token-level credit, providing “a unified basis for explaining phenomena across existing algorithms.”
PACT adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and “better align the critic with the updated policy,” according to the paper. In agentic mathematical reasoning, PACT achieved 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieved a 67.4% pass rate, surpassing PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points.
In related work, arxiv.org published research on VERPO (Verified Evidence Regularized Policy Optimization), which addresses limitations in Group Relative Policy Optimization by converting “evidence-conditioned guidance into reward-aligned token-level credit assignment.” According to the paper, VERPO decomposes teacher guidance into an evidence-free reference term and signed, evidence-induced corrections at each token, using a stopped controller that balances alignment with local GRPO update direction against Fisher movement cost.
Both approaches aim to refine how credit is assigned during LLM training, potentially improving model performance on complex reasoning tasks.