Three new research papers published on arXiv demonstrate novel approaches to improving large language model training and performance with limited data.
According to arxiv.org, researchers propose Prior-Guided Tuning (PGT), which incorporates natural-language priors as auxiliary learning signals for low-resource LLM training. The paper introduces Contrastive Prior Steering (CPS), achieving 97.6% average exact-match accuracy on AmbiMath and improving average Macro F1 by 9.5 percentage points over standard fine-tuning on Jigsaw. Notably, CPS “with 1/10 of the experimental training data slightly exceeds full-data plain fine-tuning” on Jigsaw.
A separate arxiv.org paper presents DKL (Decoupled Knowledge Learning) for instruction-tuned language models. DKL performs extended pre-training on base LLMs rather than instruction-tuned models, then merges the knowledge-infused weights to avoid compromising instruction-following capabilities. According to the paper, DKL improves RAG accuracy from 54.17 to 79.26 on retrieval failure cases “while outperforming prior approaches with substantially less training data.”
Finally, researchers introduced InternGeometry, an LLM agent for olympiad-level geometry problems. According to arxiv.org, InternGeometry “solves 44 of 50 IMO geometry problems (2000-2024), exceeding the average gold medalist score (40.9), using only 13K training examples, just 0.004% of the data used by AlphaGeometry 2.” The system uses Complexity-Boosting Reinforcement Learning and can “propose novel auxiliary constructions for IMO problems that do not appear in human solutions.”