Researchers have published four studies addressing different aspects of fine-tuning large language models for specialized tasks.
According to arxiv.org, a paper titled “NeuroProlog” presents a neurosymbolic framework that compiles math word problems into executable Prolog programs with formal verification guarantees. The researchers propose a multi-task “Cocktail training strategy” that jointly optimizes three objectives: mathematical formula-to-rule translation, natural language-to-program synthesis, and program-answer alignment. Evaluation on the GSM8K dataset demonstrated that cocktail training improves accuracy over single-task baselines, with the paper noting that “larger models exhibit more readily correctable errors, whereas smaller models show reduced syntactic errors but persistent semantic failures.”
In a separate study on arxiv.org, researchers applied LoRA fine-tuning to Qwen2.5 models for control systems course Q&A. Using 360 system-user-assistant conversations, they tested LoRA ranks of r=4, 8, and 16. According to the paper, the 7B-r16 configuration achieved the highest ROUGE-L score of 0.4093 and BERTScore-F1 of 0.8643.
A third arxiv.org paper introduces FedTAR, a task-aware federated fine-tuning method for Mixture-of-Experts (MoE) LLMs that addresses heterogeneous client data challenges. The method uses Singular Value Decomposition to extract task coordinates and update directions.
Finally, researchers on arxiv.org demonstrated inference-time control over refusal behavior in Llama 3 8B by leveraging categorical refusal tokens, showing that this approach “consistently reduce[s] over-refusals on benign prompts while increasing refusal rates on harmful prompts.”