Research Reveals LLMs Abandon Correct Beliefs Under Persuasion Attacks

A new study published on arXiv exposes a critical vulnerability in large language models: their susceptibility to adversarial persuasion. According to the research, “a single targeted persuasive argument is enough to collapse model accuracy to near zero, even when the argument is factually false.”

The researchers introduced an adversarial reinforcement learning framework that trains persuader agents to change a target model’s answer in a single interaction. The results are striking: RL-trained persuaders increased persuasion success from approximately 24% to over 93% against the training-time target model, according to the paper.

These learned strategies also transferred to unseen models, achieving 83% attack success on Qwen-14B, 79% on Llama-3.1-8B, and 25% on GPT-4o-mini, the study reports. When researchers applied curriculum learning—bootstrapping on more persuadable open-weight models before targeting harder ones—GPT-4o-mini attack success increased from 25% to 38%.

According to the findings, optimized persuaders increasingly relied on credibility-based tactics, “including fabricated citations and false authoritative evidence.” The researchers conclude that these findings “expose a critical weakness in current LLM agents: even when they initially reason correctly, they can be steered toward false conclusions by optimized natural language influence.”

The paper positions persuasion robustness as “a necessary safety criterion for multi-agent and human-AI decision-making systems.”