xOffense Framework Achieves 79% Success Rate in Automated Penetration Testing Using Fine-Tuned LLM
Researchers have introduced xOffense, an AI-driven multi-agent framework designed to automate penetration testing by shifting the process “from labor-intensive, expert-driven manual efforts to fully automated, machine-executable workflows,” according to a paper published on arxiv.org.
The framework leverages a fine-tuned, mid-scale open-source language model (Qwen3-32B) to drive reasoning and decision-making in penetration testing. According to the research, xOffense assigns specialized agents to reconnaissance, vulnerability scanning, and exploitation, with an orchestration layer ensuring coordination across phases.
The model was fine-tuned on Chain-of-Thought penetration testing data to “generate precise tool commands and perform consistent multi-step reasoning,” the paper states. When evaluated on two benchmarks—AutoPenBench and AI-Pentest-Benchmark—xOffense achieved a sub-task completion rate of 79.17%, according to the researchers.
According to arxiv.org, these results “decisively surpass” leading systems such as VulnBot and PentestGPT. The researchers conclude that their findings “highlight the potential of domain-adapted mid-scale LLMs, when embedded within structured multi-agent orchestration, to deliver superior, cost-efficient, and reproducible solutions for autonomous penetration testing.”
The 17-page paper was submitted to arxiv.org in September 2025 by Quyen Nguyen Huu and colleagues.