A new study accepted for publication at ESORICS 2026 reveals that current AI safety benchmarks fail to reliably evaluate Small Language Models (SLMs), according to research published on arxiv.org.
The researchers conducted a large-scale assessment of five widely used benchmark suites across 26 open-source SLMs. According to the paper, “ambiguous judgments dominate and correlate with prompt complexity and model architecture, indicating that LLM-centric safety benchmarks are insufficient as standalone evidence for SLM safety assessment.”
The study found that ambiguity rates increase with lexical density, output perplexity, and output length, while decreasing with lexical sophistication and self-coherence. This reveals what the researchers call a “capability-safety confound that mixes model capability with apparent safety.”
According to the findings, the prevalence of ambiguous results makes aggregate safety rankings “mathematically brittle”—model rankings change significantly under different interpretations of ambiguous responses, even when the underlying outputs remain unchanged.
The research evaluated responses using a unified rubric that assigns scores of 0 (harmful), 1 (safe), or 0.5 (ambiguous/irrelevant). The results highlight challenges in assessing SLMs, which are increasingly deployed in resource-constrained and privacy-sensitive settings where safety failures can cause security and societal risks, according to the paper.