IndicSafeEval Framework Exposes Safety Vulnerabilities in Multilingual LLMs

IndicSafeEval Framework Exposes Safety Vulnerabilities in Multilingual LLMs

Researchers have introduced IndicSafeEval, a new evaluation framework revealing significant safety vulnerabilities in large language models when tested in non-English languages. According to arxiv.org, the benchmark combines ten safety-critical content categories with six persuasive strategies across four Indian languages—Hindi, Bengali, Marathi, and Punjabi—resulting in 7,200 adversarial prompts.

The study, accepted to Findings of EMNLP 2026, found that “the model does not behave equally safely across all languages and prompt styles,” according to the paper. Instead, safety performance “depends strongly on both the languages used and the way a request is phrased using persuasive cues.” Different risk categories showed varying vulnerability levels, with some harmful content types proving “significantly more susceptible to persuasion-based jailbreaks than others.”

The findings underscore limitations in current safety evaluations, which the researchers note are “largely English-centric.” According to arxiv.org, this reveals gaps in understanding “how alignment failures manifest in low-resource and culturally diverse languages.”

Separately, related research on arxiv.org proposes “representational similarity optimization,” which aligns model representations with human moral judgments rather than just responses. This approach reportedly “consistently improved adversarial robustness across model scales on diverse benchmarks and attack strategies,” offering an alternative method for achieving generalizable safety.

The IndicSafeEval implementation is available on GitHub, though the paper contains a warning about potentially offensive example data.