Researchers have introduced CESBench, a new benchmark designed to evaluate large language models on cryptographic engineering security for Internet of Things devices, according to a paper published on arxiv.org.
According to the paper, CESBench contains 380 expert-written items spanning six sub-domains: side-channel attacks, fault injection, implementation, countermeasures, evaluation, and integration. The benchmark uses four task types targeting different competencies: 209 multiple-choice items testing recall, 67 judgment items requiring security verdicts with justifications, 63 scenario items requiring engineering diagnoses, and 41 code tasks graded by 572 test cases.
The researchers validated the benchmark by testing 11 open-weight and proprietary LLMs. According to arxiv.org, composite scores ranged from 54.4% to 83.6%. The top scores by task type were 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment tasks.
The paper highlights a significant gap in model performance: “Across models, 88.5% of verdicts are correct, yet their justifications earn only 53.4% of the rubric marks,” according to arxiv.org. The researchers note that while multiple choice is near ceiling performance for the strongest models and most code tasks are solved, “justifying a security verdict remains the weakest competence.”
The benchmark, prompts, and per-item results have been made publicly available.