Researchers have introduced CESBench, a new benchmark designed to evaluate large language models on cryptographic engineering security for Internet of Things (IoT) devices, according to a paper published on arxiv.org.
According to the paper, CESBench contains 380 expert-written items spanning six sub-domains: side-channel, fault injection, implementation, countermeasures, evaluation, and integration. The benchmark features four task types testing different competencies: 209 multiple-choice items test recall, 67 judgment items require security verdicts with justifications, 63 scenario items require engineering diagnoses, and 41 code tasks graded by 572 test cases.
The researchers tested 11 open-weight and proprietary LLMs on the benchmark, with composite scores ranging from 54.4% to 83.6%. According to the results, the top score on each task type was 98.6% for multiple choice, 95.1% for code, and 88.4% for scenario diagnosis, but only 58.8% for judgment tasks.
The paper notes that while 88.5% of verdicts were correct across models, their justifications earned only 53.4% of the rubric marks. According to the researchers, multiple choice is near its ceiling for the strongest models and most code tasks are solved, whereas justifying a security verdict remains the weakest competence. The benchmark, prompts, and per-item results have been made public.