ConfidenceBench Reveals Calibration Gaps in Frontier LLMs, New Benchmarks Test Medical Watermarking and Lightweight Model Efficiency

Researchers have released ConfidenceBench, a new benchmark evaluating confidence calibration in 15 frontier large language models (LLMs) using the Brier score. According to arxiv.org, the benchmark comprises 200 private multiple-choice questions across spatial reasoning, high-precision mathematics, word lookup, and unknowable questions. Claude Opus 4.6 and Gemini 3.1 Pro Preview achieved the lowest Brier scores at 0.103, “substantially outperform[ing] the calibrated-random baseline of 0.1875,” while Gemini 3.1 Flash-Lite scored 0.367, indicating “severe miscalibration.” The study found that “the most accurate model is not the best-calibrated, and several models perform worse than the calibrated-random Brier baseline despite reasonable accuracy.”

In a separate study on arxiv.org, researchers present “the first rigorous study of how LLM watermarks affect medical performance,” benchmarking 5 watermarking schemes across 11 LLMs and 7 vision-language models (VLMs). The research found that “watermarking can induce substantial degradation across multiple failure modes, including lexical corruption, hallucinated terminology, and amplified misattribution or omission of image findings.”

Meanwhile, according to arxiv.org, another study introduces a PTME-based framework measuring Precision, execution Time, peak Memory usage, and Energy consumption for lightweight LLMs. The research found that “no single model dominates across all PTME dimensions” and that “selecting lightweight LLMs by size, FLOPs, latency, or accuracy alone can select the wrong deployment candidate.”