New Research Examines LLM Confidence, Failure Prediction, and Domain-Specific Performance

New Research Examines LLM Confidence, Failure Prediction, and Domain-Specific Performance

Several new research papers published on arXiv examine how large language models handle uncertainty and perform in specialized domains.

According to a study on medical reasoning (arXiv:2608.14552), researchers developed a controlled clinical benchmark testing diagnostic choice and confidence in medical LLMs. Using 45 synthetic vignettes comparing Alzheimer-type neurocognitive disorder versus depression-related cognitive impairment, the study found that gpt-4.1-nano achieved 93.5% diagnostic accuracy with a mean confidence of 78.4% and AUROC2 of 0.876. The research identified “partial metacognitive sensitivity,” with confidence increasing with evidence distance from diagnostic boundaries and decreasing when information was missing. However, errors clustered in moderate, conflicting cases where “the model shifted toward DRCI and retained more confidence than empirical accuracy justified.”

A separate study (arXiv:2608.14927) examined whether LLMs can predict when collaboration will help solve problems. According to the research, a post-answer probe using gpt-oss-120b ranked baseline failures with 0.8847 AUROC on 4,151 competition-level math problems, but was “much weaker for identifying PER- or Broadcast-specific value.”

Meanwhile, research on automated program repair (arXiv:2608.14065) found that “over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques,” though higher-cost LLMs don’t consistently yield better cost-efficiency, according to the study accepted at ESEM 2026.