According to a paper published on arxiv.org, large language models (LLMs) now pass medical licensing examinations and “in curated cases, can rival physicians at diagnostic reasoning.” However, the research team warns that for autonomous triage of self-presenting patients with minimal clinician oversight, “the evidence of safety does not yet exist.” The authors state that “the gap is not in medical knowledge but in the fidelity of clinical evaluation,” noting that “a model optimized to continue the most probable text is not optimized to act safely.”
In parallel research on benchmark evaluation, arxiv.org published work introducing a “dataset-centric meta-evaluation framework” that audits benchmark datasets at the sample level. The researchers annotated five influential benchmarks—MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA—revealing “pronounced internal heterogeneity that is not captured by aggregate accuracy scores,” according to the paper.
Separately, research on automated program repair published on arxiv.org describes RePaCA, a technique leveraging reasoning LLMs for patch correctness assessment. The approach achieved “83.1% accuracy and an 84.8% F1-score” on Defects4J-derived tests, according to the authors.
Additionally, arxiv.org reports on llvm-harness, described as “the first harness designed to assist LLM agents in understanding and fixing compiler bugs,” which improved performance by 62% when applied to five frontier models.