Two recent papers on arxiv.org address fundamental scaling challenges in AI systems, focusing on medical evaluation and multilingual capabilities.
According to arxiv.org, a study by Thomas A. Buckley and colleagues examines how to scale clinical judgment for evaluating medical AI. The research addresses limitations in current methods for assessing clinical reasoning in large language models (LLMs), which typically rely on “blinded physician evaluation” considered by many as the gold standard. However, this approach is “difficult to scale,” leading prior studies to depend on “small physician panels, often from a single institution or specialty.” This limitation restricts “the scientific questions investigated” and creates uncertainty about whether findings would be reproducible across different contexts.
In a separate paper, researchers demonstrate that existing multilingual watermarking methods for LLMs are “not truly multilingual.” According to the study, current methods “fail to remain robust under translation attacks in medium- and low-resource languages,” despite claims of cross-lingual robustness. The research traces this failure to “semantic clustering, which fails when the tokenizer vocabulary contains too few full-word tokens for a given language.”
To address this gap, the researchers introduce STEAM, a detection method using Bayesian optimization to search among 126 candidate languages for optimal watermark recovery. According to arxiv.org, STEAM achieves “average gains of +0.23 AUC and +37%p TPR@1%” and is “compatible with any watermarking method, robust across different tokenizers and languages.”