According to a new paper published on arXiv.org, large language models face fundamental limitations when following multiple instructions simultaneously, despite handling individual constraints well.
The research introduces Constraint Saturation Evaluation (CSE), a benchmark testing 15 models across 36 constraint types with 369,753 checks at constraint counts ranging from 1-12. According to the paper, the evaluation uses “deterministic, rule-based verifier and zero LLM-judge involvement.”
The study reports three key findings. First, while per-constraint pass rates decay gradually, the probability of satisfying all constraints “collapses” - a model passing individual constraints at approximately 41% at k=8 succeeds on all eight just 5.7% of the time, according to arxiv.org.
Second, structural constraints degrade twice as fast as lexical ones, losing “2x more baseline capability per added constraint,” the paper states. Third, failures are “nearly independent,” making the accumulation multiplicative.
The research concludes that “reliable instruction following breaks down beyond 5-6 simultaneous constraints,” with probe-level success falling below 50% at 7 constraints for the strongest model tested, and “at 3 or fewer for 12 of 15” models evaluated. The paper was reviewed in the ARR May 2026 cycle, according to the arXiv listing.