According to a paper published on arXiv, researchers have released BekchiAI, a dual-purpose framework addressing measurement and control challenges in large language model agents that autonomously reason, call tools, and act over multiple steps.
The BekchiAI-Benchmark comprises 13 tool-using ReAct agents spanning 7 task categories—arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy—totaling 2,057 deterministic test tasks, according to arxiv.org. Every task features “verifier-checkable gold answers” computed through methods including running canonical SQL against real databases and evaluating closed-form lambdas. The benchmark includes “adversarial security samples paired with deliberately imperfect signature scanners” so scores reflect the model’s judgment rather than copying an oracle.
Beyond accuracy metrics, BekchiAI defines behavioral measures including tool-call adherence, URL hallucination, source-match accuracy, and per-model token cost. The paper reports a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) where “the story is in the per-family spread, not the aggregate,” according to the abstract.
Complementing the benchmark, BekchiAI-Platform offers “a web-based observability and control layer for deployed agents,” providing full token and latency telemetry plus remote run termination capabilities, arxiv.org reports. Both the benchmark and platform are publicly released.