Bioinformaticians are handing ever larger analyses over to AI agents, often in long sessions with many intermediate steps. Most benchmarks, by contrast, test only a short task with a single prompt. According to a publication on arXiv, the new OmicsBench benchmark is meant to close this gap.
How the benchmark is built
OmicsBench comprises 97 tasks. Each consists of dependent subtasks linked as a directed acyclic graph. Put simply, some steps build on the results of earlier ones, and there are no loops. In total, the tasks contain 662 subtasks.
Each subtask is posed as its own user turn and is scored automatically against an answer key. Optionally, if the answer is wrong, the agent receives feedback in its context and can try again.
Results: still a long way to go
12 models were tested on three harnesses. These are software frameworks that turn a language model into an agent in the first place. The slowest agent needed an average of 3.6 hours per task.
No combination comes close to the maximum score. Without feedback, the mean scores range from 0.43 to 0.57 out of a possible 1; with feedback, they range from 0.54 to 0.67.
The multi-step format also reveals something that single prompts do not capture. On subtasks that depend on earlier ones, the agents perform worse. Feedback makes up for part of this loss.
How agents fail
A deterministic evaluation only tells you whether an answer is correct, not why it is wrong. The authors therefore developed an unsupervised pipeline. It describes failed attempts, maps them to vectors and clusters them so that error patterns become visible on their own.
Applied to OmicsBench, the pipeline reveals qualitative weaknesses. Some agents use their own statistical filters even though others were specified. Others write entire answers from memory instead of computing them from the data.
- OR-License: How Good Are LLM Agents at Operations Research?
- Taste-Bench: AI Agents Struggle With Foresight
- Microsoft: AI Agents as an Insider Risk in Companies
- PAWM: World Model Warns LLM Agents of Pitfalls
- Industrial AI Agents: LLMs in Smart Manufacturing
What this means in practice
Anyone who uses agents for analyses should check intermediate results and verify that the specified methods were actually applied. Especially in long workflows, early errors can carry over into later steps.
