Can AI agents based on large language models (LLMs) take over the work of operations research (OR) professionals when all they get is a client's original documents? A new paper on arXiv presents the benchmark OR-License to address this question. It is based on real, paid consulting projects. The result is mixed: individual solutions outperform the experts, but overall no model matches their reliability.
What happened
According to the publication in the Computation and Language category, OR-License consists of 31 tasks (instances) from 10 industrial scenarios. They come from paid OR projects with a total value of 3.68 million US dollars. After anonymization, the original requirements, the input data, the required delivery formats, and the client's information boundaries are preserved.
This sets the benchmark apart from many earlier tests. In those, the task usually begins with a self-contained text description or an already formulated optimization problem. The difficult translation from messy business material to a professionally validated solution hardly appears at all.
The client simulator ORSim
Real projects run not only through documents but through follow-up questions. For this purpose, the authors introduce ORSim, a controlled client simulator. It reveals business facts but supplies no OR expertise. The agent must therefore work out for itself which information is missing and ask targeted questions.
Evaluation against the expert solution
The evaluation remains hidden from the agents. It first checks whether a delivery is feasible. It then compares the quality of the objective function with the original expert solution, which the client had already accepted before the benchmark was built.
The main metric is Deployment-Adjusted Quality (DAQ). It is the average of the quality relative to the expert across all instances. Infeasible deliveries receive zero points. The expert reference is thus set at 1.
The results
11 leading language models were tested, running in coding-agent environments. The key findings:
- Several models exceed the expert reference in conditional objective quality, that is, where they actually deliver a feasible solution.
- No model reaches the experts' DAQ value.
- The strongest main configuration, using Claude Code, reaches 88.39% strict feasibility and a DAQ of 0.87.
| Metric | Expert reference | Strongest Claude Code configuration |
|---|---|---|
| DAQ | 1 | 0.87 |
| Strict feasibility | – | 88.39% |
The authors also observe that gathering information from the client involves more than mere clarifying questions. It runs through solution development and validation. In some later phases, the share of confirmed, resolved problems is higher.
The analysis of the trajectories shows two typical types of errors: errors in problem formulation and errors in implementation. The task-adjusted completion rate of the "research chain" ranges between 24% and 45% across the evaluable tasks, depending on the model.
Background
Operations research deals with mathematical decision support in companies. The question is how scarce resources can be used so that a goal is achieved as well as possible, for example in planning, allocation, or process control. A typical setup consists of three elements: an objective function, constraints, and data.
In practice, the main work often lies not in solving but in what comes before. Clients supply spreadsheets, requirements, and meeting notes, but rarely a clean mathematical model. Anyone who builds a model that overlooks an important rule ends up with a solution that is formally optimal but unusable in operations. That is why feasibility counts so heavily in the new benchmark: a deliverable that violates rules receives no partial credit.
Earlier OR benchmarks for language models mostly test whether a model can derive a model and a solution from a well-described task. OR-License shifts the focus to the entire path: reading the material, spotting gaps, asking the client questions, modeling, implementing, checking, and delivering in the required format. This fits a broader trend of measuring agents no longer only on short individual tasks, but on long, realistic workflows.
Another arXiv paper, from the field of Artificial Intelligence, shows a different facet of this trend. The RealtimeBench benchmark measures agents in games that keep running while they are thinking. There, the point is that a correct action can also come too late. Both works share the basic idea that real-world conditions challenge agents differently than self-contained laboratory questions. The topics are different, however, and the results cannot be compared directly.
- Industrial AI Agents: LLMs in Smart Manufacturing
- PAWM: World Model Warns LLM Agents of Pitfalls
- Taste-Bench: AI Agents Struggle With Foresight
- Microsoft: AI Agents as an Insider Risk in Companies
- Anything LLM: Local AI with Chat, Web Search and Knowledge
What this means
The findings are a snapshot of a benchmark with 31 instances. Generalizations should be made cautiously. Nevertheless, a few practical conclusions can be drawn.
For developers of agent systems
The error analysis suggests that problem formulation and clean implementation are especially critical. Those building agents for optimization tasks can derive the following:
- Feasibility should be checked automatically before a result is delivered.
- Follow-up questions to clients or specialist departments should not only come at the start, but should also be possible during solving and during validation.
- The agent should learn from errors in a traceable way. The authors speak of self-improvement that is tied to evidence.
For small and medium-sized businesses
The gap between 0.87 and 1 sounds small. But it means that even the strongest configuration does not match the reliability of the experts, because infeasible deliverables enter the average as zero. Anyone considering tackling optimization tasks such as planning or allocation with an agent should therefore have the results reviewed by a specialist. Checking whether all business rules are met is particularly important. That individual solutions turn out better than those of the experts is a sign of potential, but not a free pass.
For interested readers
The study shows that good language models are not automatically good consultants. Consulting means recognizing incomplete information, asking the right questions, and translating a problem faithfully. The completion rates of the research chain, from 24% to 45%, make clear that long task chains have so far often not been run through completely.
Overall, the authors name three directions of development for reliable agents over long time horizons: closing information gaps persistently, formulating problems faithfully to the original, and improving themselves on the basis of evidence. Whether this succeeds will have to be shown by further studies with more tasks.
Frequently asked questions
What is OR-License?
OR-License is an end-to-end benchmark with 31 cases from 10 industrial scenarios. The cases come from paid operations research projects and, after anonymization, retain the requirements, data and delivery formats.
How does this benchmark differ from earlier OR tests?
Earlier tests usually start with a finished description or optimization task. OR-License starts from the original client documents, without an expert first reformulating the problem.
What does the DAQ metric measure?
Deployment-Adjusted Quality averages the quality relative to the expert solution across all cases. Infeasible deliveries receive a value of zero.
How did the tested models perform?
11 leading LLMs were tested. Some exceed the expert solution in conditional objective quality, but none reaches its DAQ. The best Claude Code configuration reaches 88.39% strict feasibility and a DAQ of 0.87 versus 1.
Where do the agents mainly fail?
The analysis of the trajectories points to errors in problem formulation and implementation. The completion rate of the research chain ranges from 24% to 45%, depending on the model.
