Anyone who deploys LLM agents with tool access knows the problem: two runs from exactly the same starting state do not necessarily end the same way. A new paper on arXiv introduces Recursive Semantic Divergence (RSD), a metric designed to measure exactly this discontinuity. It can also be used as a training signal to make agents more reliable.
What happened
According to the paper "Recursive Semantic Divergence for LLM Agent Consistency," LLM agents that use tools behave inconsistently. When they are started several times independently from the same state, they sometimes call different tools, produce contradictory outputs, or pursue different strategies. The authors say these deviations compound over time and undermine reliability in practical use.
What is RSD?
RSD is a metric for consistency at the level of entire trajectories. It requires no supervision through labels (unsupervised) and is derived from so-called bisimulation metrics from reinforcement learning. RSD measures the expected semantic disagreement between two independent runs of the same agent from the same starting state, accumulated over time.
To assess whether two statements contradict each other, RSD uses Natural Language Inference (NLI). This is a technique that checks whether two texts are logically compatible or mutually exclusive.
How does RSD work?
The metric is defined recursively over future states: the divergence now is the divergence at the current step plus the expected divergence in the following steps. Since one does not want to compute the future in full, the authors approximate this quantity with a neural network. It is trained on independent runs.
The trained model estimates, from the current state, how far the trajectories are likely to diverge in the future. This yields a consistency value in every round, without the entire trajectory having to be completed.
The results
When RSD is used as a reward signal during fine-tuning, the study reports that the spread in task progress drops significantly on two benchmarks:
| Benchmark | Variation before | Variation with RSD |
|---|---|---|
| tau2-bench | 132 % | 91 % |
| toolsandbox | 90 % | 44 % |
The authors also report improved results across several benchmarks and model sizes. When RSD is used as an additional reward signal, training reaches the results of supervised fine-tuning, but with considerably lower variance. No true reference labels (ground truth) are needed for this.
Background
The usual evaluations of agents mainly ask: was the task solved? This is usually done via final scores, that is, scores that rate the success of a single run. Such metrics barely capture how stably an agent behaves across multiple runs. An agent can perform well on average and still fluctuate strongly from run to run.
Previous analytical approaches to consistency also have a gap. One can look at behavior at the token level, that is, at the probability distributions with which the model selects the next piece of text. Or one can look at individual responses and group them by meaning (semantic clustering). In the authors' view, neither captures how an inconsistency propagates through later rounds of a trajectory. This is exactly what is decisive for agents: they act in multiple steps, and an early deviation can shift later decisions.
Why the approach comes from reinforcement learning
In reinforcement learning, an agent learns through rewards. Bisimulation metrics measure there how similar states are with respect to their future behavior. RSD transfers this idea to language agents: instead of states, it compares two independent runs of the same agent and examines the meaning of their outputs. The idea of recursively summing up deviations over the future fits multi-step tasks well.
A look at related work
Consistency is also a topic in other areas of language processing. Another paper from the same arXiv category, for example, examines how well LLM-based annotators agree with human raters. There, agreement is measured with Fleiss' kappa. The topic is a different one, but the underlying question is similar: how reliable and traceable are the decisions of language models when several instances make judgments? The studies are not directly comparable and do not build on each other.
What This Means
The results come from a preprint on arXiv. They therefore represent the current state of research, not an independently confirmed standard. Still, a few cautious conclusions can be drawn.
What does RSD mean for users?
For people who use agents in everyday life, for example for research, scheduling, or customer inquiries, reliability is often more important than a single peak score. If the same request leads to one result at one time and a different result at another, trust suffers. Consistency measurement such as RSD could help agents respond more predictably to identical starting conditions. Whether and when such methods will reach products is not stated in the material.
For developers
- RSD provides a consistency score for each round. This can help identify uncertain points in a trajectory even before the run ends.
- The metric requires no reference labels. This can reduce effort when there are no clearly correct solutions for your own task.
- As an additional reward signal during fine-tuning, RSD reduced fluctuations in the study. Those who train their own agents could examine whether a similar signal makes sense in their own setup.
- Consistency should not stand alone: an agent that always makes the same mistake would be consistent, but not good. Combining it with measurements of task quality remains sensible.
For small businesses
Those who integrate agents into workflows, for example in support or internal processes, should test runs several times from the same state, not just once. The study shows that the spread in task progress can be considerable. Such a repeat check can show whether the agent is reliable enough for its intended use. Note: RSD relies on NLI models to detect contradictions. Their own errors can affect the measurement. The material does not address this in detail, so a critical review is worthwhile.
Open questions
- The results relate to the benchmarks named in the study. How well they transfer to other tasks remains to be examined.
- The figures on spread describe the variation in task progress, not reliability in every conceivable scenario.
- Whether RSD will be widely used in practice is an open question.
Frequently asked questions
What is RSD?
RSD stands for Recursive Semantic Divergence. It is an unsupervised, trajectory-level metric that measures the consistency of LLM agents that use tools. It is derived from bisimulation metrics in reinforcement learning.
How does RSD work?
RSD measures the expected cumulative semantic divergence between two independent runs of the same agent starting from the same state. Logical contradictions are detected with natural language inference. The metric is defined recursively over future states and is approximated by a neural network.
Why are existing evaluation metrics not enough?
Common metrics mainly measure the correctness of task completion through final scores. They do not capture how consistent an agent's behavior is across multiple runs, nor how deviations propagate through later steps.
What does RSD mean for users?
Agents fine-tuned with RSD as a reward vary less in task progress: from 132% to 91% on tau2-bench and from 90% to 44% on toolsandbox. This can make deployment more reliable.
Does RSD need ground-truth labels?
No. As an additional reward signal, RSD matches the performance of supervised fine-tuning with much lower variance, without ground-truth labels.
