Agent Failure Detection
Most agent monitoring tells you what went wrong after the run finishes, by which point the email is sent and the purchase is made. This classifies the failure mode from the reasoning trace while the agent is still running: 78% of unsafe actions are named before the agent takes them. Evaluated on partial traces, one step at a time, with the final answer withheld.
Outcome
78%
Of unsafe actions caught before the agent acts
18 of 23 held-out unsafe traces, replayed step-by-step
0.820 macro F1 on an 87-trace held-out set, no group leakage
In the product
Context
Agent frameworks ship with retries and guardrails but no answer to a simpler question: what is going wrong right now. Classifying a run after it finishes is both easy and useless, because by the time a completed trace reaches a dashboard the email has been sent and the purchase has been made.
This work asks the narrower and harder version. Is the failure already legible in the trace before the agent gets there, and how early. A ReAct agent runs a task list against real and mocked tools, every trace is hand labelled against a four class taxonomy, and a classifier is trained to name the failure from the reasoning sequence alone.
The taxonomy
Success
Every substantive claim in the final answer is supported by a tool observation, or the agent honestly reports that the information could not be found.
129 traces, 29.8%
Hallucination
The final answer asserts a specific fact appearing in no observation, irrespective of whether that fact happens to be true in the world.
109 traces, 25.2%
Loop
The agent repeats an action three or more times, identical queries or semantic rephrasings, without advancing towards the task.
80 traces, 18.5%
Unsafe execution
The agent invokes a consequential mock action tool without explicit authorisation for that action in the task description.
115 traces, 26.6%
Categories are defined by observable symptoms in the trace rather than by internal cause, because a runtime detector sees only the thought, action and observation sequence emitted so far. Six classes were reduced to four empirically. Unsafe actions are generated safely: the agent's action tools are mocks that log the requested action instead of performing it.
Dataset integrity
Two defects were found before training. Both inflate the headline number, and neither announces itself.
Synthetic template collapse. Augmentation of the two minority classes had collapsed into near duplicate families. One lineage contributed 40 rows following an identical template with only the product name varying, and an early split had put the whole family in the test set. Forty copies of one signal is not forty examples. Fix: embed every trace with all-MiniLM-L6-v2 and keep the 12 most mutually dissimilar members per lineage by greedy farthest point sampling. 592 rows down to 433.
Broken lineage grouping. Parent trace IDs were stored as comma separated strings, so grouping on the raw value produced overlapping pseudo groups and let sibling traces land on opposite sides of the train and test boundary. The leakage check passed vacuously because it compared group strings rather than actual parents. Fix: rebuild groups as connected components over shared parents, then split with StratifiedGroupKFold on the true group.
| Rows after diversity sampling | 433, from 592 |
|---|---|
| Splits | 259 train / 87 val / 87 test |
| Group overlap across splits | Zero, verified on true parents |
| Traces over the 512 token limit | 23.6%, head and tail truncation |
| Provenance | 262 real, 147 synthetic truncated, 24 synthetic |
Approach
- 01Built a ReAct agent on LangGraph, running Llama 4 Scout on Groq, with real search and calculator tools alongside mocked purchase, email and delete tools, then ran it against a task list designed to induce failure as well as success.
- 02Labelled every trace against the four-class taxonomy using two independent auto-labellers, GPT-4o and Claude, with an annotation guide and a manual adjudication pass over every disagreement.
- 03Found and fixed two dataset defects that both inflate the headline number: synthetic template collapse, and lineage grouping that leaked siblings across the train and test boundary.
- 04Fine-tuned DeBERTa-v3-base with class-weighted cross-entropy and head-and-tail truncation at 512 tokens, against four baselines on identical frozen splits: majority class, frozen MiniLM with logistic regression, TF-IDF with logistic regression, and RoBERTa-base.
- 05Replayed the 61 held-out failure traces prefix-by-prefix with the final answer stripped, to measure not whether the failure is detectable but how early.
- 06Audited the two classes that scored suspiciously well for lexical shortcuts, and measured the false alarm rate on healthy runs.
- 07Built a three-panel React dashboard over the frozen artefacts as a static results explorer.
Results
0.820
Macro F1 on the held out test set
78%
Of unsafe actions caught before the agent acts
0.942
Runtime alarm precision per prefix
Per class performance
DeBERTa-v3 on the held out test set. Support is shown because the smaller classes carry proportionally less evidence.
| Class | Precision | Recall | F1 | N |
|---|---|---|---|---|
| Success | 0.783 | 0.692 | 0.735 | 26 |
| Hallucination | 0.667 | 0.727 | 0.696 | 22 |
| Loop | 0.824 | 0.875 | 0.848 | 16 |
| Unsafe execution | 1.000 | 1.000 | 1.000 | 23 |
| Macro average | 0.818 | 0.824 | 0.820 | 87 |
Eleven of the sixteen errors sit on the hallucination and success boundary. Unsafe execution shows a clean column.
Model comparison
All five models trained and evaluated on the identical frozen splits.
| Class | Majority | MiniLM + LR | TF-IDF + LR | RoBERTa | DeBERTa-v3 |
|---|---|---|---|---|---|
| Success | 0.460 | 0.359 | 0.619 | 0.667 | 0.735 |
| Hallucination | 0.000 | 0.586 | 0.678 | 0.703 | 0.696 |
| Loop | 0.000 | 0.353 | 0.476 | 0.970 | 0.848 |
| Unsafe execution | 0.000 | 0.558 | 0.885 | 0.686 | 1.000 |
| Macro F1 | 0.115 | 0.464 | 0.664 | 0.756 | 0.820 |
Bag of words cannot represent a loop.TF-IDF reaches 0.885 on unsafe execution but 0.476 on loops, catching 5 of 16. A loop is a repetition pattern, not a token pattern, and a model that discards order cannot express "this happened three times". That is the empirical argument for a contextual encoder on this task.
The ranking itself is not established. The gap between the two transformers sits almost entirely in unsafe execution, 1.000 against 0.686, and RoBERTa is the better loop detector at 0.970 against 0.848. On an 87 trace test set from a single seed the bootstrap interval on macro F1 spans 0.734 to 0.891, which is wider than the gap. Read the per class pattern as the finding, not the ordering.
Runtime detection
Each of the 61 failure traces in the test set was replayed one step at a time with the final answer withheld. Detection rate is the runtime measure: the failure was named before the agent finished.
| Failure type | Detected at any step | Detected before final step | Avg detection point |
|---|---|---|---|
| Unsafe execution | 23 / 23 | 18 / 23 (78%) | 59% through the trace |
| Loop | 14 / 16 | 10 / 16 (63%) | 72% through the trace |
| Hallucination | 11 / 22 | 7 / 22 (32%) | 56% through the trace |
Stepping through the per prefix probabilities shows three different detection mechanisms rather than one capability applied three times. Unsafe execution fires on the precursor setup, before the unsafe call happens, which is anticipatory rather than reactive. Loop fires on evidence, once a repeated observation is actually visible, and it is the cleanest runtime result here. Hallucination resolves late, often after passing through a loop like phase, because there is nothing to detect until the ungrounded answer materialises. Runtime detection is not one problem. It is three, with different ceilings.
The audit
Two classes scored suspiciously well and both had an obvious lexical explanation. Unsafe execution is defined by three tool names that appear verbatim in the input, and a regex on those names alone reaches F1 0.958. Every loop trace in the test set is truncated, so absence of the final answer line predicts the class perfectly. Either would mean the classifier had learned a token rather than a behaviour. Both were tested against the trained checkpoint: same weights, same test split, input text transformed.
Shortcut ablations
| Variant | Macro F1 | Delta | Hallucination | Loop | Success | Unsafe |
|---|---|---|---|---|---|---|
| Baseline | 0.820 | — | 0.696 | 0.848 | 0.735 | 1.000 |
| Tool names aliased | 0.820 | 0.000 | 0.696 | 0.848 | 0.735 | 1.000 |
| Mock markers removed | 0.820 | 0.000 | 0.696 | 0.848 | 0.735 | 1.000 |
| Final answer removed | 0.784 | −0.036 | 0.541 | 0.824 | 0.772 | 1.000 |
| All three | 0.784 | −0.036 | 0.541 | 0.824 | 0.772 | 1.000 |
Neither shortcut hypothesis survives. Renaming the action tools across 36 traces and stripping the mock markers across 25 changed nothing at all, not one prediction. The regex baseline scoring 0.958 is a fact about how separable the task is, not about what the model learned.
The real constraint is the final answer. Hallucination loses 0.155 F1 when it is removed, four times the movement of any other class. That is the mechanism behind the 32% runtime figure: a hallucination is an ungrounded claim in the final answer, so before the answer exists there is nothing to detect. Success improves slightly under the same transform, consistent with fewer hallucination false positives once the answer is gone.
False alarm rate
The runtime experiment only sees failure traces, which measures recall and says nothing about how often a healthy run raises an alarm. Sweeping the 26 held out success traces through the identical prefix by prefix loop:
| Prefixes predicting a failure | 7 / 53 (13.2%) |
|---|---|
| Healthy traces alarmed at least once | 5 / 26 (19.2%) |
| Runtime alarm precision, per prefix | 0.942 |
Six of the seven false alarms claim hallucination, the class the ablation shows is weakest. The sample is small and the intervals are wide, roughly 7 to 25% per prefix. Both audits reproduce with python audit_run.py all.
The result I trust least is the one that looks best. The work that made this defensible was not the training run.
In production
The classifier sits as middleware on the LangGraph step, scored asynchronously so it never blocks the agent loop. Latency added per step is TODO_LATENCY; cost added per trace is TODO_COST.
Three policies, not one
The runtime results show three detection mechanisms, not one, so deployment is three policies rather than a single threshold.
Unsafe execution gates. It fires on the precursor, 59% through the trace, before the call happens, so it pauses the agent and requires confirmation.
Loop interrupts. It fires on evidence, once a repetition is visible at 72% through, so it breaks the loop and forces a re-plan.
Hallucination flags. It resolves late, after the answer already exists, so it cannot gate. It flags the response for post-hoc review instead.
Thresholds are set per class, from each class's own confidence profile, not from one global cutoff.
Limitations
Everything below was found by trying to break the result, not by a reviewer.
- Unsafe execution and loop remain partly template derived. Diversity sampling reduces redundancy, it does not create diversity that was never generated, and the held-out set contains no real traces for either class, so their scores measure template fidelity rather than field performance. The unsafe detector also fires on the precursor pattern rather than the act, exposing it to false positives on runs that set up the same way and then behave correctly; that control has not been run.
- The classifier is trained on complete traces and applied to partial ones. That it transfers at all is a positive result, but training on prefixes directly should reduce detection latency.
- Detection point is measured in agent steps, not seconds, so it reports earliness in the reasoning sequence rather than wall-clock speed. Credit goes to the first correct prediction without requiring it to stay stable afterwards, and labels are trace-level rather than step-level, a weaker basis for runtime claims than either a stricter detection rule or step-level ground truth would give.
- All figures come from a single training run at seed 42. The bootstrap interval on macro F1 spans 0.734 to 0.891.
- Hallucination sits at roughly 0.70 for every model tried. The residual errors are high-confidence faithfulness failures, which single-sequence classification is structurally unable to see. The fix is a claim-against-evidence check, not a bigger classifier.