Inside OpenDiscoveryTrace: Why Output-Only AI Evaluation Fails Autonomous Agents

Autonomous AI agents are increasingly deployed to automate complex scientific discovery, ranging from automated hypothesis generation to molecular design and literature synthesis. However, standard evaluation frameworks suffer from a foundational flaw: they judge agents exclusively on their final deliverables—such as code, generated hypotheses, or scientific summaries. This end-state evaluation creates a severe visibility gap, discarding the entire intermediate reasoning process and making it impossible to distinguish rigorous methodology from serendipitous guessing.

To address this critical auditing limitation, researchers have introduced OpenDiscoveryTrace (arXiv:2609.09203), a public dataset and evaluation framework containing 558 full-execution scientific agent trajectories. By capturing structured, step-level traces across frontier and open-weight language models, the study demonstrates that models with virtually indistinguishable top-line success rates exhibit fundamentally different execution behaviors, failure modes, and error recovery patterns.

The Black-Box Evaluation Dilemma in Scientific AI

When autonomous agents execute complex multi-step workflows, measuring accuracy solely at the endpoint obscures the reliability of the underlying system. A model might deliver a correct scientific conclusion through invalid intermediate assumptions, uncalibrated tool invocations, or brute-force trial and error. In high-stakes environments such as drug discovery, materials science, and genomics, these silent process anomalies introduce significant operational and safety risks.

Without structured trajectory analysis, system architects and safety auditors cannot:

• Audit intermediate scientific reasoning steps against domain ground truth.
• Diagnose specific systemic failure points during tool interaction and state management.
• Determine whether an agent possesses calibrated confidence or suffers from silent hallucinations during intermediate planning.
• Evaluate how effectively an agent identifies its own mistakes and triggers revisions.

The OpenDiscoveryTrace Schema and Dataset Scope

OpenDiscoveryTrace formalizes trajectory logging by capturing a standardized 9-field record at every single execution step. Rather than saving arbitrary text logs, the framework tracks:

• Internal Thoughts: The chain-of-thought and intermediate deduction steps generated prior to action.
• Tool Invocations: The exact API and environment calls executed by the agent.
• Environment Observations: Direct feedback, data tables, or execution outputs returned from tools.
• Error Flags: Detailed taxonomies of runtime, execution, or syntactic failures encountered.
• Revision Triggers: Explicit inflection points where the agent recognized an error state and modified its trajectory.
• Self-Reported Confidence: Quantitative assessments of the model's certainty regarding its intermediate steps.

The dataset captures 124 distinct scientific tasks across four core domains: drug discovery, materials science, genomics, and scientific literature analysis. The corpus contains 558 total trajectories across seven major language models, along with 60 live-retrieval variant trajectories designed to evaluate dynamic information lookup under real-world web constraints.

The evaluation encompasses three leading frontier models—GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro (124 trajectories each, balanced across domains and difficulty tiers)—and four open-weight models: Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini, and Qwen2.5-1.5B (30 trajectories each).

Empirical Findings: Surface Parity vs. Process Divergence

A pilot analysis of 363 LLM-judged trajectories revealed a stark divergence between output metrics and process execution integrity. When evaluated on final output alone, frontier models demonstrated comparable competency, with success rates clustered tightly between 84% and 89%. However, granular process inspection exposed significant behavioral differences.

While Claude Opus 4.6 and GPT-5.4 reached parity in final task resolution, Claude generated 30 times more errors per trajectory than GPT-5.4 (2.5 errors versus 0.08 errors per trajectory; p < 0.0001, Cliff's δ = 0.613). This substantial variance demonstrates that similar final task outcomes can arise from fundamentally dissimilar execution dynamics.

Divergent Error Taxonomies: Tool Misuse vs. Reasoning Breakdown

Beyond raw error frequencies, the study categorizes qualitative differences in where frontier systems encounter friction during multi-step execution:

• Claude Opus 4.6: Errors were predominantly driven by Tool Misuse (66.7% of all recorded errors). The agent exhibited aggressive tool-calling behaviors, frequently encountering schema mismatches, parameter syntax failures, or uncalibrated tool invocations before self-correcting back onto a viable execution path.
• GPT-5.4: In contrast, GPT-5.4 maintained high interface precision with external tools, but when failures occurred, they were characterized by Reasoning Errors (83.6% of recorded errors). In these instances, the model misapplied domain logic or misread environment observations while maintaining syntactically valid tool interactions.

This operational distinction is critical for developers deploying autonomous agent harnesses. A system susceptible to tool misuse can often be fortified through constrained decoding, runtime schema enforcement, and defensive API wrappers. Conversely, an agent experiencing core reasoning breakdowns requires structured chain-of-thought validation, external verification routines, or multi-agent debate protocols.

Benchmark Definitions and Machine Learning Baselines

To establish rigorous standards for process-level auditing, the OpenDiscoveryTrace release defines five dedicated benchmark tasks. These tasks test an evaluation model's ability to classify step-level correctness, predict downstream trajectory failure before task completion, detect anomalous confidence spikes, and identify unprompted revision triggers.

The researchers established baseline performance markers across classical and deep learning architectures, evaluating:

• Linear Baselines (Logistic Regression)
• Ensemble Methods (Random Forests)
• Sequential Deep Models (LSTMs)
• Attention-Based Sequence Models (Transformers)

These baselines provide reference performance metrics for future work aimed at automated trajectory auditing, run-time guardrailing, and early-abort mechanisms for malfunctioning autonomous workflows.

Governance and Engineering Implications for AI Systems

The release of OpenDiscoveryTrace under the Creative Commons Attribution 4.0 International license (CC BY 4.0)—encompassing the trajectory dataset, the trace schema, the execution harness, and the benchmark specifications—marks an important step toward transparent AI evaluation. For enterprise teams and researchers engineering agentic systems, several immediate conclusions emerge:

1. Output-Only Validation Is Inadequate: Validating agent performance solely on unit tests, final code execution, or synthesized summaries conceals silent inefficiencies and structural vulnerabilities. Process traces must become standard artifacts in AI evaluation pipelines.
2. Model Selection Dictates System Guardrails: Knowing whether a base model predominantly fails at tool interaction (such as Claude Opus 4.6) or logical deduction (such as GPT-5.4) allows engineers to tailor middleware validation layers specifically to the model's known failure modes.
3. Proactive Failure Interception: Step-level trace schemas enable real-time anomaly detection, allowing orchestration harnesses to terminate, reprompt, or redirect agents before compute budgets are exhausted on degraded reasoning paths.

As autonomous agents transition from experimental research into mission-critical scientific and enterprise pipelines, process-level observability will be essential to establish verifiable, safe, and auditable artificial intelligence.