Motivation and research questions

A failed task is a starting point, not an explanation. Tool-using agents can fail through decisions, policy misunderstandings, tool interactions, runtime problems or uncertain grading.

  • Trustworthy measurement: which checks and coverage measures support defensible comparisons?
  • Evidence-led diagnosis: which trajectory evidence supports a failure hypothesis, and where is attribution still uncertain?
  • Controlled improvement: do prompt, skill or scaffold changes transfer beyond development tasks?

Research architecture

Agent Improvement Loop

Development Evaluation / iterative research

  1. Task EnvironmentDevelopment tasks, policy, tools and state
  2. AgentModel, prompts, skills and scaffold
  3. TrajectoryMessages, tool calls and outcomes
  4. Development EvaluatorExplicit criteria and grader identity
  5. Failure AnalysisEvidence and human hypotheses
  6. OptimizationControlled changes; future experiments
  7. Improved AgentCandidate / research objective

Improved Agent returns to Development Evaluation
Re-run the updated candidate through the full execution and evaluation pipeline.

Separate lane / Frozen Held-out Evaluation

  1. Freeze candidate and protocol
  2. Untouched held-out tasks
  3. Fixed evaluator
  4. Generalization assessment

Freeze model, prompts, skills, scaffold, split, rubric, judge, seeds, trials, metrics and budget before assessment. Held-out outcomes and traces do not feed prompt/skill selection or optimization. Audit prior task exposure; native train/test labels alone do not establish an untouched holdout.

"Improved Agent" names a candidate to test, not an observed improvement. If held-out feedback informs later development, disclose exposure and establish a fresh holdout before another generalization claim.

Evaluation methodology

Start with tau2 Retail and its native task criteria and reward combination. Preserve interaction evidence before grading, distinguish agent, simulated user and conditional judge roles, and separate task outcomes from execution or grading errors.

The DeepSeek NL judge is an explicitly identified variant, not original GPT-4.1 scoring or a verified equivalent. Synthetic OpenRouter/DeepSeek connectivity checks establish a call path, not Retail success or grader quality. Comparisons must identify the task set, model, grader, protocol and coverage.

Evaluation Harness v0.1 / target design
  1. Configuration and safety preflightExplicit tasks, trials, model roles and workload limits
  2. Local Python adapter / native tau2 interactionAgent and User Simulator / Retail tools and environment
  3. Pre-grading trajectory checkpointPreserve evidence before the evaluator runs
  4. Native evaluator + explicit DeepSeek NL judge variantNative task criteria and reward combination
  5. Metrics, report and human annotationCoverage, error separation and evidence-led diagnosis

Roadmap and current work

  1. Evaluation Harness — design in progress. Configuration safety, reproducible evidence, transparent grading and reports. Integration and the existing baseline wrapper establish prerequisites; new implementation and offline tests remain planned.
  2. Failure taxonomy — planned. Human-reviewed categories and hypotheses tied to concrete trajectory evidence.
  3. Skill and prompt optimization — future research. Controlled development experiments, followed by separate frozen held-out assessment.
  4. Post-training — future research. Explore learning from reviewed trajectories after establishing evaluation. AutoDL is optional planned compute, not an existing integration.

Experiment summaries

Experiment 001: tau2 Retail baseline — planned. Establish a small, inspectable baseline with fixed task/model/grader provenance and explicit coverage. No benchmark scores or improvement claims are available yet.

Summaries will be manually reviewed before publication. Raw trajectories, private experiment artifacts and credentials are never automatically exported to this website. The Python research environment remains independent.