Compare two agent versions

A case-by-case comparison shows which cases a new agent version fixed and which it broke, so an unchanged mean cannot hide a regression. Use it before you release a new prompt, model, or tool change. To run the same comparison automatically on every pull request, use Gate pull requests with evals.

Before you begin

Before you start, make sure you have:

Working eval: An eval that runs, as in Run your first eval.
Two versions: The deployed version of your agent and the candidate, both callable as task functions.

Fix everything except task

A comparison is fair only when one thing changed. Hold these fixed and record what changed in config:

Same eval name. Scorers are registered under the eval name, so "baseline" and "candidate" as names would create two different scorers. The run’s identity belongs in config.
Same dataset. Each experiment pins a snapshot, and results pair on dataset_record_id. If you change the cases and the agent together, you can’t tell which change moved the score.
Same scorers. If you change a scorer’s rubric, the new scorer version applies only to later runs.

Run baseline and candidate

This script runs the deployed version and a candidate over the same three cases. baseline_experiment_id records which run the candidate is compared with: the app shows the pair side by side under Evals → Experiments, and client.experiments.list(baseline_experiment_id=...) lists every run compared with a baseline.

compare.py

import os
from atlanai import AtlanClient, Eval, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "release-regression-cases", [
    {"name": "arithmetic", "input": {"question": "What is 2 + 2?"}, "expected": {"value": "4"}, "categories": ["math"]},
    {"name": "geography", "input": {"question": "What is the capital of France?"}, "expected": {"value": "Paris"}, "categories": ["facts"]},
    {"name": "units", "input": {"question": "How many grams in a kilogram?"}, "expected": {"value": "1000"}, "categories": ["math"]},
])


def exact_match(*, output: str, expected: str, **_) -> float:
    return float(output.strip().lower() == expected.strip().lower())


def deployed(question: str) -> str:   # replace with the version in production
    return {"What is 2 + 2?": "4", "How many grams in a kilogram?": "1000"}.get(question, "Lyon")


def candidate(question: str) -> str:  # replace with the version under test
    return {"What is 2 + 2?": "4", "What is the capital of France?": "Paris"}.get(question, "100")


# Same eval name, same dataset, same scorer: only the task and its config differ.
baseline = Eval("release-check", dataset=suite.id, task=deployed, scores=[exact_match],
                config={"agent_revision": "v1"})
challenger = Eval("release-check", dataset=suite.id, task=candidate, scores=[exact_match],
                  config={"agent_revision": "v2"}, baseline_experiment_id=baseline.experiment_id)

print("mean", baseline.summary["scores"]["exact_match"]["mean"], "->",
      challenger.summary["scores"]["exact_match"]["mean"])

Diff results by case

Join the two runs’ results on dataset_record_id, and label each case with its categories from the run’s snapshot. experiment.dataset_snapshot.records carries each record’s categories as they were when the run started, so you can see which kind of case regressed.

categories = {r.id: r.categories or [] for r in challenger.experiment.dataset_snapshot.records}


def by_record(experiment_id):
    return {r.dataset_record_id: r for r in client.experiments.results.list(experiment_id, limit=500).items}


before, after = by_record(baseline.experiment_id), by_record(challenger.experiment_id)
for record_id in sorted(before.keys() & after.keys()):
    b, a = before[record_id].scores["exact_match"], after[record_id].scores["exact_match"]
    if a != b:
        change = "improved" if a > b else "REGRESSED"
        print(f"{change:9} {categories[record_id]} {record_id} trace={after[record_id].trace_id}")

The output shows identical means, yet the candidate broke a math case:

mean 0.6666666666666666 -> 0.6666666666666666
improved  ['facts'] dataset_record_01example trace=2e16cf1e812221c6e8f380ebf8f71b40
REGRESSED ['math'] dataset_record_02example trace=64699fd78c22c3f5bdb06f1089a4685e

Verify regression in trace

Open the regressed case’s trace in the app, or read its spans:

spans = client.experiments.traces.list_spans(challenger.experiment_id, trace_id, fields="core,io")
for s in spans.items:
    print(s.span_type, s.name, str(s.output)[:120])

Compare it with the baseline’s trace for the same record: the model’s reply, a tool call or its arguments, or a retrieval result. The regression is confirmed when the candidate’s trace shows the step that produced the wrong answer.

Track series over time (optional)

Every run of the same eval on the same dataset is one point in a series. List them newest first, and filter on config to follow one model or branch:

runs = client.experiments.list(dataset_id=suite.id, experiment_status="completed",
                               config="agent_revision:v2", sort="-created_at", limit=20)
for e in runs.items:
    print(e.created_at, e.config.get("agent_revision"), e.summary["scores"]["exact_match"]["mean"])

For runs bound to an agent, client.experiments.list(subject_kind="agent", subject_id=...) gives the agent’s history, and the agent’s page in the app shows it.

Troubleshooting

If the baseline and candidate have different scorer IDs, they ran under different eval names. For this and other run errors, see Eval errors.

Next steps

Gate pull requests with evals: Run this comparison on every pull request, with a threshold and a minimum count of regressed cases.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Compare two agent versions

Before you begin

Troubleshooting

Next steps

See also