Configure eval runs

Eval options control what a run records, how it runs, and how you recover when it stops: run settings, concurrency, async code, pinned context, and resuming. Configure them once your eval works and you need runs you can compare, debug, and finish after a crash. For a first run, start with Run your first eval; the Eval options and errors lists every option.

Before you begin

Before you start, make sure you have:

Working eval: An Eval call that runs, as in Run your first eval.
Client: An AtlanClient for reading results back, set up as in the quickstart’s verify step.

Set run options

These are the options to decide on for every run:

from atlanai import Eval

run = Eval(
    "support-agent",                     # stable eval name: scorer names and CI baselines key on it
    dataset="support-agent-cases",       # or data=[...] for inline cases
    task=answer,
    scores=[exact_match, required_facts],
    config={"model": "example-model", "prompt": "v7", "git_sha": "0123abc"},
    baseline_experiment_id=None,         # the run this one should be compared with
    subject_kind="agent", subject_id="agent_01example",   # optional: show it on the agent
    max_concurrency=4,                   # cases in flight; helps only with async tasks
)

import { Eval } from "@atlanai/sdk";

const run = await Eval("support-agent", {
  dataset: "support-agent-cases",        // or data: () => [...]
  task: answer,
  scores: [exactMatch, requiredFacts],
}, {
  config: { model: "example-model", prompt: "v7", git_sha: "0123abc" },
  baselineExperimentId: undefined,
  subjectKind: "agent", subjectId: "agent_01example",
});

Name: Keep it the same across branches and runs. Auto-registered scorers are named after it, and the CI action finds baselines by it.
config: Everything that distinguishes this run: model, prompt revision, commit, temperature. It is stored on the experiment, and you can filter on it with GET /eval/v1/experiments?config=model:example-model. Keep secrets out of it.
Concurrency: Python runs max_concurrency cases at a time, which speeds up only async tasks. TypeScript runs cases one at a time.

Verify run evidence

Read the run back before you trust a mean. A failure is handled like this:

Failure	What `Eval` does	The experiment
The task raises on a case	Records the error on that case, skips its scorers, continues	Completes. The case has no scores, so it is absent from each mean.
A scorer raises	Appends `"<scorer>: <message>"` to the case error; other scorers still run	Completes
A trace does not arrive, a result upload is refused, or the process dies	Raises, naming the experiment ID and the argument to resume it with	Left `running`, so it can be resumed

A run where no scorer produces a usable value finalizes as completed, then raises with the experiment ID. This includes runs where every task errored. Pass require_scores=False (requireScores: false) only when an empty score set is an expected outcome. Read the stored experiment’s summary.metrics.failed_cases (failedCases in TypeScript) and each result’s error to find the cause. The CI action treats an error rate above 20% as a broken run. In TypeScript, summary.scores is typed unknown; cast it before reading, for example run.summary.scores as Record<string, { mean: number; count: number }>.

experiment = client.experiments.get(run.experiment_id)
print(experiment.experiment_status, experiment.summary["scores"])

for r in client.experiments.results.list(run.experiment_id, limit=500).items:
    if r.error or any(v == 0 for v in (r.scores or {}).values() if v is not None):
        print(r.case_id, r.scores, r.error, r.trace_id)

spans = client.experiments.traces.list_spans(run.experiment_id, r.trace_id, fields="core,io")

const experiment = await client.experiments.get(run.experimentId);
const results = await client.experiments.results.list(run.experimentId);
for (const r of results.items) if (r.error) console.log(r.caseId, r.scores, r.error, r.traceId);

const spans = await client.experiments.traces.listSpans({ experimentId: run.experimentId, traceId: results.items[0].traceId });

Results list 50 per page by default and at most 500; page with offset. input and output are stored as objects, so a string or number your task returned reads back as {"value": x}. The span read returns core,io fields by default; add attributes, usage, cost, or error to fields for more.To check a finished run’s evidence, for example after a crash or in a CI job that did not run Eval itself, call verify_experiment (verifyExperiment). It confirms the experiment is terminal and every result has a distinct, queryable trace, and checks a sample of traces for their score spans and span nesting. A content-policy check is reported as MANUAL: review it yourself.

from atlanai import verify_experiment

verify_experiment(client, run.experiment_id).raise_for_status()

Before uploading a result, Eval waits up to 30 seconds for each case trace to be queryable, so no result points to a missing trace. The wait covers a trace that is not there yet and a trace plane that is down or slow: a trace read that fails with traces_unavailable (503) or request_timeout (504) is retried inside the window, respecting the server’s Retry-After when one is sent, until the window is spent and its error fails the run. Any other trace-read error fails the run immediately. If your trace export is slow, raise the wait with trace_verification_timeout_s (traceVerificationTimeoutMs). Turn verification off only for tests against a stub.

Pin context that changed (optional)

config records settings. A context manifest records the version and SHA-256 digest of each input you identify, such as instructions, prompts, skills, tool schemas, or a retrieval snapshot. The SDK writes the manifest, and one digest over it, into the experiment config. Two runs with the same digest recorded the same manifest. Your runner must load and use the content it records; the digest alone does not verify that execution used it.

import hashlib
from pathlib import Path
from atlanai import ContextItem, ContextManifest, Eval

def sha256(path: str) -> str:
    return "sha256:" + hashlib.sha256(Path(path).read_bytes()).hexdigest()

context = ContextManifest([
    ContextItem(kind="prompt", name="system-prompt", version="git:0123abc", digest=sha256("prompts/system.md")),
    ContextItem(kind="tool_schema", name="order-tools", version="git:0123abc", digest=sha256("tools/schema.json")),
])
run = Eval("support-agent", dataset="support-agent-cases", task=answer, scores=[exact_match],
           context_manifest=context)

import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import { createContextManifest, Eval } from "@atlanai/sdk";

const sha256 = (path: string) => `sha256:${createHash("sha256").update(readFileSync(path)).digest("hex")}`;

const contextManifest = await createContextManifest([
  { kind: "prompt", name: "system-prompt", version: "git:0123abc", digest: sha256("prompts/system.md") },
  { kind: "tool_schema", name: "order-tools", version: "git:0123abc", digest: sha256("tools/schema.json") },
]);
const run = await Eval("support-agent", { dataset: "support-agent-cases", task: answer, scores: [exactMatch] },
  { contextManifest });

Field	Required	Rule
`kind`	Yes	A lowercase category: `file`, `prompt`, `skill`, `tool_schema`, `knowledge`, `policy`, `memory`, `harness`, or your own
`name`	Yes	Stable name; `kind` + `name` is unique in one manifest
`version`	Yes	A pinned revision such as a git SHA or release. `latest` is rejected.
`digest`	Yes	`sha256:` plus the hex digest of the exact content
`artifact_id`, `version_ordinal`	No	The registry artifact and version, when the input is registered, such as a skill

The manifest is stored, but the referenced content is not fetched or checked, so point each entry at something you can still retrieve. Then compare runs by what you held constant:

Hold constant	Change	The comparison measures
Dataset and model	Context manifest	A prompt, skill, tool, or knowledge change
Dataset and context	Model config	A model or setting change
Everything	Nothing	Run-to-run noise

Use async_eval in event loops (optional)

Python Eval starts its own event loop, so it fails inside an existing one, such as Jupyter, FastAPI, or an async test, with Eval() cannot run inside an event loop. Await async_eval there instead; it takes the same arguments.

from atlanai import async_eval

run = await async_eval("support-agent", dataset="support-agent-cases", task=answer, scores=[exact_match])

The task and scorers can be sync or async in both SDKs.

Pass case details to task (optional)

A task that declares a second parameter receives hooks: the case’s expected, metadata, and tags, and the span the task runs in. Use the span to attach details to the trace.

def answer(question, hooks):
    hooks.span.update(metadata={"retrieval_k": 8, "case_tags": hooks.tags})
    return your_agent(question, locale=hooks.metadata.get("locale", "en"))

import type { EvalHooks } from "@atlanai/sdk";

// EvalHooks<Expected, Metadata>: match the types of your cases' expected and metadata.
const answer = async (question: string, hooks: EvalHooks<string, { locale?: string }>) => {
  hooks.span.update({ metadata: { retrieval_k: 8, case_tags: hooks.tags } });
  return yourAgent(question, hooks.metadata.locale ?? "en");
};

Resume stopped run (optional)

Save the experiment ID as soon as the run starts, then resume with the same arguments. Eval skips the case IDs already stored and finishes the rest.

from pathlib import Path
from atlanai import Eval

saved = Path("eval-run-id.txt")
run = Eval(
    "support-agent", dataset="support-agent-cases", task=answer, scores=[exact_match],
    config={"model": "example-model"},
    on_start=lambda event: saved.write_text(event["experiment_id"]),
    resume_experiment_id=saved.read_text().strip() if saved.exists() else None,
)
saved.unlink(missing_ok=True)

import { existsSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { Eval } from "@atlanai/sdk";

const saved = "eval-run-id.txt";
const run = await Eval("support-agent", { dataset: "support-agent-cases", task: answer, scores: [exactMatch] }, {
  config: { model: "example-model" },
  onStart: ({ experimentId }) => writeFileSync(saved, experimentId),
  resumeExperimentId: existsSync(saved) ? readFileSync(saved, "utf8").trim() : undefined,
});
rmSync(saved, { force: true });

Resume requires the experiment to still be running, in the same workspace, with the same dataset or inline cases, and the same config and context manifest. case_id on a record-backed run is the dataset record’s ID. Run one writer per experiment.

If your task has side effects, such as sending email or writing records, make it idempotent before you resume: cases without a stored result run again.

To close a run you will not resume, mark it failed with the ID from the error message. An experiment left running stays in the list and never becomes a baseline. A failed experiment is frozen like a completed one.

client.experiments.update("experiment_01example", {"experiment_status": "failed"})

import { rawEval } from "@atlanai/sdk";

await client.experiments.update("experiment_01example", { experimentStatus: rawEval.EvalExperimentStatus.Failed });

client.experiments.list(experiment_status="running") lists the open runs. Close only runs you own: another job’s run may still be in progress.

Troubleshooting

Run failures include an Eval call inside an event loop, a run left running after a trace timed out or a trace read failed, and a 409 on a duplicate experiment name. See Eval errors.

Next steps

Gate pull requests with evals: Run the eval on every pull request and compare it with the default branch.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Before you begin

Troubleshooting

Next steps

See also