Evaluate an agent

The Eval API is the durable record of an evaluation, not a hosted runner. Your harness or CI executes each case and produces traces; the SDK stores the dataset, experiment, immutable per-case result, scorer definition, and final summary.

dataset → records → experiment → external runner and traces → results → summary → completed

Start from an existing dataset

SDK 0.2.1 adds one shared start path for live evaluation runners. Pass a dataset artifact ID or its exact name. The SDK resolves one dataset, pins its current Registry version, creates a running experiment, and returns the experiment ID before the runner starts.

Pin the context that changes behavior

A context manifest is the fingerprint of the agent setup used for a run. It answers a specific question: which version of every behavior-shaping input was active when this result was produced? The manifest contains references and hashes, not the underlying content. The SDK sorts its entries, writes the canonical manifest to the experiment config, and computes one manifest digest. Python and TypeScript produce the same digest for the same entries, regardless of input order. Each context item has the following shape:

Field	Required	Example	Rule
`kind`	Yes	`skill`	A lowercase category such as `file`, `prompt`, `skill`, `tool_schema`, `knowledge`, `policy`, `memory`, or `harness`.
`name`	Yes	`support-response`	A stable name for this independently versioned input. The pair of `kind` and `name` must be unique in one manifest.
`version`	Yes	`git:7d9f2c1`	The pinned revision used by the run. A Git SHA, release version, or snapshot ID works. `latest` is rejected.
`digest`	Yes	`sha256:…`	SHA-256 of the exact content or canonical bundle represented by the item.
`artifact_id`	No	`skill_01example`	The Registry artifact ID when the input is registered.
`version_ordinal`	No	`4`	The exact Registry artifact version, starting at 1. Use it with `artifact_id` when available.

TypeScript accepts artifactId and versionOrdinal, then serializes the same snake-case manifest as Python. Use one item for each input that can change independently:

Agent input	Suggested `kind`	What to hash
Root agent instructions such as `AGENTS.md` or `CLAUDE.md`	`file`	Exact file bytes
System or task-routing prompt	`prompt`	Rendered prompt template before task data is inserted
Installed skill	`skill`	Canonical skill bundle or Registry artifact version
Tool or MCP contract	`tool_schema`	Canonical tool names, descriptions, and input schemas
Attached knowledge or retrieval snapshot	`knowledge`	Frozen document set or index snapshot manifest
Guardrail or operating policy	`policy`	Exact policy content
Shared memory loaded before every case	`memory`	Frozen memory snapshot
Agent wrapper or harness configuration	`harness`	Canonical behavior-affecting harness configuration

Keep run data on its native Eval surface:

Data	Store it in
Task input, expected output, category, and source	Dataset record
Model, temperature, thinking effort, and baseline experiment	Experiment config
Actual output, error, duration, trace ID, and session ID	Experiment result
Scorer definition and output contract	Versioned scorer
Score value, explanation, scorer ID, and scorer version	Score span and experiment summary
Model calls, tool calls, token usage, and cost	OTLP trace
Secrets and credentials	Never the manifest, experiment, result, or trace

kind is extensible, but the item structure is fixed in SDK 0.2.1. Use a new lowercase kind for a new artifact category. Do not put arbitrary metadata or raw protected content into the manifest.

Change an item’s version and digest whenever its effective content changes. Keep the manifest unchanged when only the dataset case, run timestamp, or model configuration changes. That separation supports three useful comparisons:

Hold constant	Change	What the comparison measures
Dataset and model config	Context manifest	Prompt, skill, tool, or knowledge impact
Dataset and context manifest	Model config	Model or reasoning-setting impact
Dataset, context manifest, and model config	Nothing material	Run-to-run stability

from atlanai import ContextItem, ContextManifest, start_experiment

context_manifest = ContextManifest([
    ContextItem(
        kind="file",
        name="AGENTS.md",
        version=release_commit,
        digest=agent_instructions_digest,
    ),
    ContextItem(
        kind="skill",
        name="support-response",
        version="4",
        digest=skill_digest,
        artifact_id="skill_01example",
        version_ordinal=4,
    ),
    ContextItem(
        kind="tool_schema",
        name="support-tools",
        version=tool_contract_commit,
        digest=tool_schema_digest,
    ),
])

run = start_experiment(
    client,
    "daily-agent-tasks",  # exact name, or a dataset_... artifact ID
    {"name": "candidate-run", "config": {"model": "candidate-model"}},
    context_manifest=context_manifest,
)

with run.trace():
    output = existing_runner()

print(run.experiment_id)

import {
  createContextManifest,
  startExperiment,
} from "@atlanai/sdk";
import { propagateAttributes } from "@atlanai/sdk/tracing";

const contextManifest = await createContextManifest([{
  kind: "file",
  name: "AGENTS.md",
  version: releaseCommit,
  digest: agentInstructionsDigest,
}, {
  kind: "skill",
  name: "support-response",
  version: "4",
  digest: skillDigest,
  artifactId: "skill_01example",
  versionOrdinal: 4,
}, {
  kind: "tool_schema",
  name: "support-tools",
  version: toolContractCommit,
  digest: toolSchemaDigest,
}]);

const run = await startExperiment(
  client,
  "daily-agent-tasks", // exact name, or a dataset_... artifact ID
  { name: "candidate-run", config: { model: "candidate-model" } },
  { contextManifest },
);

const output = await propagateAttributes(
  run.traceOptions,
  () => existingRunner(),
);

console.log(run.experimentId);

run.experiment is the generated experiment-create response. Python exposes its ID as run.id and run.experiment_id; TypeScript exposes run.id and run.experimentId. A name match is exact and workspace-scoped. Multiple exact matches fail instead of selecting one. The experiment config keeps both context_manifest and context_manifest_digest. The trace scope carries the digest alongside the experiment ID. This gives every result a path back to the exact prompt, skill, tool contract, and knowledge versions that shaped it.

The helper starts the Registry lifecycle. Your harness still executes the dataset, uploads results, summarizes score spans, and marks the experiment completed or failed through client.experiments.

Record the whole evaluation

Start by creating a dataset and the cases it contains. Every Eval create body needs workspace_id, even when the client has a default workspace header.

Python

from atlanai import AtlanClient

workspace_id = "workspace_01example"
client = AtlanClient(
    "https://gateway.example",
    bearer_token="...",
    workspace=workspace_id,
)

dataset = client.datasets.create({
    "workspace_id": workspace_id,
    "name": "support-quality-set",
    "display_name": "Support quality set",
    "extra": {"data_snapshot_ref": "support-sample-2026-09"},
})

record = client.datasets.records.create(dataset.id, {
    "workspace_id": workspace_id,
    "name": "reset-password-case",
    "input": {"question": "How do I reset my password?"},
    "expected": {"must_include": ["reset link", "security guidance"]},
    "source_kind": "manual",
    "categories": ["support", "account"],
})

scorer = client.scorers.create({
    "workspace_id": workspace_id,
    "name": "support-answer-quality",
    "scorer_kind": "code",
    "scope": "result",
    "spec": {"entrypoint": "evaluate_support_answer"},
    "outputs": {"quality": {"type": "numeric", "min": 0, "max": 1}},
})

experiment = client.experiments.create({
    "workspace_id": workspace_id,
    "name": "support-agent-candidate",
    "dataset_id": dataset.id,
    "subject_kind": "agent",
    "subject_id": "agent_01example",
    "config": {"model": "candidate-model", "prompt_version": "v2"},
})

# The external runner executes the case and emits its trace and score spans.
result = client.experiments.results.create(experiment.id, {
    "workspace_id": workspace_id,
    "name": "reset-password-result",
    "dataset_record_id": record.id,
    "input": {"question": "How do I reset my password?"},
    "expected": {"must_include": ["reset link", "security guidance"]},
    "output": {"answer": "Use the reset-password link on the sign-in page."},
    "duration_ms": 410,
    "trace_id": trace_id_from_runner,
})

# Persist the trace-score rollup, then seal the immutable run.
client.experiments.summarize(experiment.id)
client.experiments.update(experiment.id, {"experiment_status": "completed"})

The TypeScript client has the same resource tree, with camelCase body keys and awaited calls: await client.datasets.records.create(datasetId, body) and await client.experiments.results.create(experimentId, body). An experiment’s trace views scope OTel data by the atlan.eval.experiment_id span attribute. run.trace() in Python and propagateAttributes(run.traceOptions, fn) in TypeScript stamp that association and the context-manifest digest on every span created inside the runner scope.

Keep evidence that remains useful

The value of an evaluation is being able to diagnose a regression months later, not only its average score. Preserve these fields as part of each run:

Keep	Why it matters later	Eval surface
Input, expected output, categories, and source reference	Reproduce the case and slice failures by scenario or provenance.	Dataset record
Dataset snapshot reference	Know which frozen source data a benchmark actually used.	Dataset `extra.data_snapshot_ref`
Subject, model/prompt/configuration, and baseline experiment	Compare candidates fairly and explain why a score moved.	Experiment
Actual output, duration, error, trace ID, and session ID	Move from a failed row to the exact execution and its latency or failure.	Experiment result
Scorer kind, scope, definition, and output schema	Keep the interpretation of a score stable as evaluators evolve.	Scorer
Trace-level model calls, tool calls, token usage, cost, and score comments	Diagnose whether a quality shift came from the model, tools, prompt, cost, or the scorer.	Tracing SDK and experiment trace views

Scorers are versioned artifacts. Results are create-only, and a completed or failed experiment is frozen. That combination preserves the run as evidence instead of allowing later dataset edits or result updates to rewrite history.

Read and compare

Use the resource tree to review an experiment’s durable results and the live trace detail behind them:

Python

results = client.experiments.results.list(experiment.id)
traces = client.experiments.traces.list(experiment.id)
one_trace = client.experiments.traces.get(experiment.id, trace_id_from_runner)
spans = client.experiments.traces.list_spans(experiment.id, trace_id_from_runner)

For the complete endpoint list, including search, bulk result upload, archive, and trace statistics, see the generated resource references below.

Datasets

Curate dataset records and preserve their provenance.

Experiments

Record runs, results, traces, and score rollups.

Scorers

Version the score definition alongside the evaluation.

Tracing and scores

Capture the execution evidence that makes an evaluation explainable.

Overview

CLI

SDK

Cookbooks

Operate safely

API reference

Start from an existing dataset

Pin the context that changes behavior

Record the whole evaluation

Keep evidence that remains useful

Read and compare

Datasets

Experiments

Scorers

Tracing and scores

​Start from an existing dataset

​Pin the context that changes behavior

​Record the whole evaluation

​Keep evidence that remains useful

​Read and compare

Datasets

Experiments

Scorers

Tracing and scores

Start from an existing dataset

Pin the context that changes behavior

Record the whole evaluation

Keep evidence that remains useful

Read and compare