> ## Documentation Index
> Fetch the complete documentation index at: https://platform.atlan.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate an agent

> Create datasets, record evaluation runs, preserve their evidence, and compare outcomes over time.

The Eval API is the durable record of an evaluation, not a hosted runner. Your
harness or CI executes each case and produces traces; the SDK stores the
dataset, experiment, immutable per-case result, scorer definition, and final
summary.

```text theme={null}
dataset → records → experiment → external runner and traces → results → summary → completed
```

## Start from an existing dataset

SDK `0.2.1` adds one shared start path for live evaluation runners. Pass a
dataset artifact ID or its exact name. The SDK resolves one dataset, pins its
current Registry version, creates a `running` experiment, and returns the
experiment ID before the runner starts.

### Pin the context that changes behavior

A context manifest is the fingerprint of the agent setup used for a run. It
answers a specific question: which version of every behavior-shaping input was
active when this result was produced?

The manifest contains references and hashes, not the underlying content. The
SDK sorts its entries, writes the canonical manifest to the experiment config,
and computes one manifest digest. Python and TypeScript produce the same digest
for the same entries, regardless of input order.

Each context item has the following shape:

| Field             | Required | Example            | Rule                                                                                                                  |
| ----------------- | -------- | ------------------ | --------------------------------------------------------------------------------------------------------------------- |
| `kind`            | Yes      | `skill`            | A lowercase category such as `file`, `prompt`, `skill`, `tool_schema`, `knowledge`, `policy`, `memory`, or `harness`. |
| `name`            | Yes      | `support-response` | A stable name for this independently versioned input. The pair of `kind` and `name` must be unique in one manifest.   |
| `version`         | Yes      | `git:7d9f2c1`      | The pinned revision used by the run. A Git SHA, release version, or snapshot ID works. `latest` is rejected.          |
| `digest`          | Yes      | `sha256:…`         | SHA-256 of the exact content or canonical bundle represented by the item.                                             |
| `artifact_id`     | No       | `skill_01example`  | The Registry artifact ID when the input is registered.                                                                |
| `version_ordinal` | No       | `4`                | The exact Registry artifact version, starting at 1. Use it with `artifact_id` when available.                         |

TypeScript accepts `artifactId` and `versionOrdinal`, then serializes the same
snake-case manifest as Python.

Use one item for each input that can change independently:

| Agent input                                                | Suggested `kind` | What to hash                                          |
| ---------------------------------------------------------- | ---------------- | ----------------------------------------------------- |
| Root agent instructions such as `AGENTS.md` or `CLAUDE.md` | `file`           | Exact file bytes                                      |
| System or task-routing prompt                              | `prompt`         | Rendered prompt template before task data is inserted |
| Installed skill                                            | `skill`          | Canonical skill bundle or Registry artifact version   |
| Tool or MCP contract                                       | `tool_schema`    | Canonical tool names, descriptions, and input schemas |
| Attached knowledge or retrieval snapshot                   | `knowledge`      | Frozen document set or index snapshot manifest        |
| Guardrail or operating policy                              | `policy`         | Exact policy content                                  |
| Shared memory loaded before every case                     | `memory`         | Frozen memory snapshot                                |
| Agent wrapper or harness configuration                     | `harness`        | Canonical behavior-affecting harness configuration    |

Keep run data on its native Eval surface:

| Data                                                         | Store it in                                      |
| ------------------------------------------------------------ | ------------------------------------------------ |
| Task input, expected output, category, and source            | Dataset record                                   |
| Model, temperature, thinking effort, and baseline experiment | Experiment config                                |
| Actual output, error, duration, trace ID, and session ID     | Experiment result                                |
| Scorer definition and output contract                        | Versioned scorer                                 |
| Score value, explanation, scorer ID, and scorer version      | Score span and experiment summary                |
| Model calls, tool calls, token usage, and cost               | OTLP trace                                       |
| Secrets and credentials                                      | Never the manifest, experiment, result, or trace |

<Note>
  `kind` is extensible, but the item structure is fixed in SDK `0.2.1`.
  Use a new lowercase `kind` for a new artifact category. Do not put arbitrary
  metadata or raw protected content into the manifest.
</Note>

Change an item's `version` and `digest` whenever its effective content changes.
Keep the manifest unchanged when only the dataset case, run timestamp, or model
configuration changes. That separation supports three useful comparisons:

| Hold constant                               | Change           | What the comparison measures             |
| ------------------------------------------- | ---------------- | ---------------------------------------- |
| Dataset and model config                    | Context manifest | Prompt, skill, tool, or knowledge impact |
| Dataset and context manifest                | Model config     | Model or reasoning-setting impact        |
| Dataset, context manifest, and model config | Nothing material | Run-to-run stability                     |

<CodeGroup>
  ```python Python theme={null}
  from atlanai import ContextItem, ContextManifest, start_experiment

  context_manifest = ContextManifest([
      ContextItem(
          kind="file",
          name="AGENTS.md",
          version=release_commit,
          digest=agent_instructions_digest,
      ),
      ContextItem(
          kind="skill",
          name="support-response",
          version="4",
          digest=skill_digest,
          artifact_id="skill_01example",
          version_ordinal=4,
      ),
      ContextItem(
          kind="tool_schema",
          name="support-tools",
          version=tool_contract_commit,
          digest=tool_schema_digest,
      ),
  ])

  run = start_experiment(
      client,
      "daily-agent-tasks",  # exact name, or a dataset_... artifact ID
      {"name": "candidate-run", "config": {"model": "candidate-model"}},
      context_manifest=context_manifest,
  )

  with run.trace():
      output = existing_runner()

  print(run.experiment_id)
  ```

  ```typescript TypeScript theme={null}
  import {
    createContextManifest,
    startExperiment,
  } from "@atlanai/sdk";
  import { propagateAttributes } from "@atlanai/sdk/tracing";

  const contextManifest = await createContextManifest([{
    kind: "file",
    name: "AGENTS.md",
    version: releaseCommit,
    digest: agentInstructionsDigest,
  }, {
    kind: "skill",
    name: "support-response",
    version: "4",
    digest: skillDigest,
    artifactId: "skill_01example",
    versionOrdinal: 4,
  }, {
    kind: "tool_schema",
    name: "support-tools",
    version: toolContractCommit,
    digest: toolSchemaDigest,
  }]);

  const run = await startExperiment(
    client,
    "daily-agent-tasks", // exact name, or a dataset_... artifact ID
    { name: "candidate-run", config: { model: "candidate-model" } },
    { contextManifest },
  );

  const output = await propagateAttributes(
    run.traceOptions,
    () => existingRunner(),
  );

  console.log(run.experimentId);
  ```
</CodeGroup>

`run.experiment` is the generated experiment-create response. Python exposes
its ID as `run.id` and `run.experiment_id`; TypeScript exposes `run.id` and
`run.experimentId`. A name match is exact and workspace-scoped. Multiple exact
matches fail instead of selecting one.

The experiment config keeps both `context_manifest` and
`context_manifest_digest`. The trace scope carries the digest alongside the
experiment ID. This gives every result a path back to the exact prompt, skill,
tool contract, and knowledge versions that shaped it.

<Note>
  The helper starts the Registry lifecycle. Your harness still executes the
  dataset, uploads results, summarizes score spans, and marks the experiment
  `completed` or `failed` through `client.experiments`.
</Note>

## Record the whole evaluation

Start by creating a dataset and the cases it contains. Every Eval create body
needs `workspace_id`, even when the client has a default workspace header.

```python Python theme={null}
from atlanai import AtlanClient

workspace_id = "workspace_01example"
client = AtlanClient(
    "https://gateway.example",
    bearer_token="...",
    workspace=workspace_id,
)

dataset = client.datasets.create({
    "workspace_id": workspace_id,
    "name": "support-quality-set",
    "display_name": "Support quality set",
    "extra": {"data_snapshot_ref": "support-sample-2026-09"},
})

record = client.datasets.records.create(dataset.id, {
    "workspace_id": workspace_id,
    "name": "reset-password-case",
    "input": {"question": "How do I reset my password?"},
    "expected": {"must_include": ["reset link", "security guidance"]},
    "source_kind": "manual",
    "categories": ["support", "account"],
})

scorer = client.scorers.create({
    "workspace_id": workspace_id,
    "name": "support-answer-quality",
    "scorer_kind": "code",
    "scope": "result",
    "spec": {"entrypoint": "evaluate_support_answer"},
    "outputs": {"quality": {"type": "numeric", "min": 0, "max": 1}},
})

experiment = client.experiments.create({
    "workspace_id": workspace_id,
    "name": "support-agent-candidate",
    "dataset_id": dataset.id,
    "subject_kind": "agent",
    "subject_id": "agent_01example",
    "config": {"model": "candidate-model", "prompt_version": "v2"},
})

# The external runner executes the case and emits its trace and score spans.
result = client.experiments.results.create(experiment.id, {
    "workspace_id": workspace_id,
    "name": "reset-password-result",
    "dataset_record_id": record.id,
    "input": {"question": "How do I reset my password?"},
    "expected": {"must_include": ["reset link", "security guidance"]},
    "output": {"answer": "Use the reset-password link on the sign-in page."},
    "duration_ms": 410,
    "trace_id": trace_id_from_runner,
})

# Persist the trace-score rollup, then seal the immutable run.
client.experiments.summarize(experiment.id)
client.experiments.update(experiment.id, {"experiment_status": "completed"})
```

The TypeScript client has the same resource tree, with `camelCase` body keys
and awaited calls: `await client.datasets.records.create(datasetId, body)` and
`await client.experiments.results.create(experimentId, body)`.

An experiment's trace views scope OTel data by the
`atlan.eval.experiment_id` span attribute. `run.trace()` in Python and
`propagateAttributes(run.traceOptions, fn)` in TypeScript stamp that association
and the context-manifest digest on every span created inside the runner scope.

## Keep evidence that remains useful

The value of an evaluation is being able to diagnose a regression months later,
not only its average score. Preserve these fields as part of each run:

| Keep                                                                       | Why it matters later                                                                      | Eval surface                           |
| -------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------- |
| Input, expected output, categories, and source reference                   | Reproduce the case and slice failures by scenario or provenance.                          | Dataset record                         |
| Dataset snapshot reference                                                 | Know which frozen source data a benchmark actually used.                                  | Dataset `extra.data_snapshot_ref`      |
| Subject, model/prompt/configuration, and baseline experiment               | Compare candidates fairly and explain why a score moved.                                  | Experiment                             |
| Actual output, duration, error, trace ID, and session ID                   | Move from a failed row to the exact execution and its latency or failure.                 | Experiment result                      |
| Scorer kind, scope, definition, and output schema                          | Keep the interpretation of a score stable as evaluators evolve.                           | Scorer                                 |
| Trace-level model calls, tool calls, token usage, cost, and score comments | Diagnose whether a quality shift came from the model, tools, prompt, cost, or the scorer. | Tracing SDK and experiment trace views |

Scorers are versioned artifacts. Results are create-only, and a completed or
failed experiment is frozen. That combination preserves the run as evidence
instead of allowing later dataset edits or result updates to rewrite history.

## Read and compare

Use the resource tree to review an experiment's durable results and the live
trace detail behind them:

```python Python theme={null}
results = client.experiments.results.list(experiment.id)
traces = client.experiments.traces.list(experiment.id)
one_trace = client.experiments.traces.get(experiment.id, trace_id_from_runner)
spans = client.experiments.traces.list_spans(experiment.id, trace_id_from_runner)
```

For the complete endpoint list, including search, bulk result upload, archive,
and trace statistics, see the generated resource references below.

<CardGroup cols={2}>
  <Card title="Datasets" icon="list" href="/sdk/reference/datasets">
    Curate dataset records and preserve their provenance.
  </Card>

  <Card title="Experiments" icon="flask" href="/sdk/reference/experiments">
    Record runs, results, traces, and score rollups.
  </Card>

  <Card title="Scorers" icon="star" href="/sdk/reference/scorers">
    Version the score definition alongside the evaluation.
  </Card>

  <Card title="Tracing and scores" icon="diagram-project" href="/sdk/tracing">
    Capture the execution evidence that makes an evaluation explainable.
  </Card>
</CardGroup>