Concepts

Every eval object answers one question. Knowing which objects are versioned, create-only, or frozen tells you what you can rely on when you compare runs months apart.

dataset ──has──► dataset records (versioned)
   │
   └─ pinned by ─► experiment ──has──► experiment results (create-only)
                      │                     │
                      │                     └─ trace_id ─► trace ──has──► score spans
                      │                                                    │
                      └─ summary.scores ◄── folded from results            └─ cite ─► scorer version

The objects

Object	What it is	Mutability
Dataset	A named collection of cases in a workspace.	Editable. Its records are versioned.
Dataset record	One case: `input` (a JSON object), optional `expected`, `categories`, a `label`, `extra` metadata, and provenance (`source_kind`, `source_ref`).	Editing a record appends a new immutable version.
Experiment	One run of a task over a dataset or inline cases. It holds the run `config`, an optional `baseline_experiment_id`, an optional subject (the agent, harness, or skill under test), a status, and a `summary`.	`running` accepts results. Once `completed` or `failed` it is frozen, and any change is refused with `409`.
Experiment result	One case’s outcome: `case_id`, `input`, `expected`, `output`, `scores`, `error`, `duration_ms`, `trace_id`, and the `dataset_record_id` it ran.	Create-only. It can never be edited or deleted.
Scorer	The definition of what a score means. `code` scorers are functions your runner executes; `llm_judge` scorers are hosted judge definitions Atlan executes. `human` scorers catalog a manual review.	Every edit through the eval API appends a version. Scores cite the exact version.
Trace	The spans your task produced for one case: model calls, tool calls, tokens, cost, latency, errors.	Written by the tracing SDK and kept for your trace retention period.
Score span	A child span named `score.<name>` carrying the score value and the scorer ID and version that produced it.	Written once, with the trace.
Session insight	One hosted judge’s answer about one production session or trace.	Re-scoring the same session with the same scorer rewrites it.

Snapshots make runs comparable

When an experiment starts from a dataset, Atlan stamps a dataset_snapshot onto it: the dataset version plus each live record’s ID, version, content hash, input, and expected value. The run executes from that snapshot. Editing a record later changes only future experiments, so two runs on the same dataset compare the same cases unless you deliberately changed them. Results join across experiments on dataset_record_id. That is the key for “which cases got better or worse”.

Where the summary comes from

summary.scores is computed by Atlan, not by your runner. When the experiment moves from running to completed or failed, Atlan reads every stored result and replaces summary.scores with {name: {mean, count}} for each score name. A null score, meaning “not applicable”, is left out of both the mean and the count. Other summary fields your runner sends, such as case counts and duration, are kept as sent. Use the stored summary for release decisions and run history. Do not average a page of results yourself: a partial page gives a partial mean.

Offline and online scores are stored differently

	Offline (`Eval`)	Online (hosted judge)
Score value	On the result row (`scores`) and as a score span on the case trace	On a session insight
Scorer kind	`code`, run in your process	`llm_judge`, run by Atlan
Aggregate	`summary.scores` on the experiment	Per-answer rows; compare them in the Atlan app

A score span you write from application code, for example a user’s thumbs-up, is a third path. See scores from your application.

What Atlan does and does not do

Atlan stores and computes. It stores datasets, snapshots, results, scorer versions, and traces. It computes summaries. It executes hosted llm_judge scorers over recorded sessions.
Atlan does not run your agent. Offline evals execute in your process or CI job. There is no hosted runner, and no Atlan service calls your model or tools.
There is no compare endpoint. A baseline is a stored pointer. You compare two experiments by reading their summaries and joining their results on dataset_record_id. The CI action and the compare cookbook both do this for you.
An experiment does not pin an agent version. Record the version you tested in the experiment config, or pin each input with a context manifest.

Names you will see in code

Term	Meaning
`task`	The function under test. It receives a case’s `input` (and optionally hooks) and returns the output.
`case_id`	A case’s stable ID within one experiment. Dataset-backed runs use the record ID; inline cases use your `id`, or `case-<n>`.
`subject_kind`, `subject_id`	The agent, harness, or skill the experiment evaluates. When set, the run appears on that agent’s profile in the Atlan app.
`baseline_experiment_id`	The experiment this run should be compared with.
`config`	Free-form run settings: model, prompt version, git SHA. It is pinned at creation and filterable with `?config=key:value`.

Build an eval

Run and gate

Score production

Reference

The objects

Snapshots make runs comparable

Where the summary comes from

Offline and online scores are stored differently

What Atlan does and does not do

Names you will see in code

​The objects

​Snapshots make runs comparable

​Where the summary comes from

​Offline and online scores are stored differently

​What Atlan does and does not do

​Names you will see in code

The objects

Snapshots make runs comparable

Where the summary comes from

Offline and online scores are stored differently

What Atlan does and does not do

Names you will see in code