Eval object fields

This page lists each object an eval stores, in the order a run creates them, with its fields and what can change after it is created. For Eval options, return values, statuses, and limits, see Eval options and errors. Every object also has a name, an optional display_name and description, and the workspace_id it belongs to.

Object summary

Object	Created by	Holds	Can change after create
Dataset	You, with `push_dataset` or the API	A named set of cases	Name and description. Records are versioned.
Dataset record	You, in a dataset	One case	Editing adds a new version. Earlier versions stay.
Experiment	`Eval` or your runner	One run over a dataset or inline cases	Until it is `completed` or `failed`. Then frozen, except display name, description, and archiving. Its `name` is fixed at creation; a patch that changes it is refused with `422`.
Experiment result	`Eval` or your runner	One case’s outcome	Nothing. Create-only.
Scorer	`Eval`, or you for a hosted judge	What a score means	Every edit adds a version.
Score span	The tracing SDK	One score on a case’s trace	Nothing. Written once.
Session insight	A hosted judge	One answer about one session	Nothing. Scoring again adds a new answer.
Issue	Eval, or you, with `POST /eval/v1/issues`	One recurring problem across an agent’s scored runs	`display_name`, `description`, `reasoning`, and `severity`, with `PATCH /eval/v1/issues/{id}`; Eval can also raise `severity` during an issue scan; `issue_status`, `occurrence_count`, `session_count`, `last_seen_at`, and `last_seen_version` are Eval’s.
Issue trace	Eval	One trace counted toward an issue	Nothing. Write-once.

Dataset

Field	Meaning
`name`	Unique within the workspace. Pushing to an existing name adds to that dataset.
`input_key`	The key in each record’s `input` that holds the case prompt. Default `question`.
`extra`	Free-form metadata for the dataset.

Dataset record

Field	Meaning
`dataset_id`	The dataset the record belongs to. Cannot change.
`input`	The case input, as a JSON object. Every record has one.
`expected`	The expected outcome, when the case has one.
`categories`	Free-text categories for grouping cases.
`label`	A curation label for the case.
`source_kind`	Where the case came from: `manual`, `session`, `trace`, or `external`.
`source_ref`	The session or trace the case came from, when `source_kind` is `session` or `trace`.
`extra`	Free-form metadata, returned as stored.

Experiment

Field	Meaning
`experiment_status`	`running`, then `completed` or `failed`. A finished experiment refuses new results and status or summary changes with `409`.
`dataset_id`	The dataset the run used. Empty for inline cases. Cannot change.
`dataset_snapshot`	The dataset version, plus each record’s version, input, and expected value, as the run used them. Later record edits do not change it.
`config`	What ran: model, prompt version, commit. Fixed at create. Record your agent’s version here; the experiment does not record it otherwise.
`baseline_experiment_id`	The experiment this run is compared with. The CI action and the Compare two agent versions do the comparison.
`subject_kind`, `subject_id`	The `agent`, `harness`, or `skill` under test. When set, the run appears on that object’s page in the app.
`summary`	Per score name, `{mean, count}` over the stored results, written when the experiment finishes. A `null` score is left out of both. The counts `result_count`, `scored_results`, and `unscored_results` say how many of the run’s results are scored: a result is scored when any of its scores is a finite number, zero included. Empty runs report three zero counts, and a run finalized before the counts existed, or imported already final, may omit them. See Return value.

Experiment result

Field	Meaning
`case_id`	The case’s ID within the experiment: the record ID for dataset runs, your `id` for inline cases, or `case-<n>`.
`dataset_record_id`	The record the case ran. Join two experiments’ results on it to compare them case by case.
`input`, `expected`	The case as it ran.
`output`	What the task returned.
`scores`	The case’s scores. Kept after the trace’s retention period ends.
`error`	The failure, when the case errored instead of scoring.
`duration_ms`	How long the case took, in milliseconds.
`trace_id`	The trace of the case’s run.
`session_id`	The session the case ran as, when there is one.

Scorer

Field	Meaning
`scorer_kind`	`code`: a function your process runs. `llm_judge`: a hosted judge that Atlan Trident runs over recorded sessions. `human`: a manual review.
`output_type`	For a hosted judge: `classification`, a `pass` or `fail` verdict, or `score`, a value from `0` to `1`.
`messages`, `choices`	A hosted judge in rubric form: the prompt as a list of messages, and the answers the judge can pick from.
`questions`, `state`	A hosted judge in question-set form: one question, and the parts of the session the judge sees.

Each score records the scorer version that produced it, so a score stays traceable after the scorer changes.

Score span

A span named score.<name> on the case’s trace. It carries the score value and the ID and version of the scorer that produced it. Scores your application attaches to production traces use the same span; see Add scores from your application.

Session insight

One hosted judge’s answer about one production session or trace, linked to the session and the scorer version. Scoring the same session again with the same scorer adds a new answer. Earlier answers stay, and the app shows the most recent.

Issue

One recurring problem across an agent’s scored runs. Eval finds issues from the agent’s scorers’ answers — POST /eval/v1/issues/find queues a scan that reads the agent’s failing verdicts and files or updates its issues — or you can file one with POST /eval/v1/issues; either way, issues are filed on the agent: a problem in one of its skills is the agent’s issue, and the fix may be an improvement to that skill.

Field	Meaning
`subject_id`	The agent the issue is filed on.
`reasoning`	Why it happens, tied to the evidence: what the linked answers and their runs show, and which parts are inferred. At most 4,096 characters.
`severity`	How badly the person is affected when it happens: `low` (cosmetic or a minor inconvenience), `medium` (default; a partial or incomplete answer the person can work around), or `high` (no answer, or a wrong one the person may act on).
`severity_changes`	The last 20 severity raises Eval applied, oldest first: each holds the severity before and after, the scan that raised it, and a reason of at most 200 characters. Absent until a scan raises the severity.
`issue_status`	Where it stands: `to_do`, `resolved`, or `regression`. Eval sets it by rule, never from a model’s judgement. It is a different field from the artifact’s own `status`.
`occurrence_count`	How many traces show it: one row per trace per issue in `issue_traces`, each counted once.
`session_count`	How many sessions show it as a whole, apart from its runs: a session is evidence as a session, never as one of its runs.
`last_seen_at`	When the newest of those traces started.
`last_seen_version`	The newest agent version it appeared on: the highest version among its linked answers; absent while none recorded one.
`scorer_ids`	The distinct scorers whose answers its linked traces cite, sorted. Only the issue’s own read (`GET /eval/v1/issues/{id}`) returns them, computed from at most the 500 newest links’ readable answers; empty when they cannot be read.
`fix`	The accepted fix: the `improvement_id` of the newest accepted improvement and the agent version it runs from (`fix_from`). Only the issue’s own read returns it; `null` when there is no accepted fix or it cannot be read, and `fix_from` is `null` when that version is unknown or the fix is a skill improvement that is not pinned yet.

The name is the dedupe key, derived from the agent and a slug of the problem, and is unique within the workspace, so a rerun cannot file the same problem twice: filing the same slug on the same agent again is refused as taken, and the issue that holds it is the one to link evidence to. A wrong display_name, description, reasoning, or severity is edited with PATCH /eval/v1/issues/{id}; the name, the subject, the status, and the counts are Eval’s, and a patch that names one is refused. Eval can also raise a severity on its own: when an issue scan links at least three traces to an existing issue that it had not linked before and proposes a higher severity, Eval applies the raise and records it in severity_changes, with the reason. Eval never lowers a severity on its own, and an archived issue’s severity does not change. A fix is an improvement row that carries the issue’s issue_id. An issue holds one open fix per subject: while one is running or awaiting a decision, proposing another on that subject is refused with 409, and once it is decided or has failed, another can be proposed. Fixes for other issues, or for the issue on a skill linked to the agent, are not blocked. The status follows the agent version its newest accepted fix runs from and what scored traces show on fixed versions: with no running fix, or with nothing scored on a fixed version yet, the issue stays to_do; an occurrence on a fixed version makes it regression; and it is resolved once the issue’s own scorers have judged traces on fixed versions and none failed. An agent fix runs from the agent version accepting it minted, and a skill fix once the agent pins that skill version or later; evidence that records no agent version is left out.

Issue trace

One write-once link between an issue and the trace that shows it: the issue_id, the trace_id, the answer_id of the scorer answer that first showed it (a trace-level answer; the sessions that show the issue as a whole are read separately, with GET /eval/v1/issues/{id}/sessions), and observed_at, when the traced run started. Issues and traces are many-to-many. One trace per issue is recorded once even if several answers and scorers describe it, and a trace may still show several issues. The issue’s traces read (GET /eval/v1/issues/{id}/traces) also names the session and question of the answer each row cites: its session_id and question_id, either null when the answer records none.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost