Create eval datasets

A dataset keeps the cases every run is measured on in your workspace, so two runs, or a pull request and its baseline, use exactly the same cases. Create one when you move past local iteration to baselines, CI gates, or model comparisons. Inline cases in your eval code remain the quicker option while you iterate.

Before you begin

Before you start, make sure you have:

atlanai SDK: Installed with tracing, as in Run your first eval.
API key and workspace ID: In ATLAN_API_KEY and ATLAN_WORKSPACE_ID. The key must be able to create datasets in the workspace.

Check if you need dataset

Stay with inline cases for local iteration. Move to a dataset when results must be compared across runs.

	Inline `data`	Registry dataset
Where it lives	In your eval code	In the workspace, under Evals → Datasets
Case identity	Your `id`, or `case-<n>`	The record ID, stable across runs
Comparison across runs	Only if you keep the `id`s stable	Built in: results join on `dataset_record_id`
Snapshot at run start	A digest of the cases, used for resume	A full `dataset_snapshot` with every record’s version and hash
Use for	Local iteration, quick checks	Baselines, CI gates, model comparisons

An inline case is a mapping with input (any JSON value) and optional expected, metadata, tags, and id. Set id on every inline case you may want to resume or compare. If inline cases are enough, skip to Write code scorers.

Shape each record

Each record field reaches your task or scorer as follows:

Record field	Becomes	Notes
`input[input_key]`	The value passed to `task`	`input` must be a JSON object. `input_key` defaults to `question`. A record without that key fails the run.
`expected`	`expected` in your scorer	A single-key `{"value": x}` is unwrapped to `x`. Any other shape is passed as stored.
`extra`	`metadata` in your scorer and in task hooks	Case-specific settings, such as a fixture ID or a tolerance.
`categories`	The case’s `tags`	Slice results by scenario: which categories regressed?
Record ID	`case_id` and `dataset_record_id` on the result	The join key across experiments.

If the question is not under question, set input_key when you create the dataset: push_dataset(client, name, records, input_key="prompt"). It cannot change on a later push. Each record shows input[input_key] as display_input, so tables in the app and coding agents know which field is the question. To pass a whole object to your task, nest it under the key: {"question": {"text": "...", "locale": "fr"}}.Record where each case came from with source_kind and source_ref. Set them on the first push; a later push does not change them.

`source_kind`	`source_ref`	Use
`manual` (default)	none	Written by hand
`session`	A session ID in the same workspace, checked to exist	Curated from a production session. See Turn production failures into test cases.
`trace`	A trace ID	Curated from one trace
`external`	Any reference up to 255 characters	Imported from another tool or a ticket

Use label for curation state, such as needs_review, and categories for the scenario.

Push cases to dataset

push_dataset (pushDataset in TypeScript) creates the dataset on the first call and syncs its records on every later call. An unchanged record is left alone, so pushing the same suite on every CI run creates no empty versions. A run’s snapshot holds at most 10,000 records, so split a larger suite into several datasets.

Python
TypeScript

import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "support-agent-cases", [
    {
        "name": "refund-status",                       # your stable key for the case
        "input": {"question": "Where is my refund for order 1042?"},
        "expected": {"value": "processed"},
        "categories": ["refunds"],
        "extra": {"priority": "high"},
    },
    {
        "name": "shipping-eta",
        "input": {"question": "When will order 2210 arrive?"},
        "expected": {"value": "Friday"},
        "categories": ["shipping"],
    },
])
print(suite.id, [r.action for r in suite.records])

import { AtlanClient, pushDataset } from "@atlanai/sdk";

const client = new AtlanClient({
  gatewayOrigin: process.env.ATLAN_BASE_URL ?? "https://api.atlan.com",
  bearerToken: process.env.ATLAN_API_KEY!,
  workspace: process.env.ATLAN_WORKSPACE_ID!,
});

const suite = await pushDataset(client, "support-agent-cases", [
  {
    name: "refund-status",
    input: { question: "Where is my refund for order 1042?" },
    expected: { value: "processed" },
    categories: ["refunds"],
    extra: { priority: "high" },
  },
  {
    name: "shipping-eta",
    input: { question: "When will order 2210 arrive?" },
    expected: { value: "Friday" },
    categories: ["shipping"],
  },
]);
console.log(suite.id);

The first push prints ['created', 'created']. Later pushes report unchanged or updated per record.Dataset names are unique in the workspace, and pushing to an existing name adds to that dataset. If the name might be taken, check suite.created and each record’s action.

A push never deletes records. A case you remove from your code stays in the dataset and keeps running. Archive it in the app, or with client.datasets.records.archive(dataset_id, record_id).

Run eval on dataset

Pass the dataset ID, or its exact name, to Eval. A missing or ambiguous name fails rather than picking a close match.

from atlanai import Eval

run = Eval("support-agent", dataset=suite.id, task=answer, scores=[accuracy])
names = {r.id: r.key for r in suite.records}

Results carry only the record ID as case_id. Keep the names map from suite.records to print readable failures.When the run starts, it records a snapshot of every live record’s version, and runs from that snapshot. Editing a record’s input, expected outcome, label, categories, or extra adds a new version, so a correction changes only later runs; an edit to only its display name or description lands on the current version. Earlier versions stay readable at GET /eval/v1/datasets/{dataset_id}/records/{record_id}/versions/{n}. See Eval object fields.

Verify dataset records

Check in the app: Open Evals, then Datasets, and select the dataset. It lists one record per case you pushed.
Check in code: List records with client.datasets.records.list. It accepts q (a substring over input, expected, label, and categories), label, category, source_kind, and source_ref.

If a run fails with a ValueError naming input_key, a record is missing the question key. See Eval errors.

Create CSV-backed dataset (optional)

A dataset can instead be backed by an uploaded CSV file, for example one created in the app. The CSV needs a unique, non-empty case_id column and a column named after the dataset’s input_key. Files are limited to 32 MiB, 100,000 rows, and 256 columns. Each upload is a new dataset version.Reserve an upload, send the bytes, then create the dataset with the upload key:

# 1. Reserve an upload; the response carries upload_key.
curl -sS --fail-with-body -X POST "${ATLAN_BASE_URL}/file/v1/files/uploads" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/json" \
  -d "{\"size_bytes\": $(wc -c < cases.csv)}"

# 2. Send the file.
curl -sS --fail-with-body -X PUT "${ATLAN_BASE_URL}/file/v1/files/uploads/${UPLOAD_KEY}" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/octet-stream" \
  --data-binary @cases.csv

# 3. Create the dataset. Use POST /eval/v1/datasets/{id}/versions to add a version.
curl -sS --fail-with-body -X POST "${ATLAN_BASE_URL}/eval/v1/datasets" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/json" \
  -H "X-Atlan-Workspace-Id: ${ATLAN_WORKSPACE_ID}" \
  -d "{\"workspace_id\": \"${ATLAN_WORKSPACE_ID}\", \"name\": \"csv-cases\", \"upload_key\": \"${UPLOAD_KEY}\", \"filename\": \"cases.csv\"}"

Read a version’s content back with GET /eval/v1/datasets/{id}/versions/{n}/content.The SDK works with a CSV-backed dataset directly, so you do not need your own runner to run one:

Run it: Pass it to Eval as you would a record-backed dataset. Eval reads the pinned version the experiment’s snapshot points at, each row becomes a case, and each result names its row by case_id instead of dataset_record_id.
Update it: Push a suite with push_dataset (Python) or pushDataset (TypeScript). The SDK appends a new CSV version with the changed and new cases and leaves the other rows as they are, so a push where every case comes back unchanged writes no version. If the dataset changed since the SDK read it, the push fails; reload it and push again.
Read its content: client.datasets.versions.get_content(dataset_id, version) in Python, or client.datasets.versions.getContent(datasetId, version) in TypeScript, returns the CSV text of one version.

Rows the SDK writes keep typed values, such as a nested input object, a number, or a boolean; cells in a CSV you upload yourself stay text. To record a CSV dataset’s runs from your own harness instead, see Record runs from your own harness.

Troubleshooting

Dataset failures include a dataset with no cases, a CSV file that fails Atlan’s checks, a record without the input_key, and a transient record_push_failed. See Eval errors.

Next steps

Write code scorers: Score each case’s output against its expected value.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Before you begin

Troubleshooting

Next steps

See also