Datasets

A dataset holds the cases every run is measured on. Inline cases are fine while you iterate. Move to a dataset when you need two runs, or a PR and its baseline, to use exactly the same cases.

Inline cases or a dataset

	Inline `data`	Registry dataset
Where it lives	In your eval code	In the workspace, visible under Evals → Datasets
Case identity	Your `id`, or `case-<n>`	The record ID, stable across runs
Comparison across runs	Only if you keep the `id`s stable	Built in: results join on `dataset_record_id`
Snapshot at run start	A digest of the cases, used for resume	A full `dataset_snapshot` with every record’s version and hash
Use for	Local iteration, quick checks	Baselines, CI gates, model comparisons

An inline case is a mapping with input (any JSON value) and optional expected, metadata, tags, and id. Set id on every inline case you may want to resume or compare.

Push a dataset from code

push_dataset in Python and pushDataset in TypeScript create the dataset once and sync its records on every call. A record whose content has not changed is left alone, so pushing the same suite on every CI run creates no empty versions.

import os
from atlanai import AtlanClient, push_dataset

client = AtlanClient(
    os.environ.get("ATLAN_BASE_URL", "https://api.atlan.com"),
    bearer_token=os.environ["ATLAN_API_KEY"],
    workspace=os.environ["ATLAN_WORKSPACE_ID"],
)

suite = push_dataset(client, "support-agent-cases", [
    {
        "name": "refund-status",                       # your stable key for the case
        "input": {"question": "Where is my refund for order 1042?"},
        "expected": {"value": "processed"},
        "categories": ["refunds"],
        "extra": {"priority": "high"},
    },
    {
        "name": "shipping-eta",
        "input": {"question": "When will order 2210 arrive?"},
        "expected": {"value": "Friday"},
        "categories": ["shipping"],
    },
])
print(suite.id, [r.action for r in suite.records])
# First push: ['created', 'created']. Later pushes: 'unchanged' or 'updated' per record.

import { AtlanClient, pushDataset } from "@atlanai/sdk";

const client = new AtlanClient({
  gatewayOrigin: process.env.ATLAN_BASE_URL ?? "https://api.atlan.com",
  bearerToken: process.env.ATLAN_API_KEY!,
  workspace: process.env.ATLAN_WORKSPACE_ID!,
});

const suite = await pushDataset(client, "support-agent-cases", [
  {
    name: "refund-status",
    input: { question: "Where is my refund for order 1042?" },
    expected: { value: "processed" },
    categories: ["refunds"],
    extra: { priority: "high" },
  },
  {
    name: "shipping-eta",
    input: { question: "When will order 2210 arrive?" },
    expected: { value: "Friday" },
    categories: ["shipping"],
  },
]);
console.log(suite.id);

Then run against it by ID or exact name. A name must match exactly one dataset in the workspace. A missing or ambiguous name fails rather than picking a close match.

from atlanai import Eval

run = Eval("support-agent", dataset=suite.id, task=answer, scores=[accuracy])

The dataset name is unique in the workspace, and pushing to a name that already exists adds to that dataset. Check suite.created, and each record’s action in suite.records, if the name might already be taken. Each entry of suite.records also maps your case name (key) to its record ID (id); results carry only the record ID as case_id, so keep the map to print readable failures: names = {r.id: r.key for r in suite.records}.

A push never deletes records. A case you remove from your code stays in the dataset and keeps running. Archive it in the Atlan app, or with client.datasets.records.archive(dataset_id, record_id).

How a record reaches your task and scorer

Record field	Becomes	Notes
`input[input_key]`	The value passed to `task`	`input` must be a JSON object. `input_key` defaults to `question`. A record without that key fails the run.
`expected`	`expected` in your scorer	A single-key `{"value": x}` is unwrapped to `x`. Any other shape is passed as stored.
`extra`	`metadata` in your scorer and in task hooks	Use it for case-specific settings, such as a fixture ID or a tolerance.
`categories`	The case’s `tags`	Use them to slice results: which categories regressed?
Record ID	`case_id` and `dataset_record_id` on the result	The join key across experiments.

If the readable question is not under question, set input_key when the dataset is created: push_dataset(client, name, records, input_key="prompt"). It cannot be changed by a later push. Atlan exposes input[input_key] as display_input on each record, so tables in the app and coding agents know which field is the question. Pass the whole input object instead of one field by keeping it under the key. For example, {"question": {"text": "...", "locale": "fr"}} hands your task the inner object.

Versions and snapshots

Editing a record appends a new immutable version; the old one stays readable at GET /eval/v1/datasets/{dataset_id}/records/{record_id}/versions/{n}. When an experiment starts, Atlan stamps a dataset_snapshot with every live record’s ID, version, and content hash, and Eval runs from that snapshot. A correction therefore changes only later runs. A snapshot holds at most 10,000 records. Split a larger suite into several datasets.

Record where a case came from

source_kind and source_ref record provenance. Set them when the record is first pushed; a later push does not change them.

`source_kind`	`source_ref`	Use
`manual` (default)	none	Written by hand
`session`	A session ID in the same workspace; Atlan checks that it exists	Curated from a production session. See turn a failure into a test.
`trace`	A trace ID	Curated from one trace
`external`	Any reference up to 255 characters	Imported from another tool or a ticket

Use label for curation state such as needs_review, and categories for the scenario.

Filter records

GET /eval/v1/datasets/{id}/records accepts q (a substring over input, expected, label, and categories), label, category, source_kind, and source_ref. The SDK exposes the same filters on client.datasets.records.list.

CSV datasets

A dataset can also be backed by an uploaded CSV file, for example one created in the Atlan app. The CSV needs a unique, non-empty case_id column and a column named after the dataset’s input_key. Files are limited to 32 MiB, 100,000 rows, and 256 columns. Each upload is an immutable dataset version, and its content is readable at GET /eval/v1/datasets/{id}/versions/{n}/content. To create one over the API, reserve an upload, send the bytes, then create the dataset with the upload key:

# 1. Reserve an upload; the response carries upload_key.
curl -sS --fail-with-body -X POST "${ATLAN_BASE_URL}/file/v1/files/uploads" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/json" \
  -d "{\"size_bytes\": $(wc -c < cases.csv)}"

# 2. Send the file.
curl -sS --fail-with-body -X PUT "${ATLAN_BASE_URL}/file/v1/files/uploads/${UPLOAD_KEY}" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/octet-stream" \
  --data-binary @cases.csv

# 3. Create the dataset. Use POST /eval/v1/datasets/{id}/versions to add a version.
curl -sS --fail-with-body -X POST "${ATLAN_BASE_URL}/eval/v1/datasets" \
  -H "Authorization: Bearer ${ATLAN_API_KEY}" -H "Content-Type: application/json" \
  -H "X-Atlan-Workspace-Id: ${ATLAN_WORKSPACE_ID}" \
  -d "{\"workspace_id\": \"${ATLAN_WORKSPACE_ID}\", \"name\": \"csv-cases\", \"upload_key\": \"${UPLOAD_KEY}\", \"filename\": \"cases.csv\"}"

Eval runs record-backed datasets, the kind push_dataset creates. To run a CSV-backed dataset, read its content and record results with your own runner. Each result names its row by case_id instead of dataset_record_id. See bring your own runner.

Build an eval

Run and gate

Score production

Reference

Inline cases or a dataset

Push a dataset from code

How a record reaches your task and scorer

Versions and snapshots

Record where a case came from

Filter records

CSV datasets

​Inline cases or a dataset

​Push a dataset from code

​How a record reaches your task and scorer

​Versions and snapshots

​Record where a case came from

​Filter records

​CSV datasets

Inline cases or a dataset

Push a dataset from code

How a record reaches your task and scorer

Versions and snapshots

Record where a case came from

Filter records

CSV datasets