Eval options and errors

Eval options

Python: Eval(name, *, data | dataset, task, scores, **options), or await async_eval(...) with the same arguments. TypeScript: await Eval(name, { data | dataset, task, scores, experimentName?, description? }, options).

Python	TypeScript	Default	Meaning
`name`	`name`	required	Stable eval name. Auto-registered scorers are named `<name>-<scorer>`. The CI action matches baselines on it.
`data`	`data`	one of `data`/`dataset`	Inline cases: a list, iterable, async iterable, or a callable returning one. Each case has `input` and optional `expected`, `metadata`, `tags`, and `id`.
`dataset`	`dataset`	one of `data`/`dataset`	A dataset ID or its exact name, record-backed or CSV-backed.
`task`	`task`	required	`task(input)` or `task(input, hooks)`, sync or async
`scores`	`scores`	required, 1 or more	Scorers; see the contract
`experiment_name`	`experimentName`	`name`	Display name of the experiment
`description`	`description`	none	Experiment description
`config`	`config`	`{}`	Free-form run settings, stored on the experiment and filterable
`baseline_experiment_id`	`baselineExperimentId`	none	The experiment to compare with
`subject_kind`, `subject_id`	`subjectKind`, `subjectId`	none	`agent`, `harness`, or `skill`, and its ID. Set both or neither.
`context_manifest`	`contextManifest`	none	A context manifest
`max_concurrency`	not available	`1`	Cases in flight (Python). TypeScript runs cases one at a time.
`resume_experiment_id`	`resumeExperimentId`	none	Continue a `running` experiment
`on_start`	`onStart`	none	Called with the experiment ID when the run starts
`client`	`client`	from the environment	A configured `AtlanClient`
`trace_logger`	`logger`	a new logger	Your tracing logger. It must export every span with content.
`gateway_origin`, `api_key`, `workspace_id`, `project_id`	`gatewayOrigin`, `apiKey`, `workspaceId`, `projectId`	from the environment	Connection overrides
`return_results`	`returnResults`	`true`	Include per-case results in the return value
`require_scores`	`requireScores`	`true`	After finalizing, raise with the experiment ID if no scorer produced a usable value
`verify_traces`	`verifyTraces`	`true`	Wait until every case trace is queryable before uploading its result
`trace_verification_timeout_s`	`traceVerificationTimeoutMs`	`30` s	How long to wait for each trace

Task hooks (the second task parameter): expected, metadata, tags, span (the task span; call .update(...) on it), and trial_index / trialIndex, which is always 0.

Return value

Python	TypeScript	Contents
`experiment_id`	`experimentId`	The experiment ID
`experiment`	`experiment`	The finalized experiment, including `dataset_snapshot`
`summary`	`summary`	See below
`results`	`results`	One per case: `case_id`/`caseId`, `input`, `expected`, `metadata`, `tags`, `dataset_record_id` (record-backed runs only), `output`, `error`, `scores`, `trace_id`/`traceId`, `duration_ms`/`durationMs`
`dataset`	`dataset`	The resolved dataset, when one was used

{
  "scores":  {"accuracy": {"mean": 0.5, "count": 2}},
  "scorers": {"accuracy": {"scorer_id": "scorer_01example", "scorer_version": 1}},
  "metrics": {"cases": 2, "successful_cases": 2, "failed_cases": 0, "duration_ms": 3101}
}

scores is computed by Atlan from the stored results when the experiment finalizes. null scores are excluded from mean and count. metrics is written by the SDK, with snake_case keys from Python and camelCase from TypeScript (successfulCases, failedCases, durationMs). Cost and tokens are not in the summary; query them with trace stats.

Environment variables

Variable	Used by	Meaning
`ATLAN_API_KEY`	`Eval`, tracing	Bearer token for API calls and trace export
`ATLAN_WORKSPACE_ID`	`Eval`, tracing	Workspace for every object the run creates
`ATLAN_BASE_URL`	`Eval`, tracing	Gateway origin. Default `https://api.atlan.com`.
`ATLAN_DEBUG`	tracing	`true` logs dropped-score and export warnings
`ATLAN_TRACING`, `ATLAN_TRACE_CONTENT`	tracing	Either set to `false` makes `Eval` refuse to run, because results would lack evidence
`ATLAN_EVAL_GIT_BRANCH`, `ATLAN_EVAL_GIT_SHA`, `ATLAN_EVAL_GIT_PR`, `ATLAN_EVAL_MODEL`, `ATLAN_EVAL_RESULTS_FILE`	Python `Eval`, and the TypeScript shim	Set by the CI action for any command. Set them yourself only to simulate the action locally.

AtlanClient reads no environment variables. Pass the origin, token, and workspace explicitly, as the examples do.

Statuses

Object	Values
Experiment `experiment_status`	`running` → `completed` or `failed`. A terminal experiment’s status and summary are frozen.
Scoring run `run_status`	`queued` → `running` → `succeeded` or `failed`
Record `source_kind`	`manual`, `session`, `trace`, `external`
Scorer `scorer_kind`	`code`, `llm_judge`, `human`

Limits

Limit	Value
Records in a dataset snapshot	10,000
Results folded into a summary	10,000 per experiment
Items per bulk create (records or results)	100
List page size	default 50, maximum 500
List `offset`	0 to 10,000
CSV dataset	32 MiB, 100,000 rows, 256 columns
`case_id`	Unique per experiment, up to 255 characters
`trace_id` on a result	32 lowercase hex characters, not all zeros
Score comment from `ScoreValue` metadata	2,000 characters
Hosted scoring	100 sessions per run, 5 in `preview`/`test`; default concurrency 4
Sampled selection	At most 1,000 candidates read per run; a larger population returns `400`

Error codes

A refused request (400, 409, or 422) says why it was refused in the response’s detail when that cause is safe to publish; match it against the typical causes below.

Status	Typical cause
`400`	A validation failure: a bad `trace_id`, a missing `dataset_record_id` on a record-backed run, a `case_id` not in the pinned CSV, a result `workspace_id` that differs from the experiment’s workspace, a half-set subject pair, or an invalid scorer definition or selector
`403`	Scoring, archiving, or an issue write without the required role
`404`	Not found, not visible to you, or a child requested under the wrong parent
`409`	Writing to a `completed` or `failed` experiment, a duplicate `case_id` (`duplicate_case_id`) or name, or an unknown workspace; or an issue’s links written while its counters kept changing — re-send the same answers to recount them; or `scorer_archived` from hosted scoring, or from an edit that changes a definition value on a scorer you can read but that is archived — it can neither run nor be edited; or a publish that raced another — `release_state_conflict`, when the live release is no longer the one the request’s `expected_current_release_id` expected; or `POST /issues/find` while the agent already has a `queued` or `running` scan written in the last 15 minutes, whose ID the problem body names in `active_scan_id` when it can be read
`412`	A stale `If-Match` on a scorer edit
`422`	An unknown enum value, moving a dataset or experiment to another workspace, or changing the `name` of a dataset, record, or experiment, which is fixed at creation; an unknown issue field, or a `null` on an issue field a patch cannot clear; or a link to an answer that is not trace-level evidence about the issue’s agent, named as `answer_ids[i]`
`503`	Traces are temporarily unreadable, a selected session could not be read, or no judge backend is configured for hosted scoring
`207`	A bulk create: read each item’s `status_code`

API routes

All under /eval/v1. Send Authorization: Bearer <token> and X-Atlan-Workspace-Id. Create bodies need workspace_id. Adding records with POST /datasets/{id}/records or POST /datasets/{id}/records/bulk requires each record’s workspace_id to match the owning dataset’s workspace; a record in another workspace is refused with 400. PATCH is the update verb.

Resource	Routes
Datasets	`POST /datasets`, `GET /datasets`, `GET·PATCH /datasets/{id}`, `POST /datasets/{id}/archive`, `POST /datasets/search`
Dataset versions (CSV)	`POST·GET /datasets/{id}/versions`, `GET /datasets/{id}/versions/{n}`, `GET /datasets/{id}/versions/{n}/content`
Records	`POST /datasets/{id}/records`, `POST /datasets/{id}/records/bulk`, `GET /datasets/{id}/records`, `GET·PATCH /datasets/{id}/records/{rid}`, `POST …/records/{rid}/archive`, `GET …/records/{rid}/versions`, `GET …/records/{rid}/versions/{n}`
Experiments	`POST /experiments`, `GET /experiments`, `GET·PATCH /experiments/{id}`, `POST /experiments/{id}/archive`, `POST /experiments/search`
Results	`POST /experiments/{id}/results`, `POST /experiments/{id}/results/bulk`, `GET /experiments/{id}/results`, `GET /experiments/{id}/results/{rid}`
Experiment traces	`GET /experiments/{id}/traces`, `GET /experiments/{id}/traces/{tid}`, `GET /experiments/{id}/traces/{tid}/spans`, `POST /experiments/{id}/traces/stats/query`
Scorers	`POST /scorers`, `GET /scorers`, `GET·PATCH /scorers/{id}`, `GET /scorers/{id}/versions`, `GET /scorers/{id}/versions/{n}`, `POST /scorers/{id}/archive`, `POST /scorers/{id}/publish`, `GET /scorer-starters`, `POST /scorers/{id}/score`
Issues	`POST /issues`, `GET /issues`, `GET·PATCH /issues/{id}`, `GET /issues/{id}/traces`, `POST /issues/{id}/traces`
Issue scans	`POST /issues/find`, `GET /issue-scans`, `GET /issue-scans/{scan_id}`

The search routes take query, an optional strategy (fts, vector, or hybrid), and page with limit and offset. They serve no cursor and no sorts: the strategy’s ranking is the order, so list with ?sort= for an ordered browse. Useful list filters: experiments take dataset_id, subject_kind and subject_id, experiment_status, baseline_experiment_id, config=key:value, and sort=-created_at. Results take case_id, dataset_record_id, and session_id. Records take q, label, category, source_kind, and source_ref. Issues list one agent’s issues in one workspace: workspace_id and subject_id are required, and the list takes one issue_status or severity, and sort=-last_seen_at or -occurrence_count. Reading an issue and its traces needs read access to its workspace (any member, viewers included); filing, editing, and linking traces need the builder or admin role. Issue scans list one agent’s running and past scans, newest first: workspace_id and subject_id are required, the agent must be in that workspace (otherwise 404), and the list takes one status (queued, running, succeeded, or failed). Starting, listing, and reading scans need the builder or admin role. POST /issues/find answers 202 with the queued scan and a Location header naming /eval/v1/issue-scans/{scan_id}. The SDK exposes the same tree: client.datasets, client.datasets.records, client.experiments, client.experiments.results, client.experiments.traces, and client.scorers. See the generated datasets, experiments, and scorers references.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost