> ## Documentation Index
> Fetch the complete documentation index at: https://platform.atlan.com/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> To act on Atlan objects, use the Atlan MCP server at https://api.atlan.com/mcp or the SDK. To install or publish skills from a terminal, use the `aireg` CLI; `aireg --map json` prints its command map. The `atlanai` CLI is deprecated. Run a read-only identity check before any write.
> The docs MCP server at /mcp searches these docs only. It cannot read or change Atlan objects.
> SDK packages: Python `atlanai` (PyPI) and TypeScript `@atlanai/sdk` (npm). Show Python first, then TypeScript.
> These docs cover Atlan AI Registry, Atlan's hosted product. The AI registry specification and aireg CLI are maintained separately. /open-source lists publicly reachable signed installers and explains that the source repository and hosted CLI reference currently require GitHub access. Do not claim the repository or its reference docs are publicly accessible. Use these docs for Atlan-specific setup such as the adapter, sign-in and workspaces.

# Build an eval in the app

> Define a goal, select session cases, test a judge against your review, and save an eval dataset.

**Build an eval** guides you from a quality goal to a saved dataset and a judge. Use it to turn recent agent sessions into cases and check whether you agree with the grader before using its scores.

<Note>
  This flow is available only for accounts where **Build an eval** is enabled. If you cannot see it, use the [SDK quickstart](/evals/quickstart) or [create a dataset](/evals/datasets) directly.
</Note>

## Before you begin

* Evals must be enabled, and you need permission to create a dataset and scorer in the selected workspace.
* Register the agent you want to evaluate. Recorded sessions give the pilot real evidence to judge.
* For a new grader, choose an available judge backend and model. See [Configure hosted judges](/evals/judges).
* Review what the sessions contain. The hosted judge receives their recorded content under your organization's data policy.

## Define the eval

1. Open **Evals** and select **Build an eval** where it is offered.
2. In **Goal**, choose the registered **Agent** and answer **What does a good answer look like?** Use a criterion a reviewer could check, such as “Cites the governed asset and states when the source does not contain an answer.”
3. In **Cases**, select recent sessions and add manual prompts for cases the sessions do not cover.
4. In **Grader**, choose **Start from a template** or **Use an existing scorer**. For a template, choose the judge backend and model.
5. Check the destination workspace and any sessions the app says it will leave out, then continue.

The app now creates a draft dataset and its records. A template also creates a draft scorer; an existing scorer is reused. Sessions whose opening prompt cannot be read are left out with an explanation. Check the saved case count before continuing.

## Check the grader

1. In **Pilot**, select **Run the pilot**.
2. Read each verdict and its reasoning, then mark whether you agree.
3. If the grader needs a different rule, select **Yes, edit the grader**. Saving a change creates a new scorer version and reruns the pilot.
4. Continue when the grader's answers match the rule you intended.

The pilot judges a small set of the selected recorded sessions in test mode. It does not save those verdicts as production insights. A dataset made entirely from manual prompts has no recorded sessions to judge, so the app offers **Skip pilot** and leaves the grader unchecked.

## Save the dataset

In **Plan**, check the number of cases, the pass-rate uncertainty, and any warning that the pilot already scores at the top. The time estimate uses the agent's last completed bench when one exists; the app does not estimate cost here.

Enter a name and select **Save and open the dataset**. Confirm the dataset contains the intended cases, then [run an experiment](/evals/how-tos/run-desktop-experiment) or use it from the [SDK](/evals/offline).

Cancelling after draft creation attempts to archive the drafts. If cleanup is refused, the app names the draft that remains so you can find it under **Datasets**. Editing a reused scorer remains a versioned change to that scorer.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.