Skip to main content
Build an eval guides you from a quality goal to a saved dataset and a judge. Use it to turn recent agent sessions into cases and check whether you agree with the grader before using its scores.
This flow is available only for accounts where Build an eval is enabled. If you cannot see it, use the SDK quickstart or create a dataset directly.

​
Before you begin

  • Evals must be enabled, and you need permission to create a dataset and scorer in the selected workspace.
  • Register the agent you want to evaluate. Recorded sessions give the pilot real evidence to judge.
  • For a new grader, choose an available judge backend and model. See Configure hosted judges.
  • Review what the sessions contain. The hosted judge receives their recorded content under your organization’s data policy.

​
Define the eval

  1. Open Evals and select Build an eval where it is offered.
  2. In Goal, choose the registered Agent and answer What does a good answer look like? Use a criterion a reviewer could check, such as “Cites the governed asset and states when the source does not contain an answer.”
  3. In Cases, select recent sessions and add manual prompts for cases the sessions do not cover.
  4. In Grader, choose Start from a template or Use an existing scorer. For a template, choose the judge backend and model.
  5. Check the destination workspace and any sessions the app says it will leave out, then continue.
The app now creates a draft dataset and its records. A template also creates a draft scorer; an existing scorer is reused. Sessions whose opening prompt cannot be read are left out with an explanation. Check the saved case count before continuing.

​
Check the grader

  1. In Pilot, select Run the pilot.
  2. Read each verdict and its reasoning, then mark whether you agree.
  3. If the grader needs a different rule, select Yes, edit the grader. Saving a change creates a new scorer version and reruns the pilot.
  4. Continue when the grader’s answers match the rule you intended.
The pilot judges a small set of the selected recorded sessions in test mode. It does not save those verdicts as production insights. A dataset made entirely from manual prompts has no recorded sessions to judge, so the app offers Skip pilot and leaves the grader unchecked.

​
Save the dataset

In Plan, check the number of cases, the pass-rate uncertainty, and any warning that the pilot already scores at the top. The time estimate uses the agent’s last completed bench when one exists; the app does not estimate cost here. Enter a name and select Save and open the dataset. Confirm the dataset contains the intended cases, then run an experiment or use it from the SDK. Cancelling after draft creation attempts to archive the drafts. If cleanup is refused, the app names the draft that remains so you can find it under Datasets. Editing a reused scorer remains a versioned change to that scorer.