The Evals section in the app is moving to each agent’s Improve tab: the app marks it Moving, and its sections will be deprecated soon. Set up and run tests from an agent’s Improve tab.
Core offerings
Test fixed cases
Run your task and scorers with the atlanai SDK, and inspect the trace behind a failure.
Compare agent versions
Review case scores and traces before choosing a version.
Gate pull requests
Fail a CI check when scores regress.
Score recorded sessions
Use a hosted judge to assess real interactions.
Choose where to start
- New to evals? Run your first eval scores two cases and inspects the trace behind a failure.
- Setting up evals with a coding agent? Set up evals with coding agents.
- Replaying cases through your local coding client? Run experiment in Atlan Trident Desktop.
- Recording results from your own test harness? Record runs from your own harness.
- Comparing individual cases and traces? Review experiment results.
- Adding a production failure to a dataset? Turn production failures into test cases.
- An eval failed or a score is missing? Eval errors.
SDK tasks and code scorers run in your process or CI job, and Desktop experiments use your local coding client. A hosted online judge receives the selected session transcript: apply your approved data policy and Control trace privacy.
See also
- Eval options and errors: Every
Evaloption, status, limit, and error code. - Evals API: Experiment, dataset, and scorer operations.