Skip to main content
Atlan Desktop can run a dataset through a local coding agent and store each case’s outcome as an experiment. The execution happens on your machine through your signed-in CLI. Atlan stores the dataset snapshot, outputs, failures, timing, and available trace references.

​
Before you begin

  • Use Atlan Desktop with Evals enabled and permission to create experiments in the selected workspace.
  • Install and sign in to the local runner you intend to use: Claude Code or Codex.
  • Register an agent as the experiment’s subject and create a dataset with text prompts.
  • Check the local working directory and the skills installed in the runner. The registered agent is the subject of the experiment; selecting it does not replace the local runner’s context.
The web app opens the same setup drawer but offers Open in desktop app to execute the cases.

​
Configure the run

  1. Open Evals → Experiments and select New experiment, or start from the dataset’s run action.
  2. Choose the registered Subject and the Dataset.
  3. Review Scorers and remove definitions that do not apply.
  4. Under Compare to, choose a completed baseline or No baseline.
  5. Expand Runs on this machine. Choose the Local runner, Model, Permission, and working directory.
  6. Select Run. When the case count is known, the button names the number of cases it will run.
Read only is the default permission. Auto-edit permits local changes according to the coding agent’s permission policy. Review the selected directory before choosing it.
Selecting scorers records them in the experiment configuration. It does not execute arbitrary registered code scorers. For code-based scores, run the scorer functions through the SDK eval runner. Use hosted scoring separately for recorded sessions that have the required trace evidence.

​
Follow progress and verify results

The runner starts a fresh session for each case and runs cases one at a time. A failed case records an error and lets the remaining cases continue. The progress drawer stays open while the run is active. Select Stop after this case to stop between cases. A case already running finishes before the loop stops; its outcome remains in the experiment. After the run, select Open run and check:
  • The expected number of case outcomes was stored.
  • Failed cases carry errors, rather than blank outcomes counted as successes.
  • The effective model, when reported by the runner, matches the model you intended.
  • Available trace links open the recorded work. A stored output alone does not establish that a trace arrived.
If result upload fails, the app reports that the cases ran but could not be uploaded. Read the experiment before rerunning, so you can distinguish execution from missing stored results.

​
Current limits

The desktop runner uses the context already installed in the local harness. Context versions are read-only; selecting registry versions for the runner is unavailable. Repeat is disabled, so this drawer does not schedule recurring experiments. Each case must resolve to one text prompt. For ordinary records, the runner uses input.prompt, or the only nonempty text field when there is one. CSV datasets use their declared input key; structured input requires a custom runner. See Record runs from your own harness. Continue with Review experiment results.