Before you begin
Before you start, make sure you have:- Data policy approval: The judge receives session content. Apply tracing privacy controls before you collect real traffic, and confirm that your data policy allows sending transcripts to a judge model.
-
Subscription: An Atlan subscription that includes scoring. Without it, every mode returns
403. -
Judge: An
llm_judgescorer in the workspace. See Write hosted judges. - Role: The builder or admin role in the scorer’s workspace. Every mode, including preview, checks it.
-
Judge backend: A judge backend configured on your deployment. Without one, the score route returns
503. -
Shell variables: For the HTTP examples:
1
Make app traffic scoreable
A hosted judge selects sessions, then reads their traces, and the sessions must be in the same workspace as the scorer. If your app already records sessions with readable traces, skip to Preview judge request. Tracing alone records traces; to make them judgeable, give each conversation a session ID on its spans and record the session when it ends:
external_session_id joins the session to its traces. subject_id is the agent’s ID; the agent_ids selector matches on it. Check the join before you score: client.sessions.traces.list(session.id) must list the trace. Trace ingest is asynchronous, so poll it, for example every 5 seconds for up to a minute, before you conclude the join failed. A session with no traces fails with “nothing to judge”.2
Preview judge request
Preview builds the judge request for up to five sessions and returns it. It makes no model call, costs nothing, and writes nothing, so it is the fastest way to check that your template variables and Each result line carries
state paths resolve.preview_request, with the backend the call would route to and the rendered messages. Literal {{…}} left in the rendered prompt means a misspelled variable. A line that fails still names the backend the call would have routed to, even though it carries no preview_request; for a request over the classifier’s input limit, the error also reports the request’s size and the ceiling. See Eval errors.3
Test judge on few sessions
Send the same request with
"mode": "test". The judge is called on up to five sessions and each line returns its answer: choice, score, probabilities, reasoning, backend, and model. Nothing is written. Change the scorer until the test answers match your own reading of those sessions.4
Start scoring run
"mode": "score" is the default. The route resolves the sessions, records a scoring run, and returns 202 straight away. The SDK wraps the same route as score_with in Python and scoreWith in TypeScript.- HTTP
- Python
- TypeScript
Send the request from the preview step without
mode. The response:400. A session_ids entry that names no session you can read in the scorer’s workspace returns 404, the same response whether the session is absent, unreadable, archived, or in another workspace. A trace_ids entry returns 404 when the trace itself is absent or unreadable; a trace you can read but whose session cannot be resolved returns a per-subject error line saying the trace has no readable session, and the call’s other lines still get answers.5
Verify stored answers
Judging happens in the background. The run moves from
queued to running, then succeeded or failed.- Check in the app: Open Evals, then Scorers, and select the scorer. Its page shows the run’s progress and the stored answers. An agent’s verdicts on its runs also appear with the agent’s runs on the Traces page; see Investigate sessions and runs.
- Check in code: Follow the run from an agent connected to Atlan MCP, and read the answers with
GET /eval/v1/insights:- The run: Call
registry__get_artifactwith thescore_run_id. Itsmetadataholdsrun_status,total,done, andfailed. Itssource_metadata.resultsgains a line per subject as the run progresses, with the answers and theinsight_idwritten, or anerrorsaying why there is none. Once the run finishes, every subject has a line. - The answers: Call
extension__eval__list_insightswith thescorer_id. It returns one row per subject and question, newest first, with thetrace_id,session_id,execution_id(which groups one run’s answers),question_id,answer_type,choice,passedorscore,text(the answer to atextquestion, verbatim),confidence,reasoning,scorer_version,backend, andmodel. Setfulltotrueonly to inspect the stored row, includingprobabilitiesand the request the judge was sent; a page of full rows can be very large. The tool also reads an agent’s insights: withworkspace_idandagent_id(ortrace_id,session_id,execution_id, orscorer_id), it returns every scorer’s canonical trace answers in that workspace — the same read the next bullet serves over REST. - The answers over REST: Call
GET /eval/v1/insightswithworkspace_idand thescorer_id. It returns the stored answers, newest first byobserved_at, each with itstrace_idandsession_id, theagent_idandagent_versionthat produced the trace, thescorer_versionthat answered, thechoice,passedorscorethe scorer gave,confidence,reasoningwhen the judge gives one, and theexecution_idthat groups one run’s answers. Withouttrace_idthe read covers the 30 UTC days ending today; widen it withstart_dateandend_date(at most 90 days), or name atrace_idto read one trace’s answers across scorers with no date window. Narrow further withpassed,choice,score_min/score_max, or one agent, agent version, or scorer version. Pages hold 25 answers by default and at most 100, carry no total, andoffsetstops at 10,000. You need read access to the workspace (any member, viewers included), and the scorer must belong to it. - The answers for a page of runs: Call
GET /eval/v1/answerswithtrace_idsand/orsession_ids(comma-separated, at most 200 together) and optionally at most 20scorer_ids. It returns one page of stored answers, newest first, compact: each row carries thestoreit came from, thescorer_idand version, thechoice,passed,scoreortextthe scorer gave, theconfidence, thereasoningwhen the judge gives one, and the backend and model that answered. Pages carryhas_moreandnext_cursorwith no total; keep paging with the same filters untilhas_moreis false before treating a subject as unanswered, and drop repeatedids. The read spends the same per-caller budget asGET /eval/v1/insights, and every workspace a page returns answers from is checked before the page is served: a page holding an answer from a workspace that is not visible to you is refused with404.
- The run: Call
succeeded with some failed sessions is normal: sessions with no readable trace, or with a transcript larger than the model’s context, are reported and skipped. A named session outside the scorer’s workspace is refused before the run starts, so it never appears as a failed line.A run sampled by trace time (the since or agent_ids selectors) also skips readable traces that record no session: they are not judged and do not fail the run. The response and the run’s source_metadata.selector report how many of these candidates were skipped as skipped_sessionless_count, counted up to where selection stopped at its limit or the end of the scan.Scoring the same session with the same scorer again writes a new answer. Earlier answers are kept, each recording the scorer version that produced it. The app shows only the most recent answer for each session, scorer, subject, and question. Scoring at a different grain produces a separate answer.6
Score new sessions automatically (optional)
To judge sessions as they finish, open Evals, then Scorers, and select Run live; or open the scorer and turn on Automation. Only account admins can turn it on. Set Whose sessions, Each verdict covers (the grain), and How many (the sample rate). Sampling is deterministic per session, so a session is either in or out every time. Answers then appear on the scorer’s page as sessions complete.Automatic scoring runs only when a session reaches
completed or failed. A session that never records an end is not scored automatically; score it on demand with its ID.Troubleshooting
Scoring failures include403 without a scoring subscription, 409 with code scorer_archived when the scorer is archived, 503 without a judge backend, and “nothing to judge” for a session with no joined trace. See Eval errors.
Next steps
- Turn production failures into test cases: A low score is a lead, not a label. Check the session, then keep the failure as an offline case so the next pull request is gated on it.
See also
- Write hosted judges: Rubric and question-set definitions, template variables, and limits.
- Add scores from your application: Score live requests from your own code instead.
- Tool catalog: The eval MCP tools, including
extension__eval__list_insights.