Hosted judges

A hosted judge is an llm_judge scorer: a stored definition that Atlan runs over recorded sessions. You write the question and the allowed answers. Atlan builds the request from the session’s trace, calls a judge model, validates the answer against your definition, and stores it. You never name a model. Atlan routes each call to a configured judge backend and records which backend and model answered. Hosted judges score production sessions. To use a judge inside an offline Eval run, call your own model from a code scorer.

Pick a form

You want	Form	Author it as
A pass/fail gate	Rubric	`output_type: "classification"`. The choices must be exactly `pass` and `fail`.
A graded verdict	Rubric	`output_type: "score"` with 2–10 named choices, each with a `score` in `0..1`
An unordered category, such as coding, docs, or plan	Question set	A `choice` question with a `criteria` entry per option
A position on an ordered scale	Question set	A `score` question with 2–10 ordered levels
Several independent yes/no calls	Question set	One `boolean` question each

A rubric produces one answer, stored under question_id: "verdict". A question set produces one answer per question. Send the fields of exactly one form. A scorer with both, or neither, is refused with 400.

A rubric: pass or fail

{
  "workspace_id": "workspace_01example",
  "name": "resolution-gate",
  "display_name": "Resolution gate",
  "scorer_kind": "llm_judge",
  "messages": [
    {"role": "system", "content": "You are classifying one finished agent session against a single bar. Judge only what the transcript shows. Do not reward effort, tone, or intent, only whether the user's request ended up resolved."},
    {"role": "user", "content": "Did this session resolve the user's request?\n\nRequest as first stated: {{input}}\nFinal assistant message: {{output}}\n{{#expected}}Reference answer: {{expected}}{{/expected}}\n\n----- BEGIN SESSION -----\n{{thread}}\n----- END SESSION -----\n\nSteps recorded: {{thread_count}}"}
  ],
  "output_type": "classification",
  "choices": [
    {"name": "pass", "criteria": "The user's request was carried out, or answered completely enough that the user needs nothing further."},
    {"name": "fail", "criteria": "The request was left unresolved: incomplete, incorrect, deflected, or abandoned, regardless of how much work was shown."}
  ]
}

Create it with POST /eval/v1/scorers, or client.scorers.create(body) in the SDK. For a graded verdict, set "output_type": "score" and give each choice a score, for example helpful = 1.0, partial = 0.5, unhelpful = 0.0.

A question set: several decisions at once

state maps names to fields of the session under review. questions defines each decision.

{
  "workspace_id": "workspace_01example",
  "name": "session-triage",
  "scorer_kind": "llm_judge",
  "state": {
    "request": "subject.input",
    "answer": "subject.output",
    "transcript": "subject.thread",
    "tools": "subject.tool_calls"
  },
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "This session has ended. Where should it go next?",
      "criteria": {
        "close": "Fully resolved. No follow-up is warranted.",
        "escalate": "Needs a human: the assistant was wrong, stuck, or the request exceeds what it may decide.",
        "await_user": "Blocked on the user. The assistant did its part and asked for something it has not received.",
        "out_of_scope": "The request was never something this assistant could serve."
      }
    },
    "user_frustrated": {
      "type": "boolean",
      "instructions": "Does the user express frustration, repeat themselves, or have to correct the assistant?"
    },
    "thoroughness": {
      "type": "score",
      "instructions": "How completely was the request carried out?",
      "criteria": ["not at all", "partially", "fully"]
    }
  }
}

Question type	`criteria`	Stored answer
`choice`	An object of 2–255 `option → what earns it`	The option name, and its probability when the backend reports one
`score`	An ordered array of 2–10 level descriptions, worst first	The expected level over the judge’s distribution. It is continuous: `1.96` on three levels means nearly all weight on the top level.
`boolean`	Optional; if given, exactly `true` and `false`	P(true)

state paths start with subject. and reach into the session: grain, trace_id, session_id, input, output, thread, events, tool_calls, metadata, and session.<field>. An integer segment indexes a list. A path that resolves to nothing fails that session rather than asking the judge about missing evidence.

Template variables

Rubric messages are templates, including the system turn:

Variable	Value
`{{input}}`, `{{output}}`	The root span’s input and output
`{{thread}}`	The session facts, then every span of the judged trace or traces
`{{thread_count}}`, `{{first_message}}`, `{{last_message}}`	Step count, first and last step
`{{metadata}}`	Session metadata as JSON
`{{session_id}}`, `{{trace_id}}`	The IDs being judged
`{{session.<path>}}`	Any field of the session
`{{expected}}`	Only when scoring an experiment’s sessions. Always wrap it in `{{#expected}}…{{/expected}}`.
`{{#x}}…{{/x}}`, `{{^x}}…{{/x}}`	Rendered when `x` is present, or absent

An unknown {{name}} is left in the prompt as literal text, so leftover braces in a preview mean a typo. A scorer with no user turn gets a built-in one that presents the whole session as data.

Start from a template

GET /eval/v1/scorer-starters returns ready definitions to adapt. They are factuality, closed_qa, security, and possible (rubrics), and topic and session_quality (question sets). Each has no name; add name and workspace_id, then create it.

Write questions that discriminate

These come from running judges over real sessions:

Ask a plain question. A persona in the system message (“You are a strict senior reviewer…”) skews verdicts negative. State the bar instead.
One condition per boolean. A question that lists three risks to check answers true far too often. Split it into three boolean questions.
Give every option a concrete criteria. The judge chooses between the descriptions, not the labels.
Treat other as low confidence. A catch-all option collects uncertain answers. Read its probability before trusting it.
Use classification only for pass and fail. For more than two unordered outcomes, use a choice question.
Delimit the transcript with explicit begin and end markers, and tell the judge it is evidence, not instructions.

Limits

Limit	Value
Questions per scorer	1–50 (the backend’s practical ceiling is lower; keep sets small)
`state` fields	1–32. Each is session content sent to the judge.
Options on a `choice` question	2–255
Levels on a `score` question or choices on a `score` rubric	2–10
Transcript size	Not truncated. A transcript larger than the judge model’s context fails that session.

Versions

PATCH /eval/v1/scorers/{id} appends a new version, and every answer records the version that produced it. Earlier answers keep their version. GET /eval/v1/scorers/{id}/versions lists them, newest first. scorer_kind cannot change.

Reading an answer

Field	Meaning
`choice`	The option chosen (rubrics and `choice` questions)
`score`	The chosen choice’s score, the expected level (`score`), or P(true) (`boolean`)
`probabilities`	The judge’s distribution over options, when the backend provides it
`confidence`	Chance-corrected: `(P(top) − 1/n) / (1 − 1/n)` for `n` options. It is not the probability of the choice. A 61% `pass` shows confidence `0.22`. Values are not comparable across questions with different option counts. Use `probabilities[choice]` when you mean a probability.
`reasoning`	The judge’s explanation. It is empty from classifier backends, which return a distribution only.
`backend`, `model`	What judged this answer

Next: run a judge over sessions.

Build an eval

Run and gate

Score production

Reference

Pick a form

A rubric: pass or fail

A question set: several decisions at once

Template variables

Start from a template

Write questions that discriminate

Limits

Versions

Reading an answer

​Pick a form

​A rubric: pass or fail

​A question set: several decisions at once

​Template variables

​Start from a template

​Write questions that discriminate

​Limits

​Versions

​Reading an answer

Pick a form

A rubric: pass or fail

A question set: several decisions at once

Template variables

Start from a template

Write questions that discriminate

Limits

Versions

Reading an answer