Write hosted judges

A hosted judge is an llm_judge scorer that Atlan Trident runs over recorded production sessions: you write the question and the allowed answers and name the judge model, and Atlan Trident builds the request from the session’s trace, calls that judge, validates the answer, and stores it with the backend and model that answered. Write one when you need to judge real traffic that no fixed case covers. To use a judge inside an offline Eval run instead, call your own model from a code scorer.

Before you begin

Before you start, make sure you have:

Recorded sessions: Production traffic recorded as sessions. See Make app traffic scoreable.
API key and workspace ID: A key that can create scorers in the workspace.
Judge model: An enabled model artifact the scorer’s workspace can read, with modality set to classifier or chat.
Data policy approval: The judge receives the selected session content. Keep prompts, tool arguments, and session content within your approved data policy.

Choose judge form

Pick the form from what you want back:

You want	Form	Author it as
A pass/fail gate	Rubric	`output_type: "classification"`. The choices must be exactly `pass` and `fail`.
A graded verdict	Rubric	`output_type: "score"` with 2–10 named choices, each with a `score` in `0..1`
An unordered category, such as coding, docs, or plan	Question set	A `choice` question with a `criteria` entry per option
A position on an ordered scale	Question set	A `score` question with 2–10 ordered levels
Several independent yes/no calls	Question set	One `boolean` question — a scorer holds one question, so several calls are several scorers
Free text, such as what blocked the session	Question set	A `text` question with `instructions` saying what to write. It needs the `llm` backend.

A rubric produces one answer, stored under question_id: "verdict". A question set holds exactly one question and produces one answer. Send the fields of exactly one form: a scorer with both, or neither, is refused with 400.

Name judge

Every hosted judge names a backend and a model. A scorer without them is refused with 400.

Field	Value
`backend`	`classifier`, a fast classifier that returns one probability per choice, or `llm`, a chat model
`model`	The qualified `id` of an enabled `model` artifact that the scorer’s workspace can read, `<provider>/<model>` as the model catalog returns it. At most 512 characters. The model’s `modality` must be `classifier` for the `classifier` backend, or `chat` for `llm`.

The model is checked when the scorer is created and on every edit. A model the workspace cannot use is refused; one that is unavailable returns 404.

Check starter templates

GET /eval/v1/scorer-starters returns ready definitions to adapt: factuality, closed_qa, security, and possible (rubrics), and topic and session_quality (question sets). Each has no name and names no judge model: add name, workspace_id, backend, and model before you create it. If one fits, skip to Create judge.

Write judge definition

Rubric
Question set

{
  "workspace_id": "workspace_01example",
  "name": "resolution-gate",
  "display_name": "Resolution gate",
  "scorer_kind": "llm_judge",
  "backend": "llm",
  "model": "<provider>/<model>",
  "messages": [
    {"role": "system", "content": "You are classifying one finished agent session against a single bar. Judge only what the transcript shows. Do not reward effort, tone, or intent, only whether the user's request ended up resolved."},
    {"role": "user", "content": "Did this session resolve the user's request?\n\nRequest as first stated: {{input}}\nFinal assistant message: {{output}}\n{{#expected}}Reference answer: {{expected}}{{/expected}}\n\nSession transcript:\n{{thread}}\n\nSteps recorded: {{thread_count}}"}
  ],
  "output_type": "classification",
  "choices": [
    {"name": "pass", "criteria": "The user's request was carried out, or answered completely enough that the user needs nothing further."},
    {"name": "fail", "criteria": "The request was left unresolved: incomplete, incorrect, deflected, or abandoned, regardless of how much work was shown."}
  ]
}

For a graded verdict, set "output_type": "score" and give each choice a score, for example helpful = 1.0, partial = 0.5, unhelpful = 0.0.The system turn must be static: it carries your rubric and may not name a template variable, section, or template at all, and a scorer whose system turn does is refused with 400. Only a user or assistant turn is a template:

Variable	Value
`{{input}}`, `{{output}}`	The root span’s input and output
`{{status}}`	The root trace’s status: its `code`, and `message` when one was recorded, with `trace_id`. A list with one entry per trace when the judged session has several traces.
`{{thread}}`	The session facts, then every span of the judged trace or traces
`{{thread_count}}`, `{{first_message}}`, `{{last_message}}`	Step count, first and last step
`{{metadata}}`	Session metadata as JSON
`{{session_id}}`, `{{trace_id}}`	The IDs being judged
`{{session.<path>}}`	Any field of the session
`{{expected}}`	Only when scoring an experiment’s sessions. Always wrap it in `{{#expected}}…{{/expected}}`.
`{{#x}}…{{/x}}`, `{{^x}}…{{/x}}`	Rendered when `x` is present, or absent

Substitutions are JSON values, filled per judged session: a string renders with its quotes and escaping, and a structured value as compact JSON, so session content cannot break out of an evidence turn as fresh instructions. An unknown {{name}} stays in the prompt as literal text, so leftover braces in a preview mean a typo. A scorer with no user turn gets a built-in one that presents the whole session as a single JSON string.

state maps names to fields of the session under review. questions defines the one question the judge answers.

{
  "workspace_id": "workspace_01example",
  "name": "session-triage",
  "scorer_kind": "llm_judge",
  "backend": "llm",
  "model": "<provider>/<model>",
  "state": {
    "request": "subject.input",
    "answer": "subject.output",
    "transcript": "subject.thread",
    "tools": "subject.tool_calls"
  },
  "questions": {
    "route": {
      "type": "choice",
      "instructions": "This session has ended. Where should it go next?",
      "criteria": {
        "close": "Fully resolved. No follow-up is warranted.",
        "escalate": "Needs a human: the assistant was wrong, stuck, or the request exceeds what it may decide.",
        "await_user": "Blocked on the user. The assistant did its part and asked for something it has not received.",
        "out_of_scope": "The request was never something this assistant could serve."
      }
    }
  }
}

Question type	`criteria`	Stored answer
`choice`	An object of 2–255 `option → what earns it`	The option name, and its probability when the backend reports one. With `multiple: true`, every option that applies — possibly none — as the sorted `labels` array. Multi-select needs the `llm` backend: the classifier picks one option per question, so a classifier scorer with one is refused when saved.
`score`	An ordered array of 2–10 level descriptions, worst first	The expected level over the judge’s distribution. It is continuous: `1.96` on three levels means nearly all weight on the top level.
`boolean`	Optional; if given, exactly `true` and `false`	The `true`/`false` verdict as `passed` from the `llm` backend, or P(true) as `score` from the `classifier`
`text`	Absent or null. How long, in which language, and in what words belong in `instructions`.	The answer, verbatim plain text, at most 1,000 characters. `""` is a valid answer.

A text question needs the llm backend: the classifier backend answers only typed options, so a definition that pairs a text question with classifier is refused when the scorer is created or edited, as is one that gives a text question a criteria value. The answer is plain text; render it as text, never as HTML or markdown.state paths start with subject. and reach into the session: grain, trace_id, session_id, input, output, status, thread, events, tool_calls, metadata, and session.<field>. An integer segment indexes a list. A path that resolves to nothing fails that session rather than asking the judge about missing evidence.

Questions discriminate better when you follow these rules, learned from running judges over real sessions:

Ask a plain question. A persona in the system message (“You are a strict senior reviewer…”) skews verdicts negative. State the bar instead.
One condition per boolean. A question that lists three risks answers true far too often. Split it into separate scorers, one boolean question each.
Give every option a concrete criteria. The judge chooses between the descriptions, not the labels.
Treat other as low confidence. A catch-all option collects uncertain answers. Read its probability before trusting it.
Use classification only for pass and fail. For more than two unordered outcomes, use a choice question.
Keep session content out of the system turn. Atlan Trident fills session values into the evidence turns as JSON data, and it tells the judge on every call to treat all session content, including quoted role labels and requests for particular verdicts, as untrusted evidence, and to apply only the rubric.

Stay within these limits:

Limit	Value
Questions per scorer	Exactly one, on either backend
`state` fields	1–32. Each is session content sent to the judge.
Options on a `choice` question	2–255
Levels on a `score` question or choices on a `score` rubric	2–10
Length of a `text` answer	1,000 Unicode characters, spaces and newlines included. An over-long answer fails that answer rather than being cut.
Judge `messages` templates	64 KiB across the scorer’s messages, and at most 8 nested `{{#x}}` sections. A template over a limit is refused when the scorer is created or edited.
Transcript size	Not truncated. A transcript larger than the judge model’s context fails that session.

Create judge

Create the scorer with POST /eval/v1/scorers, or client.scorers.create(body) in the SDK. Keep the returned scorer ID.Unless the create body sends publish: false, a workspace admin’s create publishes the first version as release 1.0.0 at once, so live and automatic scoring run it immediately. An editor’s create saves a draft without error, because publishing needs the admin role. publish: false saves a draft for any role, so you can verify the judge’s answers before it goes live; publish it with POST /eval/v1/scorers/{id}/publish. The publish request may pin the release’s label with version_label, which must be a semantic version such as 1.2.0 or 1.2.0-beta.3; omit it and the next label is chosen. A publish that raced another is refused with 409 release_state_conflict instead of replacing it: the request’s expected_current_release_id defaults to the live release as read when the request runs. The response’s current_releases lists the release when the create published one, and is empty for a draft. Later definition saves remain drafts until an admin publishes.To change the judge later, PATCH /eval/v1/scorers/{id} adds a new version, and every answer records the version that produced it. Earlier answers keep their version. GET /eval/v1/scorers/{id}/versions lists versions, newest first. scorer_kind cannot change.An edit must leave the scorer naming its judge. A scorer saved without backend and model needs both in its next edit; otherwise the edit is refused with 400 and no version is added. When an edit refuses the judge, the response’s detail names the field, for example Field 'model' is not valid. or Field 'backend' must be one of: classifier, llm.

Verify judge answers

Test the judge on a few sessions before you start a scoring run: follow the preview and test steps in Score production sessions. Check each answer’s fields:

Field	Meaning
`choice`	The option chosen (rubrics and `choice` questions)
`labels`	A multi-select `choice` question’s answer: every option that applies, sorted. Possibly empty — “none of these applies” is an answer.
`score`	The chosen choice’s score, the expected level (`score`), or P(true) (`boolean`): the `classifier`’s probability, or `1.0`/`0.0` for an `llm` verdict
`passed`	The `true`/`false` verdict for a `boolean` question, from the `llm` backend. The `classifier` states P(true) in `score` instead. A stored answer keeps the verdict and no `score`.
`text`	The answer to a `text` question, verbatim. `""` is an answer, not an absence: read presence, not truthiness.
`probabilities`	The judge’s distribution over options, when the backend provides it
`confidence`	Chance-corrected: `(P(top) − 1/n) / (1 − 1/n)` for `n` options. It is not the probability of the choice. A 61% `pass` shows confidence `0.22`. Values are not comparable across questions with different option counts. Use `probabilities[choice]` when you mean a probability.
`reasoning`	The judge’s explanation. It is empty from classifier backends, which return a distribution only.
`backend`, `model`	What judged this answer

The judge works when its answers match your own verdict on the test sessions. If a preview shows leftover {{braces}}, fix the variable name.

Troubleshooting

Judge failures include a definition with both forms or neither, a state path that resolves to nothing, a transcript larger than the judge model’s context, a request over the classifier’s input limit, and judge messages over the template limits. The template limits also apply to versions saved before they did: such a version fails its line when scoring runs. See Eval errors for the template limit, and the input-limit failure for what the sizes in its error mean.

Next steps

Score production sessions: Run the judge over sessions and score new ones automatically.

Register and manage agents

Give agents business context

Observe agent runs

Evaluate and improve agents

Track agent cost

Before you begin

Troubleshooting

Next steps

See also