Score recorded runs

Call extension__eval__score with the inputs below. Open an MCP session with mcp__initialise first and pass its session_id. The authenticated tools/list response is the final check for the name and schema available to your account. This is a write: it runs the selected scorer and stores answers. Verify the returned result and use the insight lookup to inspect stored scores.

Inputs

Input	Type	Required	Meaning
`trace_ids`	array	No	OTel trace ids to judge, at most 100 with `session_ids` combined. Each is judged as that trace of the session it records — a run list holds these and nothing else, and a session with several runs is judged by the one named. Default: `[]`.
`session_ids`	array	No	Session artifact handles (`session_…`) to judge, each by its earliest trace. Default: `[]`.
`agent_ids`	array	No	Agent handles (`agent_…`). Judge the sessions those agents ran, newest first — narrowed by `since`/`until` when given, at most `limit` in all. An agent, not an agent version. A `session` records which agent it ran, not which version of it, so “this agent at version 3” is not a question this selector can answer and is not silently approximated. Default: `[]`.
`experiment_id`	string,null	No	An experiment handle (`experiment_…`). Judge the sessions its results ran as, in result order, at most `limit`. Results recorded by a harness that logged no session are skipped: there is nothing to judge. Default: `null`.
`since`	string,null	No	RFC 3339. Judge the scorer’s workspace’s sessions created at or after this instant, newest first. With `agent_ids`, narrows those agents’ sessions instead of selecting on its own. Default: `null`.
`until`	string,null	No	RFC 3339. Upper bound of the window; defaults to now. Default: `null`.
`limit`	integer,null	No	How many sessions to judge, at most 100. Defaults to 20. Default: `null`.
`grain`	object	No	What each verdict is about: `trace` (the default) judges one run, `session` judges every run the session recorded, as one transcript. The wide grain is asked for rather than assumed because it reads proportionally more: a session with twenty turns sends twenty runs to the judge, and the judge is charged for all of it. Default: `"trace"`.
`mode`	object	No	`score` (the default, asynchronous, writes insights), `preview` (build the judge’s request and return it, calling nothing) or `test` (judge and return the answers, writing nothing). Default: `"score"`.
`concurrency`	integer,null	No	How many judges to keep in flight. Defaults to 4; the ceiling is this deployment’s `GATEWAY_EXT_EVAL_JUDGE_CONCURRENCY_MAX` (25 when it sets none). Both numbers are unmeasured, and deliberately not dressed up. The old ceiling was read off Claude Managed Agents’ published limits; that runtime is gone from this path, and neither TypeSafe nor the gateway’s upstream providers publish a per-organization limit this extension could cite. Raise it for a run you are watching: the run row records what it ran at, what each judge call took, and whether the backend refused for load — which is what a future default should be argued from. Default: `null`.
`scorer_id`	string	Yes	The `scorer_…` handle to run.
`rationale`	string	Yes	A concise, first-person explanation of why you are calling this tool right now: what you are trying to accomplish for the user and how this specific call advances that goal. Written for a human auditor reviewing tool usage after the fact. Do not restate the arguments - explain the intent behind them.
`session_id`	string	Yes	The id `mcp__initialise` returned for this conversation, so this call is recorded against it. Required: call `mcp__initialise` first and pass the id it returns. Do not invent a value - an id that names no session of yours is refused.

Permissions and result

The call uses your authenticated Atlan identity and the target workspace’s permissions. This tool changes recorded data. A missing or inaccessible target can be refused even when the tool is listed. See MCP tool catalog for related tools and MCP Gateway errors for refusal and connection diagnostics.

CLI

SDK

MCP

API

Inputs

Permissions and result

​Inputs

​Permissions and result

Inputs

Permissions and result