extension__eval__aggregate_insights with the inputs below. In search mode, the default, run this tool through mcp__call_read_tool, the runner its run_with names: send the inputs below as its arguments, and put session_id and rationale beside them, not inside. In full mode, open a session with mcp__initialise first and pass its session_id. The authenticated mcp__describe_tools result in search mode, or tools/list response in full mode, is the final check for the name and schema available to your account.
Count stored scorer answers instead of paging them: one card per scorer over whole UTC days, with the totals and the splits by agent version and by scorer version. A pass/fail scorer’s card has passed_count, failed_count, and pass_rate; a choice scorer’s has choice_counts; a score scorer’s has score_statistics. Text answers are not counted. The read counts what is stored and runs no scorer.
One scorer’s card. Name workspace_id and scorer_id. Add agent_id to count one agent’s answers, or scorer_version to count one saved version’s. The window is start_date and end_date (YYYY-MM-DD, UTC, at most 90 days apart), by default the 30 days ending today. group_by, one or two of repo, harness, and user, adds a flat groups table to each card; user needs permission to manage the workspace.
A card per scorer for an agent. Name workspace_id and agent_id without scorer_id: the read returns a card for each scorer that answered about that agent in the window. project_id keeps the scorers of one project.
Compact by default. A card leaves out the daily series, keeps the newest 5 versions of the agent (plus answers with no version) and of the scorer, and counts the rest in agent_versions_omitted and scorer_versions_omitted. A card not narrowed to one agent has no agent_versions: name agent_id to compare versions. Flags other than is_truncated appear only when true. full: true returns each card whole, with every day and every version; off by default, a compact card is about a sixth of the size.
Did a fix work? Name the agent and compare each card’s agent_versions: the fixed version’s pass_rate (or score_statistics) against the version before, with scorer_version pinned so a scorer edit does not mix in. A version with few answers is noise: weigh answer_count before reading its rate. include_counted adds a counted block: the live version’s own answers, conversation counts, and confidence.
What it counts. Each trace’s latest answer per scorer and question (runs: latest, the default); latest_per_version keeps one per scorer version, and all counts every run. execution_id counts one score run and defaults runs to all. grain: session counts each agent session’s answer from session-grain scoring instead of each trace’s (the default, grain: trace). pass_rate counts the stored verdicts and does not apply the scorer’s failure rule. Counts can lag new answers by about a minute. An incomplete card says so (is_truncated) and has a null pass_rate, never a partial ratio.
The answers behind a card. Call extension__eval__list_insights with the same workspace_id, scorer_id, and agent_id.
The read needs read access to the workspace (viewers included) and spends the per-minute insights budget it shares with extension__eval__list_insights; after a 429 rate_limited, wait up to a minute and retry. The whole aggregation is bounded at 30 seconds and returns 503 when it cannot finish in time.