Skip to main content
These are errors from eval runs, result uploads, application scores, the CI gate, and hosted scoring. Start with the exception Eval raised, the error field on each entry of run.results, or the CI verdict: each section below starts from one of them.

reuses a Registry name with a different identity

Eval stops while it resolves a scorer, usually on the second run of an eval.

Cause

Two causes produce this message. Tell them apart by your SDK version:
  • SDK 0.3.4 or earlier: The SDK misreads scorers that earlier runs created, so every second run fails.
  • Current SDK: The workspace already has a scorer with the generated name (the eval name and scorer name, lowercased and joined by -) but a different kind, such as a hosted judge.

Solution

Upgrade first, then rename only if the error remains.
  1. Check the version with python -c "import atlanai; print(atlanai.__version__)".
  2. If it is 0.3.4 or earlier, upgrade with pip install -U 'atlanai[tracing]' or npm install @atlanai/sdk@latest.
  3. If the error remains, rename the scorer, or set registry_name (registryName in TypeScript) on it.
  4. Run the eval again and confirm it completes.

Eval() cannot run inside an event loop

Python Eval raises before it runs any case.

Cause

Python Eval starts its own event loop, so it fails when called from Jupyter, an async framework, or an async test.

Solution

Use the async entry point.
  1. Replace Eval(...) with await async_eval(...), with the same arguments.
  2. Run the cell or test again and confirm it returns a result.

Empty summary.scores

The stored experiment is completed, but summary.scores is {} and the SDK reports that the experiment completed with no usable score.

Cause

Every task failed, every scorer failed, or every scorer abstained. The SDK requires a usable score by default, so it raises after storing the completed experiment.

Solution

Find the task error.
  1. Read the experiment using the ID on the exception, then inspect its stored result errors.
  2. Fix failed tasks or scorers and run again. If every scorer abstained, check whether the suite includes any applicable cases.
  3. Confirm the new run has a usable score. Set require_scores=False (requireScores: false) only when an empty score set is intentional.

Score missing from some cases

A score has a count lower than the number of cases.

Cause

The scorer returned None (not applicable) or raised on those cases.

Solution

Tell an abstention from a failure.
  1. Read the error of a case without the score. A raise appears as "<scorer>: <message>".
  2. If there is no error, the scorer returned None on purpose. Check that the rule should not apply to that case.
  3. Fix a raising scorer, run again, and confirm the count matches the cases the rule covers.

has no records, so there is nothing to evaluate

Eval raises after it creates the experiment.

Cause

The dataset has no cases: an empty record-backed dataset, or a CSV file with only a header row.

Solution

  1. Push cases with push_dataset. On a CSV-backed dataset, it appends a new version with them. See Create eval datasets.
  2. The failed call left an experiment running. Close it as in Configure eval runs.
  3. Run again and confirm the cases run.

CSV dataset fails Atlan’s checks

Eval or a read of the dataset fails with a message about the CSV: a missing or duplicate case_id, a missing input_key column, or a row, column, or size limit.

Cause

The dataset’s CSV file failed the checks Atlan Trident applies when it reads it.

Solution

  1. Fix the CSV named in the message.
  2. Upload it as a new dataset version. The limits are in the reference.
  3. Run again and confirm the cases run.

ValueError naming input_key

The run fails on a dataset record.

Cause

A record’s input lacks the dataset’s input_key, which defaults to question.

Solution

Match the records to the key.
  1. Store the question under the dataset’s input_key, or create the dataset with input_key="...".
  2. Push the records again and confirm the run passes that case.

Experiment stays running after run raises

The run raised, and the experiment is still running.

Cause

A trace did not arrive in time, a result upload was refused, or the process died.

Solution

Resume the run, or close it.
  1. Copy the experiment ID from the error message.
  2. Resume with the same arguments, as in Configure eval runs. For slow trace export, raise trace_verification_timeout_s.
  3. If you will not resume it, mark it failed as described on the same page.
  4. Confirm the experiment is completed or failed.

AtlanAPIError during trace verification

Eval raises an AtlanAPIError during a run, while it verifies a case trace before uploading its result.

Cause

Before uploading a result, Eval reads the case trace to confirm it is queryable. A read that fails with traces_unavailable (503) or request_timeout (504) is retried inside the verification window, respecting the server’s Retry-After when one is sent, so a slow or down trace plane fails the run only when the window is spent. Any other read error fails the run at that read.

Solution

Resume the run, or close it.
  1. Check the status against Errors and retries.
  2. Resume with the same arguments once the service is healthy, as in Configure eval runs. The experiment is left running, and cases with a stored result are skipped. If the trace plane is persistently slow, raise trace_verification_timeout_s (traceVerificationTimeoutMs) to widen the window.
  3. If you will not resume it, mark it failed as described on the same page.
  4. Confirm the experiment is completed or failed.

record_push_failed from push_dataset

push_dataset fails with 500 and record_push_failed.

Cause

A transient write failure.

Solution

Retry the push.
  1. Call push_dataset again with the same records. It is idempotent: records already stored come back unchanged.
  2. Confirm every record’s action is created, updated, or unchanged.

Eval refuses to start, citing sampling or content

Eval raises before the first case and names sampling or content.

Cause

Eval traces must be complete. Your logger samples spans or drops content, or ATLAN_TRACING or ATLAN_TRACE_CONTENT is false.

Solution

Give Eval a complete logger.
  1. Unset ATLAN_TRACING and ATLAN_TRACE_CONTENT, or set them to true.
  2. Pass a logger with sample rate 1, or let Eval create one.
  3. Run again and confirm the first case starts.

Two scorers own same score name

The run fails because two scorers return the same score name.

Cause

Every score name belongs to one scorer. Two functions return the same name.

Solution

Rename one.
  1. Change the score name one of the functions returns.
  2. Run again and confirm both scores appear in summary.scores.

Baseline and candidate have different scorer IDs

Two runs you meant to compare cite different scorers.

Cause

They ran under different Eval names. Scorers are registered under the eval name.

Solution

Use one eval name.
  1. Give both runs the same eval name, and put the variant in config.
  2. Run both again and confirm summary.scorers names the same scorer ID.

409 when creating experiment

Creating an experiment returns 409.

Cause

Experiment names are unique per workspace.

Solution

Use a unique name.
  1. Eval makes names unique for you. With start_experiment, add a timestamp or run ID to the name.
  2. Create the experiment again and confirm it returns 201.

400 on result mentioning trace_id

A result upload is refused and the error names trace_id.

Cause

trace_id must be 32 lowercase hex characters, not all zeros.

Solution

Read the ID from the trace.
  1. Take it from the case’s root span: root.trace_id in Python, root.traceId in TypeScript.
  2. Upload again and confirm the item returns a 2xx status.

400 on result mentioning dataset_record_id

A result upload is refused and the error names dataset_record_id.

Cause

Record-backed runs require dataset_record_id, and it must be in the pinned snapshot. Inline runs and CSV runs must omit it.

Solution

Send the right identifier for the dataset kind.
  1. For a record-backed run, use experiment.dataset_snapshot.records[i].id.
  2. For a CSV run, send case_id only.
  3. Upload again and confirm the item returns a 2xx status.

400 when adding a dataset record

Adding a record to a dataset is refused with 400, and the error says the record workspace must match the dataset’s workspace.

Cause

A record belongs to the dataset’s own workspace. The record’s workspace_id was missing, empty, or named a different workspace. A bulk create checks each item on its own, so one wrong item is refused without failing the batch.

Solution

Send the dataset’s workspace.
  1. Read the dataset’s workspace from GET /eval/v1/datasets/{id} and use it as the record’s workspace_id.
  2. Send the create again and confirm it returns 201 (or a 2xx for each item on the bulk create).

409 on result

A result upload returns 409.

Cause

The experiment is already completed or failed, or the case_id was already uploaded. The response’s code says which: duplicate_case_id when the case already has a result, with the case named in the detail (case_id 'case-7' already has a result in this experiment.), and conflict otherwise.

Solution

Check which case applies.
  1. If the experiment is finished, it is frozen: start a new experiment.
  2. If the case_id was already uploaded (duplicate_case_id), for example on a retry, the earlier result is stored. Ignore the 409.
  3. List the experiment’s results and confirm one result per case.

207 with some items refused

A bulk upload returns 207.

Cause

Bulk creates validate each item on its own.

Solution

Read every item.
  1. Check each item’s status_code and error. Do not assume the batch succeeded.
  2. Fix and resend only the refused items.
  3. Confirm every item now returns a 2xx status.

Result input or output reads as {"value": ...}

A string or number your task returned comes back wrapped.

Cause

Results store objects. A string or number is wrapped as {"value": x}.

Solution

Unwrap the value when you read results back.
  1. Read result.output["value"] when the output was a string or number.
  2. Confirm your comparison sees the raw value.

verify_experiment reports orphaned spans

The check finds spans that do not belong to a case.

Cause

The case span was opened with an invented trace_context.

Solution

Let the SDK choose the trace ID.
  1. Remove the invented trace_context, and read the trace ID from the span.
  2. Run again and confirm verify_experiment(client, experiment_id).raise_for_status() passes.

Score never appears on trace

A score from application code is missing from the trace.

Cause

The score had no valid scorer_id and scorer_version, an empty name, or a value that does not match its data_type. Scores are dropped with a warning, never raised.

Solution

Read the warning.
  1. Set ATLAN_DEBUG=true and run again. The warning names what is missing.
  2. Fix the scorer ID, version, name, or value.
  3. Flush and confirm a score.<name> span appears on the trace.

Categorical score has no aggregate

Trace statistics show no mean for a categorical score.

Cause

Only numeric values are aggregated.

Solution

Send a number with the label.
  1. Pass a numeric value alongside string_value.
  2. Confirm the score’s aggregate appears in trace statistics.

invalid verdict with no experiments reported

The CI check fails with invalid and says no experiments were reported.

Cause

The command did not call Eval, or a TypeScript eval did not write the results file.

Solution

Report the experiment to the action.
  1. Use Python Eval, or add the results file to the TypeScript eval, as in Gate pull requests with evals.
  2. Push again and confirm the comment lists the experiment.

invalid verdict with missing ATLAN_API_KEY in log

The CI check fails with invalid, and the command’s log shows a missing ATLAN_API_KEY or workspace.

Cause

The action does not pass its api-key input to the command.

Solution

Give the command its own credentials.
  1. Set ATLAN_API_KEY and ATLAN_WORKSPACE_ID in the action step’s env, as in the workflow.
  2. Push again and confirm the command runs.

Required eval check stays pending

A pull request cannot merge because the eval check never reports.

Cause

A paths: filter skipped the workflow, so the required check never ran.

Solution

Run the workflow on every pull request.
  1. Remove the paths: filter from pull_request. To skip unrelated pull requests, test the paths inside the job. See the note in the workflow.
  2. Push again and confirm the check reports.

invalid verdict with no git provenance

The CI check fails with invalid, saying the run has no git provenance, or is not from this commit or pull request.

Cause

config.git is missing, or was set by hand.

Solution

Let the SDK record the commit.
  1. Remove any config.git you set yourself. The TypeScript results file is the exception: it copies the action’s values.
  2. Push again and confirm the verdict is no longer invalid.

invalid verdict for error rate

The CI check fails with invalid because too many cases errored.

Cause

More than max-error-rate (default 20%) of cases errored.

Solution

Fix the failing cases.
  1. Read the error count in the pull request comment, and the errors on the experiment in the app.
  2. Fix the task or scorers, push again, and confirm the verdict is pass or regression.

no-baseline on every pull request

Every pull request passes with a no-baseline warning.

Cause

There is no completed run on the default branch with the same eval name and dataset.

Solution

Create the baseline.
  1. Run the workflow on main once.
  2. Keep the eval name identical across branches.
  3. Push to a pull request and confirm the comment compares with the baseline.

Regression you cannot reproduce

The gate reports a regression that does not repeat.

Cause

Run-to-run variance is larger than the threshold.

Solution

Make the gate tolerate noise.
  1. Raise min-regressed-cases, widen thresholds, or add cases.
  2. Set model temperature to 0 where your agent allows it.
  3. Rerun the check and confirm the verdict is stable.

No API key on fork pull requests

The job fails on pull requests from forks with no API key.

Cause

GitHub does not pass secrets to forks.

Solution

Skip fork pull requests.
  1. Add the if: condition shown in the workflow.
  2. Confirm fork pull requests skip the job.

503 from /score

Every scoring call, including preview, returns 503.

Cause

No judge backend is configured on the deployment, or a selected session could not be read at the moment of the call.

Solution

Find out which one it is.
  1. Ask your administrator whether a judge backend is configured on the deployment; if not, ask them to configure one.
  2. If a judge backend is configured, send the request again and confirm it returns results.

403 from /score

A scoring call returns 403.

Cause

  • You do not have the builder or admin role in the scorer’s workspace.
  • Or your subscription does not include scoring.

Solution

Get access to scoring.
  1. Request the builder or admin role in the scorer’s workspace, or create the scorer in a workspace where you have it.
  2. If you have the role, confirm with your administrator that your subscription includes scoring.
  3. Send a preview request again and confirm it returns results.

400 naming selector

A scoring call returns 400, and its detail names the problem.

Cause

A selector or request option failed validation. The detail says which: two selector families, a session_kind sent without the since it requires, MCP sessions requested at trace grain (they have no trace), no subjects matched the selectors and sample rate, more than 100 session_ids and trace_ids combined, a limit outside 1–100, a sampled selection that resolves more than 1,000 candidates before the sample is chosen, a concurrency outside 1 and the deployment’s ceiling, or more than 5 subjects in preview or test.

Solution

Read the detail, then send one family with options in range.
  1. Send one of: session or trace IDs, a time window, agents, or an experiment.
  2. Fix the field or option the detail names. If the detail says the selection exceeds 1,000 candidates, narrow the time window or selector: limit cannot rescue it, because it is applied to the sample after sampling.
  3. Confirm the call returns a run or results.

409 from /score naming the scorer archived

A scoring call returns 409, and its code is scorer_archived.

Cause

You can read the scorer, but it is archived. Archived scorers cannot run in any mode, including preview and test. This is the response for any selector on that scorer; it is not a 404, because the scorer exists and is visible to you.

Solution

Score with a scorer that can run.
  1. Confirm on the scorer’s page that it is archived, and that the scorer ID in the call is the one you meant.
  2. Select a scorer that is not archived, or create a new hosted judge. See Write hosted judges.
  3. Send the call again and confirm it returns a run or results.

409 editing an archived scorer

Editing a scorer returns 409, and its code is scorer_archived.

Cause

The scorer is archived, and the edit changes a definition value. Archived scorers cannot be edited. An edit that changes nothing in the definition — renaming the scorer, or re-describing it — still lands, and so does one that re-sends the definition unchanged.

Solution

Change a scorer that is not archived.
  1. If the definition should change, create a new scorer with it. For a hosted judge, see Write hosted judges.
  2. If the edit was a rename or a re-description, send it again without definition fields, and confirm it answers 200.

400 reading stored answers

A read of stored answers with GET /eval/v1/insights returns 400, and the detail names a filter rule.

Cause

The read validates its filters together. The query breaks one of them: it names none of scorer_id, trace_id, session_id, agent_id, or execution_id; score_min or score_max is not a finite number; score_min is greater than score_max; is_score_missing=true is sent with score_min or score_max; is_agent_version_missing=true is sent with agent_version; or offset is outside 0 to 10,000. The answer filters have their own rules: choice and passed cannot be combined; choice needs a single-choice scorer (a single-choice question or a rubric) and an option it offers, spelled as stored; and passed needs a scorer whose answers carry a verdict — a boolean question on the llm backend, or a classification rubric, where passed selects the pass or fail choice. A classifier boolean stores P(true) in score and no verdict, so passed is refused there: filter it with score_min/score_max instead. Each filter is checked against the scorer’s newest saved definition first; when that would refuse, it is checked against the scorer_version the query pins, or else against the scorer’s earlier saved versions, newest first. A filter any checked version answers is accepted, one that no checked version answers is refused, and with more than 20 saved versions the filter is sent unchecked.

Solution

Follow the detail.
  1. The detail states the rule the query broke, for example “score_min must not exceed score_max” or “choice is not an option of this scorer; GET /scorers/{scorer_id} lists its options.”
  2. Fix the query and send it again. The filters and paging are in Score production sessions.
  3. Confirm the call returns answers.

Session line says nothing to judge

A session’s result line says there is nothing to judge.

Cause

The session has no readable trace.

Solution

Join the session to its trace.
  1. Check that the trace is ingested and belongs to the scorer’s workspace.
  2. Confirm client.sessions.traces.list(session_id) lists it. See Make app traffic scoreable.
  3. Score the session again.

Line says the trace has no readable session

A result line fails with “trace has no readable session; it cannot be scored”.

Cause

The trace’s spans carry no session_id, or the session it names is not readable to you in the scorer’s workspace. A trace you can read without a session is reported as its own error line; the call’s other subjects are still judged. Naming a trace that is absent or unreadable is different: that refuses the whole request with 404.

Solution

Give the trace a session, then score it again.
  1. Set a session_id on the trace’s spans and record the session when the conversation ends. See Make app traffic scoreable.
  2. Confirm client.sessions.traces.list(session.id) lists the trace.
  3. Score the trace again and confirm the line has an answer.

Model gateway returned 404

A session’s result line says the model gateway returned 404. When the gateway’s response names a reason, the line names it too: the gateway’s error code and, when the response carries one, the trace ID of the rejected call, as in the model gateway returned HTTP 400 (invalid_request; trace_id 4bf92f3577b34da6a3ce929d0e0e4736). A rejection whose response names no reason says only the status. The line never quotes the gateway’s message text, because lines are stored with your answers.

Cause

The deployment’s judge model is not registered in your organization’s model catalog.

Solution

Register the judge model.
  1. Ask your administrator to register the deployment’s judge model in your organization’s model catalog, quoting the line’s error code and trace ID so they can find the rejected call.
  2. Score the session again and confirm the line has an answer.

Line fails on context length

A session’s result line fails on context length.

Cause

The transcript is larger than the judge model’s context. Transcripts are not truncated.

Solution

Send less to the judge.
  1. Use grain: "trace" rather than session, or narrow state to the fields that matter.
  2. Score the session again and confirm the line has an answer.

Line fails on the judge input limit

A result line fails with the Jev state and questions exceed the input limit, followed by input_chars and max_input_chars.

Cause

The request to the classifier backend, the scorer’s questions together with the state evidence, is larger than the classifier’s input ceiling. input_chars is the size of the complete request in UTF-8 bytes, JSON escaping included; max_input_chars is the ceiling. Evidence is not truncated, and the call is not sent to another backend instead. A judge model that refuses a transcript on its context is the context length failure.

Solution

Send less evidence to the classifier.
  1. Compare input_chars with max_input_chars to see how much to cut.
  2. Narrow state to the fields the verdict needs, and trim the criteria on choice questions: both are part of the request.
  3. Judge a single run rather than a whole session where you can: grain: "trace" reads less than session.
  4. Preview the scorer again and confirm the line carries a preview_request.

experiment_id selector matches nothing

Scoring by experiment_id finds no sessions.

Cause

Results written by Eval carry no session.

Solution

Select the sessions another way.
  1. Select the sessions by ID or by time window instead.
  2. Confirm the call returns a run.

New sessions not scored by Run live

Sessions finish, but automatic scoring does not judge them.

Cause

Only sessions that reach completed or failed are scored automatically.

Solution

Record each session’s end.
  1. Make sure your agent sets the session’s status to completed or failed when it ends.
  2. Score earlier sessions on demand by ID.
  3. Confirm new answers appear on the scorer’s page.

Low confidence for confident answer

An answer looks confident, but its confidence is low.

Cause

confidence is chance-corrected, not a probability.

Solution

Read the probability instead.
  1. Read probabilities[choice]. See the answer fields in Write hosted judges. Through MCP, extension__eval__list_insights returns probabilities only when full is true.
  2. Confirm the probability matches the answer’s strength.

400 or 404 about judge on scorer create or edit

Creating or editing a hosted judge returns 400 or 404 about its judge. An edit’s detail names the field, such as Field 'model' is not valid.

Cause

The scorer does not name a valid backend and model, its workspace cannot use the model, a system message contains a template such as {{thread}}, or a question that only the llm backend can answer is paired with classifier: a text question, or a multi-select choice question. The error says so, for example Text questions need the LLM judge: the Jev classifier answers only typed options. Set backend to llm, or ask a choice, score or boolean question. A text question with a criteria value is refused too: criteria must be absent or null for a text question. A system message must be static, and the error says so: System messages must be static; move {{variables}} and template sections to user or assistant messages.

Solution

  1. Set backend to classifier or llm. A text or multi-select question needs llm.
  2. Set model to the qualified model id (<provider>/<model>) of an enabled model the scorer’s workspace can read. See Name judge.
  3. Move any template from a system message to a user or assistant message. See Write judge definition for what each turn may hold.
  4. Create or edit the scorer again, and confirm the response returns the scorer with a new version.
A scorer saved with a template in its system message before this check existed keeps failing each session it scores with the same message until an edit moves that template into a user or assistant message.

409 publishing a version that is already live

Publishing with POST /eval/v1/scorers/{id}/publish returns 409, and its code is release_label_not_increasing or version_already_released.

Cause

The version being published is the one already live, and the request names a version_label other than the live one. Publishing the live version changes nothing and answers 200 when the request omits version_label or names the live one. Another label is refused: one that does not sort above the live label is release_label_not_increasing, and one that sorts above it is version_already_released, because it would re-release the version that already shipped under the live label.

Solution

Ship a new label with a new version.
  1. To leave the live release as it is, publish again without version_label, or with the live one, and confirm it answers 200.
  2. To release under a new label, save a new draft of the definition you want and publish that draft. See Write hosted judges.

Line fails on a message template limit

A scoring run’s session line fails with messages templates must total at most 64 KiB. or messages templates may nest at most 8 sections.

Cause

The judge’s messages templates exceed what rendering allows: 64 KiB across the scorer’s messages, and at most 8 nested {{#x}} sections. The check applies when the scorer is created or edited, and also when scoring runs a version saved before the limits existed — such a version fails its line instead of rendering the oversized template.

Solution

Trim or flatten the judge’s messages.
  1. Edit the scorer to add a new version that drops turns the verdict does not need, or flattens nested {{#x}}…{{/x}} sections. See Write hosted judges.
  2. Score the sessions again and confirm the line has an answer.

See also

Need help

If you need assistance after trying these steps, contact Atlan support.