> ## Documentation Index > Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-2772-1782277917413.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Saving Eval Results > API reference for saving eval results to MCPJam Reporting authenticates with **MCPJam API keys (`sk_…`)** from Settings → API keys and uploads through the [MCPJam public API](/reference/public-api) (`/api/v1/projects/:projectId/eval-ingest/*`). The legacy project API keys (`mcpjam_…`) and their `/sdk/v1/evals/*` endpoints are retired — old SDK versions pointing there receive `410 Gone`. Upgrade `@mcpjam/sdk` to keep uploading. The SDK provides APIs to save eval results to MCPJam for visualization in the CI Evals dashboard. Results can be saved automatically via `EvalTest`/`EvalSuite`, or manually using the APIs below. ## Environment Variables | Variable | Required | Default | Description | | ------------------- | -------- | ------------------------ | --------------------------------------------------- | | `MCPJAM_API_KEY` | Yes | - | An MCPJam API key (`sk_…`) from Settings → API keys | | `MCPJAM_PROJECT_ID` | No | org's Default project | Project the results are filed under | | `MCPJAM_BASE_URL` | No | `https://app.mcpjam.com` | MCPJam API base URL override | Use `MCPJAM_BASE_URL` only when you need to override the default ingest host, such as internal development against a non-production backend. `MCPJAM_API_KEY` controls whether results are uploaded. **Replay credential** capture only happens when you provide `serverReplayConfigs`, `agent`, or `mcpClientManager`. **MCP App widget snapshots** in each result’s trace (for iframe replay in MCPJam) come from [`PromptResult.getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots), which is only populated when [`HostRunner`](/sdk/reference/host-runner) was constructed with [`mcpClientManager`](/sdk/reference/host-runner#parameters). *** ## Tool execution and `passed` Uploaded iterations have a single boolean `passed` that drives pass rate on the dashboard. The SDK distinguishes: 1. **Structural pass** — Your test returned `true`, or expected tool calls were satisfied (Inspector/UI flows use shared matching logic). 2. **Tool execution** — The trace shows a real tool failure: MCP results with `isError: true`, timeline spans where a tool step ended in `error`, tool-result parts the UI would treat as errors, or a runner-level `iterationError` after a thrown tool step. **Default behavior:** When the SDK **derives** `passed` from a trace (for example `EvalTest` / `EvalSuite` auto-save, [`PromptResult.toEvalResult()`](/sdk/reference/prompt-result#toevalresult), `createEvalRunReporter` helpers, or Inspector suite runs), structural success is **not enough** if tool execution failed — the iteration is recorded as **failed** unless you opt out. **Opt out:** set `failOnToolError: false` on [`MCPJamReportingConfig`](#mcpjamreportingconfig) (global for that reporter or auto-save), or pass it on specific helper options (`addFromPrompt` / `recordFromRun` / `RunToEvalResultsOptions`, etc.). Use this when you only care that the model *invoked* the right tools with the right arguments, not that every MCP call returned a success payload. **Manual `reportEvalResults`:** If you build `results[]` yourself and set `passed` explicitly, MCPJam stores your values as-is. The execution gate applies to code paths that **compute** `passed` from prompts, traces, and iterations. **Programmatic reuse:** `@mcpjam/sdk` also exports `finalizePassedForEval`, `traceIndicatesToolExecutionFailure`, `isCallToolResultError`, `traceMessagePartIndicatesToolFailure`, `classifyToolFailurePart`, and `extractToolErrors` for custom ingestion pipelines. The `@mcpjam/sdk/predicates` sub-package exports the full predicate evaluator (`evaluatePredicate`, `evaluatePredicates`, `allPredicatesPassed`, `buildIterationTranscript`) and all predicate types. *** ## reportEvalResults() One-shot reporting. Sends all results in a single call. Throws on failure. ```typescript theme={null} import { MCPClientManager, reportEvalResults } from "@mcpjam/sdk"; ``` ### Signature ```typescript theme={null} reportEvalResults(input: ReportEvalResultsInput): Promise ``` ### Example ```typescript theme={null} const manager = new MCPClientManager({ asana: { url: process.env.MCP_SERVER_URL!, refreshToken: process.env.MCP_REFRESH_TOKEN!, clientId: process.env.MCP_CLIENT_ID!, clientSecret: process.env.MCP_CLIENT_SECRET, }, }); await manager.connectToServer("asana"); const output = await reportEvalResults({ suiteName: "Nightly", mcpClientManager: manager, results: [ { caseTitle: "healthcheck", passed: true }, { caseTitle: "tool-selection", passed: true, durationMs: 1200 }, { caseTitle: "edge-case", passed: false, error: "Wrong tool called" }, ], passCriteria: { minimumPassRate: 90 }, ci: { branch: "main", commitSha: "abc123", }, }); console.log(`Run ${output.runId}: ${output.result}`); // "Run abc123: passed" console.log(`${output.summary.passed}/${output.summary.total} passed`); ``` *** ## reportEvalResultsSafely() Same as `reportEvalResults()`, but returns `null` instead of throwing on failure. Warnings are logged to the console. ```typescript theme={null} import { MCPClientManager, reportEvalResultsSafely } from "@mcpjam/sdk"; ``` ### Signature ```typescript theme={null} reportEvalResultsSafely(input: ReportEvalResultsInput): Promise ``` ### Example ```typescript theme={null} const manager = new MCPClientManager({ asana: { url: process.env.MCP_SERVER_URL!, refreshToken: process.env.MCP_REFRESH_TOKEN!, clientId: process.env.MCP_CLIENT_ID!, }, }); await manager.connectToServer("asana"); const output = await reportEvalResultsSafely({ suiteName: "Nightly", mcpClientManager: manager, results: [{ caseTitle: "healthcheck", passed: true }], }); if (output) { console.log(`Reported: ${output.summary.passRate * 100}% pass rate`); } else { console.log("Reporting failed (non-blocking)"); } ``` Use `reportEvalResultsSafely()` when you don't want eval reporting failures to break your CI pipeline. Use `reportEvalResults()` (strict) when reporting is critical. *** ## createEvalRunReporter() Creates an incremental reporter for long-running processes. Results are buffered and flushed in batches (up to 200 results or 1MB per batch). ```typescript theme={null} import { createEvalRunReporter } from "@mcpjam/sdk"; ``` ### Signature ```typescript theme={null} createEvalRunReporter(input: CreateEvalRunReporterInput): EvalRunReporter ``` `CreateEvalRunReporterInput` accepts the same replay source fields as `reportEvalResults()`: `serverReplayConfigs`, `agent`, and `mcpClientManager`. ### EvalRunReporter Methods | Method | Description | | ------------------------------ | --------------------------------------------------- | | `add(result)` | Buffer a result (no network call) | | `record(result)` | Buffer a result and auto-flush when buffer is large | | `flush()` | Upload all buffered results | | `finalize()` | Flush remaining results and finalize the run | | `getBufferedCount()` | Number of results in the buffer | | `getAddedCount()` | Total results added (including flushed) | | `setExpectedIterations(count)` | Set expected iteration count for progress tracking | #### PromptResult Helpers | Method | Description | | -------------------------------------------- | --------------------------------------------------- | | `addFromPrompt(promptResult, overrides?)` | Convert a `PromptResult` and buffer it | | `recordFromPrompt(promptResult, overrides?)` | Convert a `PromptResult`, buffer it, and auto-flush | #### EvalTest/EvalSuite Run Helpers | Method | Description | | --------------------------------------- | ---------------------------------------------- | | `addFromRun(run, options)` | Convert all iterations from an `EvalTest` run | | `recordFromRun(run, options)` | Convert and auto-flush from an `EvalTest` run | | `addFromSuiteRun(suiteRun, options)` | Convert all iterations from an `EvalSuite` run | | `recordFromSuiteRun(suiteRun, options)` | Convert and auto-flush from an `EvalSuite` run | ### Example ```typescript theme={null} // Assumes `agent` was created with `mcpClientManager: manager` const reporter = createEvalRunReporter({ suiteName: "Integration Tests", passCriteria: { minimumPassRate: 85 }, agent, ci: { branch: process.env.GITHUB_REF_NAME, commitSha: process.env.GITHUB_SHA, }, }); ``` ### Example with manual replay source resolution ```typescript theme={null} const reporter = createEvalRunReporter({ suiteName: "Integration Tests", passCriteria: { minimumPassRate: 85 }, mcpClientManager: manager, }); // Add results as tests complete await reporter.record({ caseTitle: "test-1", passed: true, durationMs: 500 }); await reporter.record({ caseTitle: "test-2", passed: false, error: "timeout" }); await reporter.record({ caseTitle: "test-3", passed: true }); // Finalize the run const output = await reporter.finalize(); console.log(`${output.summary.passed}/${output.summary.total} passed`); ``` ### Replay credential sources Authenticated HTTP evals can securely persist replay credentials for reruns and debugging. Manual reporting APIs resolve replay configs in this order: 1. `serverReplayConfigs` 2. `agent.getServerReplayConfigs()` 3. `mcpClientManager.getServerReplayConfigs()` Prefer passing `agent` or `mcpClientManager` directly. Use `serverReplayConfigs` only when you need a low-level override. If you also provide `serverNames`, inferred replay configs are filtered to those server IDs before upload. Explicit `serverReplayConfigs` are left unchanged. ### Replay metadata for the MCPJam UI Uploaded runs can show **Replay this run** / server-side MCP replay when the ingest payload includes derived **`serverReplayConfigs`** (stored as **`hasServerReplayConfig`** on the run). In practice: * **HTTP MCP (`url`)** — Replay configs are built for typical streamable HTTP connections. **Stdio** transports **do not** produce entries from [`MCPClientManager.getServerReplayConfigs()`](/sdk/reference/mcp-client-manager); use HTTP when you need dashboard replay. * **`HostRunner` vs reporter** — Putting `mcpClientManager` on [`HostRunner`](/sdk/reference/host-runner) fills **MCP App widget snapshots** on [`PromptResult`](/sdk/reference/prompt-result). The **reporter** (and one-shot `report*`) resolves **server replay** from its **own** `agent` / `mcpClientManager` fields. Pass `agent` or `mcpClientManager` into [`createEvalRunReporter`](#createevalrunreporter) as well; **agent-only** wiring can still upload traces and widgets but **omit** `hasServerReplayConfig`. * **Teardown order** — `finalize()` / one-shot reporting calls `getServerReplayConfigs()` against **connected** registrations. In `afterAll`, run **`await reporter.finalize()`** (or `reportEvalResults`) **before** **`await manager.disconnectAllServers()`**. Disconnecting first clears manager state and uploads **without** replay metadata. **LLM API keys** are **not** stored on the run. Replaying in the MCPJam UI still requires provider keys (e.g. OpenRouter) in **Settings** for your suite’s models. ### Using with PromptResult ```typescript theme={null} // Pass `agent` or `mcpClientManager` when you need server replay metadata in MCPJam const reporter = createEvalRunReporter({ suiteName: "Prompt Tests", agent }); const result = await agent.run("Add 2 and 3"); reporter.addFromPrompt(result, { caseTitle: "addition", passed: result.hasToolCall("add"), }); const output = await reporter.finalize(); ``` ### Using with EvalTest Runs ```typescript theme={null} const reporter = createEvalRunReporter({ suiteName: "Full Suite" }); const test = new EvalTest({ name: "addition", test: async (agent) => (await agent.run("Add 2+3")).hasToolCall("add"), }); const run = await test.run(agent, { iterations: 10 }); await reporter.recordFromRun(run, { casePrefix: "addition" }); const output = await reporter.finalize(); ``` *** ## uploadEvalArtifact() Parses test artifacts (JUnit XML, Jest JSON, Vitest JSON) and reports the results to MCPJam. ```typescript theme={null} import { uploadEvalArtifact } from "@mcpjam/sdk"; ``` ### Signature ```typescript theme={null} uploadEvalArtifact(input: UploadEvalArtifactInput): Promise ``` ### Supported Formats | Format | Description | | --------------- | --------------------------------------- | | `"junit-xml"` | JUnit XML test reports | | `"jest-json"` | Jest JSON output (`--json` flag) | | `"vitest-json"` | Vitest JSON reporter output | | `"custom"` | Custom parser via `customParser` option | ### Example ```typescript theme={null} import { readFileSync } from "fs"; // Upload JUnit XML await uploadEvalArtifact({ suiteName: "CI Results", format: "junit-xml", artifact: readFileSync("test-results.xml", "utf-8"), }); // Upload Jest JSON await uploadEvalArtifact({ suiteName: "Jest Results", format: "jest-json", artifact: readFileSync("jest-results.json", "utf-8"), }); // Custom parser await uploadEvalArtifact({ suiteName: "Custom", format: "custom", artifact: myData, customParser: (data) => [ { caseTitle: "test-1", passed: true }, { caseTitle: "test-2", passed: false, error: "failed" }, ], }); ``` *** ## Types ### ReportEvalResultsInput ```typescript theme={null} type ReportEvalResultsInput = MCPJamReportingConfig & { suiteName: string; results: EvalResultInput[]; agent?: { getServerReplayConfigs?: () => MCPServerReplayConfig[] | undefined; }; mcpClientManager?: MCPClientManager; }; ``` ### MCPJamReportingConfig | Property | Type | Required | Description | | --------------------- | ------------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `boolean` | No | Enable/disable reporting (default: `true`) | | `apiKey` | `string` | No | MCPJam API key (`sk_…`; falls back to `MCPJAM_API_KEY` env var) | | `baseUrl` | `string` | No | MCPJam API base URL override (useful for internal development or tests) | | `project` | `string` | No | Project id results are filed under (falls back to `MCPJAM_PROJECT_ID`, then the org's Default project) | | `suiteName` | `string` | No | Suite name for the run | | `suiteDescription` | `string` | No | Description of the suite | | `serverNames` | `string[]` | No | MCP server names being tested | | `serverReplayConfigs` | `MCPServerReplayConfig[]` | No | Advanced override for replay credential capture | | `notes` | `string` | No | Free-form notes | | `passCriteria` | `{ minimumPassRate: number }` | No | Pass threshold (0-100) | | `failOnToolError` | `boolean` | No | When not `false`, results derived from traces treat tool execution failures as failed iterations (default: strict). See [Tool execution and `passed`](#tool-execution-and-passed) | | `strict` | `boolean` | No | Throw on upload errors (`false` = warn only) | | `externalRunId` | `string` | No | Custom run ID (auto-generated if omitted) | | `framework` | `string` | No | Test framework name (e.g., `"jest"`, `"vitest"`) | | `ci` | `EvalCiMetadata` | No | CI/CD pipeline context | | `expectedIterations` | `number` | No | Expected total iterations for progress tracking | | `tags` | `string[]` | No | Free-form tags attached to the run for filtering in the MCPJam dashboard | | `agent` | `{ getServerReplayConfigs?: () => MCPServerReplayConfig[] \| undefined }` | No | Preferred replay source for manual reporting; use an executor (HostRunner / HostRuntime) created with `mcpClientManager` so results can include [`widgetSnapshots`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) from [`PromptResult.toEvalResult()`](/sdk/reference/prompt-result). Field name predates the Stage 4 rename; any `HostExecutor` is accepted. | | `mcpClientManager` | `MCPClientManager` | No | Replay source when no `agent` is provided; does not populate `widgetSnapshots` unless your `results[].widgetSnapshots` or trace payloads already include them | | `host` | `Host` | No | Explicit host snapshot override. The reporter prefers `iteration.hostSnapshot` (per-iteration capture from `HostRuntime`) and falls back to `executor.getHostSnapshot?.()`; supply `host` only when neither source is available. See [Run-level host snapshot](#run-level-host-snapshot) below. | ### Run-level host snapshot When a usable host snapshot is available, the reporter includes a normalized + content-hashed host config alongside the run body (the v1 ingest surface always accepts the pair — the old per-`baseUrl` capability probe is gone): ```jsonc theme={null} POST /api/v1/projects/{projectId}/eval-ingest/runs/start { "suiteName": "...", "results": [...], "hostConfig": {/* canonical HostConfigInputV2 with serverIds stripped */}, "hostConfigHash": "sha256(canonical)" } ``` **Source order (highest priority first):** 1. `iteration.hostSnapshot` — per-iteration capture from `HostRuntime` (Stage 4). Reflects the bound `Host` state at the iteration's end. 2. `executor.getHostSnapshot?.()` — fallback for executors that don't expose per-iteration snapshots. 3. `MCPJamReportingConfig.host` — explicit override, compatibility path. **Pass-1 homogeneity gate.** The reporter only sends a run-level `{ hostConfig, hostConfigHash }` when all available iteration snapshots canonicalize to the same hash. Heterogeneous runs (e.g. mutating the bound `Host` between iterations) omit the run-level field — per-iteration wire support is a later stage. **Fail-safe omission.** Any error while resolving or canonicalizing the snapshot — malformed `hostSnapshot`, an executor that throws, non-canonicalizable host JSON — logs a warning and omits the wire pair rather than failing the eval upload. The body shape stays the same without it. **Server-id normalization.** Runtime-manager identifiers (`serverIds`, `optionalServerIds`, `serverConnectionOverrides`) are stripped by `normalizeSdkEvalHostConfigForWire` on both ends so SDK runtime ids like `"everything"` never reach the backend's `validateServerScope` as `Id<'servers'>`. The persisted `hostConfigsV2` row's hash will differ from the wire `hostConfigHash` because suite-resolved Convex server ids are layered on top at storage time — the wire hash is a transport-integrity check, not the storage id. ### MCPServerReplayConfig Advanced replay override. Most users should not construct this manually. | Property | Type | Required | Description | | -------------- | --------- | -------- | ------------------------------------------------- | | `serverId` | `string` | Yes | MCP server identifier | | `url` | `string` | Yes | MCP server URL | | `preferSSE` | `boolean` | No | Prefer SSE transport for replay | | `accessToken` | `string` | No | Static bearer token for replay | | `refreshToken` | `string` | No | Refresh token for replay | | `clientId` | `string` | No | OAuth client ID, required with `refreshToken` | | `clientSecret` | `string` | No | OAuth client secret when needed for token refresh | ### EvalCiMetadata | Property | Type | Description | | ------------ | -------- | ------------------------------------------ | | `provider` | `string` | CI provider (e.g., `"github"`, `"gitlab"`) | | `pipelineId` | `string` | Pipeline/workflow identifier | | `jobId` | `string` | Job identifier | | `runUrl` | `string` | URL to the CI run | | `branch` | `string` | Git branch name | | `commitSha` | `string` | Git commit SHA | ### EvalResultInput | Property | Type | Required | Description | | --------------------- | ----------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `caseTitle` | `string` | Yes | Test case title | | `passed` | `boolean` | Yes | Whether the test passed | | `query` | `string` | No | The prompt/query sent | | `durationMs` | `number` | No | Test duration in ms | | `provider` | `string` | No | LLM provider name | | `model` | `string` | No | Model identifier | | `expectedToolCalls` | `EvalExpectedToolCall[]` | No | Expected tool calls | | `actualToolCalls` | `EvalExpectedToolCall[]` | No | Actual tool calls made | | `tokens` | `{ input?, output?, total? }` | No | Token usage | | `error` | `string` | No | Error message | | `errorDetails` | `string` | No | Detailed error info | | `trace` | `EvalTraceInput` | No | Conversation trace | | `externalIterationId` | `string` | No | Custom iteration ID | | `externalCaseId` | `string` | No | Custom case ID | | `metadata` | `Record` | No | Custom metadata. When `successPredicates` are evaluated, the runner persists `{ predicate, passed, reason }[]` under `metadata.predicates` automatically. | | `successPredicates` | `Predicate[]` | No | State-based predicate gate evaluated over the iteration transcript. See [Predicate gate](#predicate-gate). | | `isNegativeTest` | `boolean` | No | Whether this is a negative test | | `matchOptions` | `EvalMatchOptions` | No | Per-result match options. When present, the inspector snapshots these onto the iteration so historical pass/fail computation honors them. See [`EvalMatchOptions`](/sdk/reference/validators#evalmatchoptions) | | `widgetSnapshots` | `EvalWidgetSnapshotInput[]` | No | MCP App HTML replay payloads (typically from [`getWidgetSnapshots()`](/sdk/reference/prompt-result#mcp-app-widget-snapshots) on [`PromptResult`](/sdk/reference/prompt-result)). Omitted when not using MCP Apps or when `HostRunner` had no `mcpClientManager` | ### Predicate gate `successPredicates` adds a deterministic, state-based assertion layer on top of tool-call matching. Each predicate is a pure function of the iteration transcript — same transcript, same verdict — which makes it suitable as a CI release gate. A case passes the predicate gate iff **all** predicates pass. An absent or empty list means no predicate gate (the case is judged solely by tool-call matching and `failOnToolError`). #### Predicate types | Type | Fields | Description | | ------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `toolCalledWith` | `toolName`, `args` ([`ArgMatcher`](#argmatcher)), `minCount?` | A call to `toolName` whose args satisfy `args` occurred at least `minCount` (default 1) times. | | `toolCalledAtLeastOnce` | `toolName` | `toolName` was called at least once (args irrelevant). | | `toolNeverCalled` | `toolName` | `toolName` was never called (forbidden-tool check). | | `responseContains` | `needle`, `caseSensitive?` | The final assistant message contains `needle`. Case-insensitive by default. | | `responseMatches` | `pattern` | The final assistant message matches the regular expression `pattern` (regex source string, no flags). | | `noToolErrors` | — | No tool produced an error (neither MCP `isError: true` nor a JSON-RPC/transport failure). | | `finalAssistantMessageNonEmpty` | — | The final assistant message is a non-empty, non-whitespace string. | | `tokenBudgetUnder` | `tokens` | Total token usage for the iteration is strictly under `tokens`. Fails closed when usage is unavailable. | #### ArgMatcher Used by `toolCalledWith` to specify expected arguments and matching mode. | Property | Type | Description | | ------------------ | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `args` | `Record` | Expected argument shape. | | `argumentMatching` | `"partial" \| "exact" \| "ignore"` | Matching mode. `"partial"` (default) checks only the keys present in `args`; `"exact"` requires deep equality; `"ignore"` skips argument comparison. | #### Fail-closed behavior Malformed predicates (unknown type, missing required fields, invalid `minCount`, empty `needle`/`pattern`) always fail the case rather than silently passing. `responseMatches` also fails closed on patterns with nested quantifiers (ReDoS guard) and on messages exceeding 100,000 characters. #### Example ```typescript theme={null} import type { Predicate } from "@mcpjam/sdk/predicates"; const predicates: Predicate[] = [ // Tool was called with the right airline argument { type: "toolCalledWith", toolName: "book_flight", args: { args: { airline: "DL" } }, }, // Response mentions the booking confirmation { type: "responseContains", needle: "confirmed" }, // No tool errors occurred { type: "noToolErrors" }, // Token budget respected { type: "tokenBudgetUnder", tokens: 2000 }, ]; // Pass predicates via the per-run override in the Inspector corpus, // or use evaluatePredicates() directly in custom pipelines: import { evaluatePredicates, buildIterationTranscript } from "@mcpjam/sdk/predicates"; const transcript = buildIterationTranscript({ trace: myTrace, toolCalls: myToolCalls, usage: { totalTokens: 1200 }, }); const results = evaluatePredicates(transcript, predicates); const allPassed = results.every((r) => r.passed); ``` #### Predicate results in metadata When the Inspector runner evaluates predicates, it persists one row per predicate to `testIteration.metadata.predicates`: ```typescript theme={null} type PredicateResult = { predicate: Predicate; // The authored predicate (sensitive arg keys are redacted) passed: boolean; reason: string; // Structured explanation, e.g. "tool "book_flight" called with matching args" }; ``` *** ### ReportEvalResultsOutput | Property | Type | Description | | ------------------ | ------------------------- | --------------------------- | | `suiteId` | `string` | Created/matched suite ID | | `runId` | `string` | Created run ID | | `status` | `"completed" \| "failed"` | Run status | | `result` | `"passed" \| "failed"` | Pass/fail based on criteria | | `summary.total` | `number` | Total iterations | | `summary.passed` | `number` | Passed iterations | | `summary.failed` | `number` | Failed iterations | | `summary.passRate` | `number` | Pass rate (0.0 - 1.0) | *** ## Related * [Running Evals](/sdk/concepts/running-evals) - Conceptual guide * [EvalTest Reference](/sdk/reference/eval-test) - EvalTest API * [EvalSuite Reference](/sdk/reference/eval-suite) - EvalSuite API