Saving Eval Results - MCPJam Inspector

Reporting authenticates with MCPJam API keys (sk_…) from Settings → API keys and uploads through the MCPJam public API (/api/v1/projects/:projectId/eval-ingest/*). The legacy project API keys (mcpjam_…) and their /sdk/v1/evals/* endpoints are retired — old SDK versions pointing there receive 410 Gone. Upgrade @mcpjam/sdk to keep uploading.

The SDK provides APIs to save eval results to MCPJam for visualization in the CI Evals dashboard. Results can be saved automatically via EvalTest/EvalSuite, or manually using the APIs below.

Environment Variables

Variable	Required	Default	Description
`MCPJAM_API_KEY`	Yes	-	An MCPJam API key (`sk_…`) from Settings → API keys
`MCPJAM_PROJECT_ID`	No	org’s Default project	Project the results are filed under
`MCPJAM_BASE_URL`	No	`https://app.mcpjam.com`	MCPJam API base URL override

Use MCPJAM_BASE_URL only when you need to override the default ingest host, such as internal development against a non-production backend. MCPJAM_API_KEY controls whether results are uploaded. Replay credential capture only happens when you provide serverReplayConfigs, agent, or mcpClientManager. MCP App widget snapshots in each result’s trace (for iframe replay in MCPJam) come from PromptResult.getWidgetSnapshots(), which is only populated when HostRunner was constructed with mcpClientManager.

Tool execution and `passed`

Uploaded iterations have a single boolean passed that drives pass rate on the dashboard. The SDK distinguishes:

Structural pass — Your test returned true, or expected tool calls were satisfied (Inspector/UI flows use shared matching logic).
Tool execution — The trace shows a real tool failure: MCP results with isError: true, timeline spans where a tool step ended in error, tool-result parts the UI would treat as errors, or a runner-level iterationError after a thrown tool step.

Default behavior: When the SDK derives passed from a trace (for example EvalTest / EvalSuite auto-save, PromptResult.toEvalResult(), createEvalRunReporter helpers, or Inspector suite runs), structural success is not enough if tool execution failed — the iteration is recorded as failed unless you opt out. Opt out: set failOnToolError: false on MCPJamReportingConfig (global for that reporter or auto-save), or pass it on specific helper options (addFromPrompt / recordFromRun / RunToEvalResultsOptions, etc.). Use this when you only care that the model invoked the right tools with the right arguments, not that every MCP call returned a success payload. Manual reportEvalResults: If you build results[] yourself and set passed explicitly, MCPJam stores your values as-is. The execution gate applies to code paths that compute passed from prompts, traces, and iterations. Programmatic reuse: @mcpjam/sdk also exports finalizePassedForEval, traceIndicatesToolExecutionFailure, isCallToolResultError, traceMessagePartIndicatesToolFailure, classifyToolFailurePart, and extractToolErrors for custom ingestion pipelines. The @mcpjam/sdk/predicates sub-package exports the full predicate evaluator (evaluatePredicate, evaluatePredicates, allPredicatesPassed, buildIterationTranscript) and all predicate types.

reportEvalResults()

One-shot reporting. Sends all results in a single call. Throws on failure.

import { MCPClientManager, reportEvalResults } from "@mcpjam/sdk";

Signature

reportEvalResults(input: ReportEvalResultsInput): Promise<ReportEvalResultsOutput>

Example

const manager = new MCPClientManager({
  asana: {
    url: process.env.MCP_SERVER_URL!,
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
    clientSecret: process.env.MCP_CLIENT_SECRET,
  },
});

await manager.connectToServer("asana");

const output = await reportEvalResults({
  suiteName: "Nightly",
  mcpClientManager: manager,
  results: [
    { caseTitle: "healthcheck", passed: true },
    { caseTitle: "tool-selection", passed: true, durationMs: 1200 },
    { caseTitle: "edge-case", passed: false, error: "Wrong tool called" },
  ],
  passCriteria: { minimumPassRate: 90 },
  ci: {
    branch: "main",
    commitSha: "abc123",
  },
});

console.log(`Run ${output.runId}: ${output.result}`);
// "Run abc123: passed"
console.log(`${output.summary.passed}/${output.summary.total} passed`);

reportEvalResultsSafely()

Same as reportEvalResults(), but returns null instead of throwing on failure. Warnings are logged to the console.

import { MCPClientManager, reportEvalResultsSafely } from "@mcpjam/sdk";

Signature

reportEvalResultsSafely(input: ReportEvalResultsInput): Promise<ReportEvalResultsOutput | null>

Example

const manager = new MCPClientManager({
  asana: {
    url: process.env.MCP_SERVER_URL!,
    refreshToken: process.env.MCP_REFRESH_TOKEN!,
    clientId: process.env.MCP_CLIENT_ID!,
  },
});

await manager.connectToServer("asana");

const output = await reportEvalResultsSafely({
  suiteName: "Nightly",
  mcpClientManager: manager,
  results: [{ caseTitle: "healthcheck", passed: true }],
});

if (output) {
  console.log(`Reported: ${output.summary.passRate * 100}% pass rate`);
} else {
  console.log("Reporting failed (non-blocking)");
}

Use reportEvalResultsSafely() when you don’t want eval reporting failures to break your CI pipeline. Use reportEvalResults() (strict) when reporting is critical.

createEvalRunReporter()

Creates an incremental reporter for long-running processes. Results are buffered and flushed in batches (up to 200 results or 1MB per batch).

import { createEvalRunReporter } from "@mcpjam/sdk";

Signature

createEvalRunReporter(input: CreateEvalRunReporterInput): EvalRunReporter

CreateEvalRunReporterInput accepts the same replay source fields as reportEvalResults(): serverReplayConfigs, agent, and mcpClientManager.

EvalRunReporter Methods

Method	Description
`add(result)`	Buffer a result (no network call)
`record(result)`	Buffer a result and auto-flush when buffer is large
`flush()`	Upload all buffered results
`finalize()`	Flush remaining results and finalize the run
`getBufferedCount()`	Number of results in the buffer
`getAddedCount()`	Total results added (including flushed)
`setExpectedIterations(count)`	Set expected iteration count for progress tracking

PromptResult Helpers

Method	Description
`addFromPrompt(promptResult, overrides?)`	Convert a `PromptResult` and buffer it
`recordFromPrompt(promptResult, overrides?)`	Convert a `PromptResult`, buffer it, and auto-flush

EvalTest/EvalSuite Run Helpers

Method	Description
`addFromRun(run, options)`	Convert all iterations from an `EvalTest` run
`recordFromRun(run, options)`	Convert and auto-flush from an `EvalTest` run
`addFromSuiteRun(suiteRun, options)`	Convert all iterations from an `EvalSuite` run
`recordFromSuiteRun(suiteRun, options)`	Convert and auto-flush from an `EvalSuite` run

Example

// Assumes `agent` was created with `mcpClientManager: manager`
const reporter = createEvalRunReporter({
  suiteName: "Integration Tests",
  passCriteria: { minimumPassRate: 85 },
  agent,
  ci: {
    branch: process.env.GITHUB_REF_NAME,
    commitSha: process.env.GITHUB_SHA,
  },
});

Example with manual replay source resolution

const reporter = createEvalRunReporter({
  suiteName: "Integration Tests",
  passCriteria: { minimumPassRate: 85 },
  mcpClientManager: manager,
});

// Add results as tests complete
await reporter.record({ caseTitle: "test-1", passed: true, durationMs: 500 });
await reporter.record({ caseTitle: "test-2", passed: false, error: "timeout" });
await reporter.record({ caseTitle: "test-3", passed: true });

// Finalize the run
const output = await reporter.finalize();
console.log(`${output.summary.passed}/${output.summary.total} passed`);

Replay credential sources

Authenticated HTTP evals can securely persist replay credentials for reruns and debugging. Manual reporting APIs resolve replay configs in this order:

serverReplayConfigs
agent.getServerReplayConfigs()
mcpClientManager.getServerReplayConfigs()

Prefer passing agent or mcpClientManager directly. Use serverReplayConfigs only when you need a low-level override. If you also provide serverNames, inferred replay configs are filtered to those server IDs before upload. Explicit serverReplayConfigs are left unchanged.

Replay metadata for the MCPJam UI

Uploaded runs can show Replay this run / server-side MCP replay when the ingest payload includes derived serverReplayConfigs (stored as hasServerReplayConfig on the run). In practice:

HTTP MCP (url) — Replay configs are built for typical streamable HTTP connections. Stdio transports do not produce entries from MCPClientManager.getServerReplayConfigs(); use HTTP when you need dashboard replay.
HostRunner vs reporter — Putting mcpClientManager on HostRunner fills MCP App widget snapshots on PromptResult. The reporter (and one-shot report*) resolves server replay from its own agent / mcpClientManager fields. Pass agent or mcpClientManager into createEvalRunReporter as well; agent-only wiring can still upload traces and widgets but omit hasServerReplayConfig.
Teardown order — finalize() / one-shot reporting calls getServerReplayConfigs() against connected registrations. In afterAll, run await reporter.finalize() (or reportEvalResults) before await manager.disconnectAllServers(). Disconnecting first clears manager state and uploads without replay metadata.

LLM API keys are not stored on the run. Replaying in the MCPJam UI still requires provider keys (e.g. OpenRouter) in Settings for your suite’s models.

Using with PromptResult

// Pass `agent` or `mcpClientManager` when you need server replay metadata in MCPJam
const reporter = createEvalRunReporter({ suiteName: "Prompt Tests", agent });

const result = await agent.run("Add 2 and 3");
reporter.addFromPrompt(result, {
  caseTitle: "addition",
  passed: result.hasToolCall("add"),
});

const output = await reporter.finalize();

Using with EvalTest Runs

const reporter = createEvalRunReporter({ suiteName: "Full Suite" });

const test = new EvalTest({
  name: "addition",
  test: async (agent) => (await agent.run("Add 2+3")).hasToolCall("add"),
});

const run = await test.run(agent, { iterations: 10 });
await reporter.recordFromRun(run, { casePrefix: "addition" });

const output = await reporter.finalize();

uploadEvalArtifact()

Parses test artifacts (JUnit XML, Jest JSON, Vitest JSON) and reports the results to MCPJam.

import { uploadEvalArtifact } from "@mcpjam/sdk";

Signature

uploadEvalArtifact(input: UploadEvalArtifactInput): Promise<ReportEvalResultsOutput>

Supported Formats

Format	Description
`"junit-xml"`	JUnit XML test reports
`"jest-json"`	Jest JSON output (`--json` flag)
`"vitest-json"`	Vitest JSON reporter output
`"custom"`	Custom parser via `customParser` option

Example

import { readFileSync } from "fs";

// Upload JUnit XML
await uploadEvalArtifact({
  suiteName: "CI Results",
  format: "junit-xml",
  artifact: readFileSync("test-results.xml", "utf-8"),
});

// Upload Jest JSON
await uploadEvalArtifact({
  suiteName: "Jest Results",
  format: "jest-json",
  artifact: readFileSync("jest-results.json", "utf-8"),
});

// Custom parser
await uploadEvalArtifact({
  suiteName: "Custom",
  format: "custom",
  artifact: myData,
  customParser: (data) => [
    { caseTitle: "test-1", passed: true },
    { caseTitle: "test-2", passed: false, error: "failed" },
  ],
});

Types

ReportEvalResultsInput

type ReportEvalResultsInput = MCPJamReportingConfig & {
  suiteName: string;
  results: EvalResultInput[];
  agent?: {
    getServerReplayConfigs?: () => MCPServerReplayConfig[] | undefined;
  };
  mcpClientManager?: MCPClientManager;
};

MCPJamReportingConfig

Property	Type	Required	Description
`enabled`	`boolean`	No	Enable/disable reporting (default: `true`)
`apiKey`	`string`	No	MCPJam API key (`sk_…`; falls back to `MCPJAM_API_KEY` env var)
`baseUrl`	`string`	No	MCPJam API base URL override (useful for internal development or tests)
`project`	`string`	No	Project id results are filed under (falls back to `MCPJAM_PROJECT_ID`, then the org’s Default project)
`suiteName`	`string`	No	Suite name for the run
`suiteDescription`	`string`	No	Description of the suite
`serverNames`	`string[]`	No	MCP server names being tested
`serverReplayConfigs`	`MCPServerReplayConfig[]`	No	Advanced override for replay credential capture
`notes`	`string`	No	Free-form notes
`passCriteria`	`{ minimumPassRate: number }`	No	Pass threshold (0-100)
`failOnToolError`	`boolean`	No	When not `false`, results derived from traces treat tool execution failures as failed iterations (default: strict). See Tool execution and `passed`
`strict`	`boolean`	No	Throw on upload errors (`false` = warn only)
`externalRunId`	`string`	No	Custom run ID (auto-generated if omitted)
`framework`	`string`	No	Test framework name (e.g., `"jest"`, `"vitest"`)
`ci`	`EvalCiMetadata`	No	CI/CD pipeline context
`expectedIterations`	`number`	No	Expected total iterations for progress tracking
`tags`	`string[]`	No	Free-form tags attached to the run for filtering in the MCPJam dashboard
`agent`	`{ getServerReplayConfigs?: () => MCPServerReplayConfig[] \| undefined }`	No	Preferred replay source for manual reporting; use an executor (HostRunner / HostRuntime) created with `mcpClientManager` so results can include `widgetSnapshots` from `PromptResult.toEvalResult()`. Field name predates the Stage 4 rename; any `HostExecutor` is accepted.
`mcpClientManager`	`MCPClientManager`	No	Replay source when no `agent` is provided; does not populate `widgetSnapshots` unless your `results[].widgetSnapshots` or trace payloads already include them
`host`	`Host`	No	Explicit host snapshot override. The reporter prefers `iteration.hostSnapshot` (per-iteration capture from `HostRuntime`) and falls back to `executor.getHostSnapshot?.()`; supply `host` only when neither source is available. See Run-level host snapshot below.

Run-level host snapshot

When a usable host snapshot is available, the reporter includes a normalized + content-hashed host config alongside the run body (the v1 ingest surface always accepts the pair — the old per-baseUrl capability probe is gone):

POST /api/v1/projects/{projectId}/eval-ingest/runs/start
{
  "suiteName": "...",
  "results": [...],
  "hostConfig":     {/* canonical HostConfigInputV2 with serverIds stripped */},
  "hostConfigHash": "sha256(canonical)"
}

Source order (highest priority first):

iteration.hostSnapshot — per-iteration capture from HostRuntime (Stage 4). Reflects the bound Host state at the iteration’s end.
executor.getHostSnapshot?.() — fallback for executors that don’t expose per-iteration snapshots.
MCPJamReportingConfig.host — explicit override, compatibility path.

Pass-1 homogeneity gate. The reporter only sends a run-level { hostConfig, hostConfigHash } when all available iteration snapshots canonicalize to the same hash. Heterogeneous runs (e.g. mutating the bound Host between iterations) omit the run-level field — per-iteration wire support is a later stage. Fail-safe omission. Any error while resolving or canonicalizing the snapshot — malformed hostSnapshot, an executor that throws, non-canonicalizable host JSON — logs a warning and omits the wire pair rather than failing the eval upload. The body shape stays the same without it. Server-id normalization. Runtime-manager identifiers (serverIds, optionalServerIds, serverConnectionOverrides) are stripped by normalizeSdkEvalHostConfigForWire on both ends so SDK runtime ids like "everything" never reach the backend’s validateServerScope as Id<'servers'>. The persisted hostConfigsV2 row’s hash will differ from the wire hostConfigHash because suite-resolved Convex server ids are layered on top at storage time — the wire hash is a transport-integrity check, not the storage id.

MCPServerReplayConfig

Advanced replay override. Most users should not construct this manually.

Property	Type	Required	Description
`serverId`	`string`	Yes	MCP server identifier
`url`	`string`	Yes	MCP server URL
`preferSSE`	`boolean`	No	Prefer SSE transport for replay
`accessToken`	`string`	No	Static bearer token for replay
`refreshToken`	`string`	No	Refresh token for replay
`clientId`	`string`	No	OAuth client ID, required with `refreshToken`
`clientSecret`	`string`	No	OAuth client secret when needed for token refresh

EvalCiMetadata

Property	Type	Description
`provider`	`string`	CI provider (e.g., `"github"`, `"gitlab"`)
`pipelineId`	`string`	Pipeline/workflow identifier
`jobId`	`string`	Job identifier
`runUrl`	`string`	URL to the CI run
`branch`	`string`	Git branch name
`commitSha`	`string`	Git commit SHA

EvalResultInput

Property	Type	Required	Description
`caseTitle`	`string`	Yes	Test case title
`passed`	`boolean`	Yes	Whether the test passed
`query`	`string`	No	The prompt/query sent
`durationMs`	`number`	No	Test duration in ms
`provider`	`string`	No	LLM provider name
`model`	`string`	No	Model identifier
`expectedToolCalls`	`EvalExpectedToolCall[]`	No	Expected tool calls
`actualToolCalls`	`EvalExpectedToolCall[]`	No	Actual tool calls made
`tokens`	`{ input?, output?, total? }`	No	Token usage
`error`	`string`	No	Error message
`errorDetails`	`string`	No	Detailed error info
`trace`	`EvalTraceInput`	No	Conversation trace
`externalIterationId`	`string`	No	Custom iteration ID
`externalCaseId`	`string`	No	Custom case ID
`metadata`	`Record<string, unknown>`	No	Custom metadata. When `successPredicates` are evaluated, the runner persists `{ predicate, passed, reason }[]` under `metadata.predicates` automatically.
`successPredicates`	`Predicate[]`	No	State-based predicate gate evaluated over the iteration transcript. See Predicate gate.
`isNegativeTest`	`boolean`	No	Whether this is a negative test
`matchOptions`	`EvalMatchOptions`	No	Per-result match options. When present, the inspector snapshots these onto the iteration so historical pass/fail computation honors them. See `EvalMatchOptions`
`widgetSnapshots`	`EvalWidgetSnapshotInput[]`	No	MCP App HTML replay payloads (typically from `getWidgetSnapshots()` on `PromptResult`). Omitted when not using MCP Apps or when `HostRunner` had no `mcpClientManager`

Predicate gate

successPredicates adds a deterministic, state-based assertion layer on top of tool-call matching. Each predicate is a pure function of the iteration transcript — same transcript, same verdict — which makes it suitable as a CI release gate. A case passes the predicate gate iff all predicates pass. An absent or empty list means no predicate gate (the case is judged solely by tool-call matching and failOnToolError).

Predicate types

Type	Fields	Description
`toolCalledWith`	`toolName`, `args` (`ArgMatcher`), `minCount?`	A call to `toolName` whose args satisfy `args` occurred at least `minCount` (default 1) times.
`toolCalledAtLeastOnce`	`toolName`	`toolName` was called at least once (args irrelevant).
`toolNeverCalled`	`toolName`	`toolName` was never called (forbidden-tool check).
`responseContains`	`needle`, `caseSensitive?`	The final assistant message contains `needle`. Case-insensitive by default.
`responseMatches`	`pattern`	The final assistant message matches the regular expression `pattern` (regex source string, no flags).
`noToolErrors`	—	No tool produced an error (neither MCP `isError: true` nor a JSON-RPC/transport failure).
`finalAssistantMessageNonEmpty`	—	The final assistant message is a non-empty, non-whitespace string.
`tokenBudgetUnder`	`tokens`	Total token usage for the iteration is strictly under `tokens`. Fails closed when usage is unavailable.

ArgMatcher

Used by toolCalledWith to specify expected arguments and matching mode.

Property	Type	Description
`args`	`Record<string, unknown>`	Expected argument shape.
`argumentMatching`	`"partial" \| "exact" \| "ignore"`	Matching mode. `"partial"` (default) checks only the keys present in `args`; `"exact"` requires deep equality; `"ignore"` skips argument comparison.

Fail-closed behavior

Malformed predicates (unknown type, missing required fields, invalid minCount, empty needle/pattern) always fail the case rather than silently passing. responseMatches also fails closed on patterns with nested quantifiers (ReDoS guard) and on messages exceeding 100,000 characters.

Example

import type { Predicate } from "@mcpjam/sdk/predicates";

const predicates: Predicate[] = [
  // Tool was called with the right airline argument
  {
    type: "toolCalledWith",
    toolName: "book_flight",
    args: { args: { airline: "DL" } },
  },
  // Response mentions the booking confirmation
  { type: "responseContains", needle: "confirmed" },
  // No tool errors occurred
  { type: "noToolErrors" },
  // Token budget respected
  { type: "tokenBudgetUnder", tokens: 2000 },
];

// Pass predicates via the per-run override in the Inspector corpus,
// or use evaluatePredicates() directly in custom pipelines:
import { evaluatePredicates, buildIterationTranscript } from "@mcpjam/sdk/predicates";

const transcript = buildIterationTranscript({
  trace: myTrace,
  toolCalls: myToolCalls,
  usage: { totalTokens: 1200 },
});

const results = evaluatePredicates(transcript, predicates);
const allPassed = results.every((r) => r.passed);

Predicate results in metadata

When the Inspector runner evaluates predicates, it persists one row per predicate to testIteration.metadata.predicates:

type PredicateResult = {
  predicate: Predicate;   // The authored predicate (sensitive arg keys are redacted)
  passed: boolean;
  reason: string;         // Structured explanation, e.g. "tool "book_flight" called with matching args"
};

ReportEvalResultsOutput

Property	Type	Description
`suiteId`	`string`	Created/matched suite ID
`runId`	`string`	Created run ID
`status`	`"completed" \| "failed"`	Run status
`result`	`"passed" \| "failed"`	Pass/fail based on criteria
`summary.total`	`number`	Total iterations
`summary.passed`	`number`	Passed iterations
`summary.failed`	`number`	Failed iterations
`summary.passRate`	`number`	Pass rate (0.0 - 1.0)

Running Evals - Conceptual guide
EvalTest Reference - EvalTest API
EvalSuite Reference - EvalSuite API

​Environment Variables

​Tool execution and passed

​reportEvalResults()

​Signature

​Example

​reportEvalResultsSafely()

​Signature

​Example

​createEvalRunReporter()

​Signature

​EvalRunReporter Methods

​PromptResult Helpers

​EvalTest/EvalSuite Run Helpers

​Example

​Example with manual replay source resolution

​Replay credential sources

​Replay metadata for the MCPJam UI

​Using with PromptResult

​Using with EvalTest Runs

​uploadEvalArtifact()

​Signature

​Supported Formats

​Example

​Types

​ReportEvalResultsInput

​MCPJamReportingConfig

​Run-level host snapshot

​MCPServerReplayConfig

​EvalCiMetadata

​EvalResultInput

​Predicate gate

​Predicate types

​ArgMatcher

​Fail-closed behavior

​Example

​Predicate results in metadata

​ReportEvalResultsOutput

​Related

Environment Variables

Tool execution and `passed`

reportEvalResults()

Signature

Example

reportEvalResultsSafely()

Signature

Example

createEvalRunReporter()

Signature

EvalRunReporter Methods

PromptResult Helpers

EvalTest/EvalSuite Run Helpers

Example

Example with manual replay source resolution

Replay credential sources

Replay metadata for the MCPJam UI

Using with PromptResult

Using with EvalTest Runs

uploadEvalArtifact()

Signature

Supported Formats

Example

Types

ReportEvalResultsInput

MCPJamReportingConfig

Run-level host snapshot

MCPServerReplayConfig

EvalCiMetadata

EvalResultInput

Predicate gate

Predicate types

ArgMatcher

Fail-closed behavior

Example

Predicate results in metadata

ReportEvalResultsOutput

Related