> ## Documentation Index
> Fetch the complete documentation index at: https://mcpjam-mintlify-docs-update-pr-2772-1782277917413.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# EvalTest

> API reference for EvalTest

The `EvalTest` class runs a single test scenario multiple times and provides statistical metrics like accuracy, precision, and recall.

## Import

```typescript theme={null}
import { EvalTest } from "@mcpjam/sdk";
```

## Constructor

```typescript theme={null}
new EvalTest(options: EvalTestConfig)
```

### Parameters

<ParamField path="options" type="EvalTestConfig" required>
  Configuration for the evaluation test.
</ParamField>

### EvalTestConfig

| Property | Type           | Required | Description                    |
| -------- | -------------- | -------- | ------------------------------ |
| `name`   | `string`       | Yes      | Unique identifier for the test |
| `test`   | `TestFunction` | Yes      | The test function to run       |

### TestFunction Type

```typescript theme={null}
type TestFunction = (executor: HostExecutor) => boolean | Promise<boolean>;
```

The test function receives a [`HostExecutor`](/sdk/reference/host-runner) (the interface implemented by both `HostRunner` and `HostRuntime`) and must return a `boolean`:

* `true` = test passed
* `false` = test failed

Both `HostRunner`, `HostRuntime`, and mock executors implement the `HostExecutor` interface, so you can use any of them for testing.

### Example

```typescript theme={null}
const test = new EvalTest({
  name: "addition-accuracy",
  test: async (agent) => {
    const result = await agent.run("Add 2 and 3");
    return result.hasToolCall("add");
  },
});
```

***

## Methods

### run()

Executes the test multiple times and returns detailed results.

```typescript theme={null}
run(executor: HostExecutor, options: EvalTestRunOptions): Promise<EvalRunResult>
```

#### Parameters

| Parameter  | Type                 | Description                                                      |
| ---------- | -------------------- | ---------------------------------------------------------------- |
| `executor` | `HostExecutor`       | The executor to test with (`HostRunner`, `HostRuntime`, or mock) |
| `options`  | `EvalTestRunOptions` | Run configuration                                                |

#### EvalTestRunOptions

| Property      | Type                                                     | Required | Default | Description                                                                                                                                                                                        |
| ------------- | -------------------------------------------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `iterations`  | `number`                                                 | Yes      | -       | Number of test runs                                                                                                                                                                                |
| `concurrency` | `number`                                                 | No       | `5`     | Parallel test runs                                                                                                                                                                                 |
| `retries`     | `number`                                                 | No       | `0`     | Retry failed tests                                                                                                                                                                                 |
| `timeoutMs`   | `number`                                                 | No       | `30000` | Per-iteration wall-clock timeout in ms. The active prompt is aborted at this deadline, then given a 1 second grace period to settle so partial tool calls and trace history can still be captured. |
| `onProgress`  | `ProgressCallback`                                       | No       | -       | Progress callback                                                                                                                                                                                  |
| `onFailure`   | `(report: string) => void`                               | No       | -       | Called with a failure report if any iterations fail                                                                                                                                                |
| `mcpjam`      | [`MCPJamReportingConfig`](/sdk/reference/eval-reporting) | No       | -       | Auto-save results to MCPJam                                                                                                                                                                        |

<Note>
  Results are automatically saved to MCPJam after the run completes when an API
  key is available via `mcpjam.apiKey` or the `MCPJAM_API_KEY` environment
  variable. Set `mcpjam.enabled: false` to disable.
</Note>

#### ProgressCallback Type

```typescript theme={null}
type ProgressCallback = (completed: number, total: number) => void;
```

#### Example

```typescript theme={null}
await test.run(agent, {
  iterations: 30,
  concurrency: 5,
  retries: 2,
  timeoutMs: 30000,
  mcpjam: {
    suiteName: "Addition Eval",
    strict: false,
  },
  onProgress: (done, total) => {
    console.log(`${done}/${total}`);
  },
  onFailure: (report) => {
    console.error(report);
  },
});
```

***

### accuracy()

Returns the success rate (0.0 - 1.0).

```typescript theme={null}
accuracy(): number
```

#### Returns

`number` - Proportion of tests that passed.

#### Example

```typescript theme={null}
console.log(`Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
// "Accuracy: 96.7%"
```

***

### precision()

Returns the precision metric.

```typescript theme={null}
precision(): number
```

#### Returns

`number` - True positives / (True positives + False positives).

***

### recall()

Returns the recall metric.

```typescript theme={null}
recall(): number
```

#### Returns

`number` - True positives / (True positives + False negatives).

***

### truePositiveRate()

Returns the true positive rate (same as recall).

```typescript theme={null}
truePositiveRate(): number
```

***

### falsePositiveRate()

Returns the false positive rate.

```typescript theme={null}
falsePositiveRate(): number
```

#### Returns

`number` - False positives / (False positives + True negatives).

***

### averageTokenUse()

Returns the average tokens used per iteration.

```typescript theme={null}
averageTokenUse(): number
```

#### Returns

`number` - Mean token count.

#### Example

```typescript theme={null}
console.log(`Avg tokens: ${test.averageTokenUse()}`);
```

***

### getResults()

Returns the full run result from the last run.

```typescript theme={null}
getResults(): EvalRunResult | null
```

#### Returns

`EvalRunResult | null` - The run result, or `null` if `run()` hasn't been called.

#### EvalRunResult Type

| Property           | Type                | Description                                |
| ------------------ | ------------------- | ------------------------------------------ |
| `iterations`       | `number`            | Total iterations run                       |
| `successes`        | `number`            | Number that passed                         |
| `failures`         | `number`            | Number that failed                         |
| `results`          | `boolean[]`         | Pass/fail per iteration                    |
| `iterationDetails` | `IterationResult[]` | Detailed per-iteration results             |
| `tokenUsage`       | `object`            | Aggregate and per-iteration token usage    |
| `latency`          | `object`            | Latency stats (e2e, llm, mcp) with p50/p95 |

#### IterationResult Type

| Property     | Type                          | Description                          |
| ------------ | ----------------------------- | ------------------------------------ |
| `passed`     | `boolean`                     | Whether this iteration passed        |
| `latencies`  | `LatencyBreakdown[]`          | Latency per prompt in this iteration |
| `tokens`     | `{ total, input, output }`    | Token usage                          |
| `error`      | `string \| undefined`         | Error message if failed              |
| `retryCount` | `number \| undefined`         | Number of retries attempted          |
| `prompts`    | `PromptResult[] \| undefined` | Prompt results from this iteration   |

***

### getName()

Returns the test's name.

```typescript theme={null}
getName(): string
```

***

### getConfig()

Returns the test's configuration.

```typescript theme={null}
getConfig(): EvalTestConfig
```

***

### getAllIterations()

Returns all iteration details from the last run.

```typescript theme={null}
getAllIterations(): IterationResult[]
```

***

### getFailedIterations()

Returns only the failed iterations from the last run.

```typescript theme={null}
getFailedIterations(): IterationResult[]
```

#### Example

```typescript theme={null}
const failures = test.getFailedIterations();
console.log(`${failures.length} failures`);

for (const fail of failures) {
  console.log(`  Error: ${fail.error}`);
}
```

***

### getSuccessfulIterations()

Returns only the successful iterations from the last run.

```typescript theme={null}
getSuccessfulIterations(): IterationResult[]
```

***

### getFailureReport()

Returns a formatted failure report with traces from all failed iterations. Useful for debugging.

```typescript theme={null}
getFailureReport(): string
```

#### Example

```typescript theme={null}
await test.run(agent, { iterations: 30 });

if (test.accuracy() < 0.9) {
  console.error(test.getFailureReport());
}
```

***

## Properties

### name

The test's identifier (via `getName()`).

```typescript theme={null}
test.getName(); // "addition-accuracy"
```

***

## Test Function Patterns

### Simple Tool Check

```typescript theme={null}
test: async (agent) => {
  const result = await agent.run("Add 5 and 3");
  return result.hasToolCall("add");
};
```

### Argument Validation

```typescript theme={null}
test: async (agent) => {
  const result = await agent.run("Add 10 and 20");
  const args = result.getToolArguments("add");
  return args?.a === 10 && args?.b === 20;
};
```

### Response Content

```typescript theme={null}
test: async (agent) => {
  const result = await agent.run("What is 5 + 5?");
  return result.getText().includes("10");
};
```

### Multiple Conditions

```typescript theme={null}
test: async (agent) => {
  const result = await agent.run("Calculate 5 * 3");
  return (
    result.hasToolCall("multiply") &&
    !result.hasError() &&
    result.getText().length > 0
  );
};
```

### Multi-Turn Conversation

```typescript theme={null}
test: async (agent) => {
  const r1 = await agent.run("Create a project");
  const r2 = await agent.run("Add a task to it", { context: r1 });
  return r1.hasToolCall("createProject") && r2.hasToolCall("createTask");
};
```

### With Validators

```typescript theme={null}
import { matchToolCallWithArgs } from "@mcpjam/sdk";

test: async (agent) => {
  const result = await agent.run("Add 2 and 3");
  return matchToolCallWithArgs("add", { a: 2, b: 3 }, result.getToolCalls());
};
```

***

## Complete Example

```typescript theme={null}
import { MCPClientManager, HostRunner, EvalTest } from "@mcpjam/sdk";

async function main() {
  const manager = new MCPClientManager({
    everything: {
      command: "npx",
      args: ["-y", "@modelcontextprotocol/server-everything"],
    },
  });
  await manager.connectToServer("everything");

  const agent = new HostRunner({
    tools: await manager.getTools(),
    model: "anthropic/claude-sonnet-4-20250514",
    apiKey: process.env.ANTHROPIC_API_KEY,
    temperature: 0.1,
  });

  const test = new EvalTest({
    name: "addition",
    test: async (agent) => {
      const r = await agent.run("Add 2 and 3");
      return r.hasToolCall("add");
    },
  });

  console.log("Running evaluation...");

  const result = await test.run(agent, {
    iterations: 30,
    concurrency: 5,
    mcpjam: { suiteName: "Addition Eval" },
    onProgress: (done, total) => {
      process.stdout.write(`\r${done}/${total}`);
    },
    onFailure: (report) => {
      console.error(report);
    },
  });

  console.log("\n\nResults:");
  console.log(`  Accuracy: ${(test.accuracy() * 100).toFixed(1)}%`);
  console.log(`  Precision: ${(test.precision() * 100).toFixed(1)}%`);
  console.log(`  Recall: ${(test.recall() * 100).toFixed(1)}%`);
  console.log(`  Avg tokens: ${test.averageTokenUse()}`);
  console.log(`  Iterations: ${result.iterations}`);
  console.log(`  Successes: ${result.successes}`);

  await manager.disconnectServer("everything");
}
```

***

## Related

* [Running Evals](/sdk/concepts/running-evals) - Conceptual guide
* [EvalSuite Reference](/sdk/reference/eval-suite) - Group multiple tests
* [Saving Eval Results](/sdk/reference/eval-reporting) - Save results to MCPJam
* [Validators Reference](/sdk/reference/validators) - Assertion functions