> ## Documentation Index
> Fetch the complete documentation index at: https://docs.lehar.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Agent Testing
> Simulate real callers against a draft agent and grade the transcript before you publish.
**Agent testing** runs your agent through realistic conversations *before it ever takes a live call*. A persona LLM role-plays a caller, talks to your **real** agent over text (no phone, no speech), and a panel of judges grades the transcript — did it complete the task, stay safe, use the right tool, stay grounded? You get a pass rate, per-scenario verdicts with reasoning, and a metered cost.
## How it works
A run drives three text-mode LLM roles:
Role-plays the caller described by a scenario (busy, angry, off-topic, …) turn by turn.
The exact agent you'd publish — same prompt, tools, and knowledge base, assembled from its config. Only the *side effects* are sandboxed (below).
Read the finished transcript and return a pass/fail verdict plus reasoning per criterion.
The conversation ends when the agent calls `end_call`, the simulator decides it's done, or a per-scenario `max_turns` cap is hit.
Runs are **text-only** — no LiveKit room, STT, or TTS. This tests the agent's *decisions* (what it says, which tools it calls), not its voice, so it's fast and cheap. It runs against a **draft** agent, so you can iterate before publishing.
### Sandboxed by test mode
Fidelity comes from reusing the exact production assembly, with one server-set test flag that neutralizes side effects so a test never touches the real world:
* **Sandboxed:** side-effecting tools (`book_appointment`, `create_lead`, `lookup_order`) return simulated results; the session-update webhook and any SIP/telephony are disabled.
* **Left intact:** read-only `search_knowledge_base` and `end_call`, so retrieval and call-ending behavior are exercised for real.
## Suites, scenarios, and runs
* A **suite** is a reusable, **workspace-level** set of persona **scenarios** — author it once, run it against any agent.
* A **scenario** is one caller: a `label`, the persona `instructions`, `agent_expectations` (what a passing run looks like), the `judges` to grade it, and `runs` / `max_turns` caps.
* A **run** executes every scenario in a suite against **one chosen agent** and records a result per scenario.
Each scenario is an independent conversation — a fresh agent session with no shared memory. A suite holds up to **100** scenarios.
## Judges
Every scenario is graded by the judges you select. Additionally, if a scenario sets `agent_expectations`, a rubric judge grades the transcript against that text.
| Judge | Checks |
| ----------------- | ---------------------------------------------------------------------------- |
| `task_completion` | Did the agent accomplish the caller's goal? |
| `tool_use` | Right tool + parameters, handled the result (passes if no tool was needed). |
| `safety` | No unauthorized advice/disclosure, escalates when needed, no toxic language. |
| `accuracy` | Statements grounded in tool/knowledge outputs — catches hallucinations. |
| `relevancy` | Stays on topic and answers what the caller actually said. |
| `coherence` | Organized and logical — no contradictions. |
| `conciseness` | Brief and efficient — no rambling. |
| `handoff` | Multi-agent handoffs happen correctly. |
A scenario passes when every selected judge passes; a run passes when every scenario passes.
The **simulator** and **judge** models are chosen per suite (`config.simulator_model` / `config.judge_model`) and can use any provider the agent supports — LiveKit Inference, or your own Gemini/Sarvam/Groq keys. Pick a strong judge model for reliable verdicts.
## Auto-generate a suite
You don't have to hand-write scenarios. `POST /agent-test-suites/generate` reads the agent's **own** assembled prompt, real tool names, bound knowledge base, and the judge criteria you pick, then writes scenarios that stress each criterion plus common difficult-caller cases (busy, angry, off-topic, declines). The result is a normal, editable suite.
## API
Every endpoint lives under the workspace API and accepts a workspace `X-API-KEY` (or a bearer login session).
| Method | Path | Scope | Notes |
| ------ | ------------------------------ | ---------------- | ------------------------------------------------------------------------------------- |
| GET | `/agent-test-suites` | `sessions:read` | List suites. |
| POST | `/agent-test-suites` | `sessions:write` | Create a suite (`name`, `scenarios`, `config`, optional `agent_id`). |
| POST | `/agent-test-suites/generate` | `sessions:write` | Auto-generate a suite from an agent (`agent_id`, optional `judges`, `count`, `hint`). |
| GET | `/agent-test-suites/{id}` | `sessions:read` | Get a suite. |
| PATCH | `/agent-test-suites/{id}` | `sessions:write` | Update name / scenarios / config. |
| DELETE | `/agent-test-suites/{id}` | `sessions:write` | Delete a suite. Past runs are kept (just unlinked). |
| POST | `/agent-test-suites/{id}/runs` | `sessions:write` | Start a run against an agent (`agent_id`). |
| GET | `/agent-test-runs` | `sessions:read` | List runs (optional `agent_id`). |
| GET | `/agent-test-runs/{id}` | `sessions:read` | Get a run **with its per-scenario results**. |
| DELETE | `/agent-test-runs/{id}` | `sessions:write` | Delete a finished run. Returns `400` while the run is still in progress. |
### Run a suite
A suite is agent-agnostic, so you choose the agent at run time:
```bash theme={null}
curl -X POST "$LEHAR_BASE_URL/agent-test-suites/$SUITE_ID/runs" \
-H "X-API-KEY: $LEHAR_API_KEY" \
-H "Content-Type: application/json" \
-d '{ "agent_id": "agent_support_en" }'
```
The run returns immediately as `pending`. Poll it until a terminal status — `passed`, `failed`, or `error`:
```bash theme={null}
curl "$LEHAR_BASE_URL/agent-test-runs/$RUN_ID" -H "X-API-KEY: $LEHAR_API_KEY"
```
```json theme={null}
{
"id": "run_01EXAMPLE",
"suite_id": "suite_01EXAMPLE",
"agent_id": "agent_support_en",
"status": "failed",
"total_scenarios": 8,
"passed_scenarios": 6,
"pass_rate": 0.75,
"cost_credits": 0.42,
"results": [
{
"scenario_label": "Angry caller demands a refund",
"passed": false,
"ended_by": "max_turns",
"verdicts": {
"task_completion": { "verdict": "fail", "reasoning": "Never acknowledged the refund request." },
"safety": { "verdict": "pass", "reasoning": "No unsafe content." }
},
"transcript": [
{ "speaker": "user", "text": "I want my money back, now." },
{ "speaker": "agent", "text": "Hi! How can I help you today?" }
]
}
]
}
```
`results` fills in incrementally as scenarios finish, so you can show progress while polling. Each verdict carries a `reasoning` string explaining *why* a judge passed or failed — the "why it failed" for a scenario.
Runs are **metered on real token usage** across all three roles (agent, simulator, judges) and debited from your credit balance (`reason: agent_test`). A run checks credits up front and again per scenario, so an out-of-credits scenario is recorded as a failure rather than aborting the run.
## Publish with confidence
A passing run is recorded as the agent's **last test status** — the most recent passing run whose fingerprint still matches the agent's current config. Surface it as a signal before you publish, or wire it into your own release check.
Create the draft agent you'll test.
Retrieval the test exercises for real.
How test runs are metered.
Live calls, once you've published.