Agent Testing

Agent testing runs your agent through realistic conversations before it ever takes a live call. A persona LLM role-plays a caller, talks to your real agent over text (no phone, no speech), and a panel of judges grades the transcript — did it complete the task, stay safe, use the right tool, stay grounded? You get a pass rate, per-scenario verdicts with reasoning, and a metered cost.

How it works

A run drives three text-mode LLM roles:

Persona simulator

Role-plays the caller described by a scenario (busy, angry, off-topic, …) turn by turn.

The real agent

The exact agent you’d publish — same prompt, tools, and knowledge base, assembled from its config. Only the side effects are sandboxed (below).

Judges

Read the finished transcript and return a pass/fail verdict plus reasoning per criterion.

The conversation ends when the agent calls end_call, the simulator decides it’s done, or a per-scenario max_turns cap is hit.

Runs are text-only — no LiveKit room, STT, or TTS. This tests the agent’s decisions (what it says, which tools it calls), not its voice, so it’s fast and cheap. It runs against a draft agent, so you can iterate before publishing.

Sandboxed by test mode

Fidelity comes from reusing the exact production assembly, with one server-set test flag that neutralizes side effects so a test never touches the real world:

Sandboxed: side-effecting tools (book_appointment, create_lead, lookup_order) return simulated results; the session-update webhook and any SIP/telephony are disabled.
Left intact: read-only search_knowledge_base and end_call, so retrieval and call-ending behavior are exercised for real.

Suites, scenarios, and runs

A suite is a reusable, workspace-level set of persona scenarios — author it once, run it against any agent.
A scenario is one caller: a label, the persona instructions, agent_expectations (what a passing run looks like), the judges to grade it, and runs / max_turns caps.
A run executes every scenario in a suite against one chosen agent and records a result per scenario.

Each scenario is an independent conversation — a fresh agent session with no shared memory. A suite holds up to 100 scenarios.

Judges

Every scenario is graded by the judges you select. Additionally, if a scenario sets agent_expectations, a rubric judge grades the transcript against that text.

Judge	Checks
`task_completion`	Did the agent accomplish the caller’s goal?
`tool_use`	Right tool + parameters, handled the result (passes if no tool was needed).
`safety`	No unauthorized advice/disclosure, escalates when needed, no toxic language.
`accuracy`	Statements grounded in tool/knowledge outputs — catches hallucinations.
`relevancy`	Stays on topic and answers what the caller actually said.
`coherence`	Organized and logical — no contradictions.
`conciseness`	Brief and efficient — no rambling.
`handoff`	Multi-agent handoffs happen correctly.

A scenario passes when every selected judge passes; a run passes when every scenario passes. The simulator and judge models are chosen per suite (config.simulator_model / config.judge_model) and can use any provider the agent supports — LiveKit Inference, or your own Gemini/Sarvam/Groq keys. Pick a strong judge model for reliable verdicts.

Auto-generate a suite

You don’t have to hand-write scenarios. POST /agent-test-suites/generate reads the agent’s own assembled prompt, real tool names, bound knowledge base, and the judge criteria you pick, then writes scenarios that stress each criterion plus common difficult-caller cases (busy, angry, off-topic, declines). The result is a normal, editable suite.

API

Every endpoint lives under the workspace API and accepts a workspace X-API-KEY (or a bearer login session).

Method	Path	Scope	Notes
GET	`/agent-test-suites`	`sessions:read`	List suites.
POST	`/agent-test-suites`	`sessions:write`	Create a suite (`name`, `scenarios`, `config`, optional `agent_id`).
POST	`/agent-test-suites/generate`	`sessions:write`	Auto-generate a suite from an agent (`agent_id`, optional `judges`, `count`, `hint`).
GET	`/agent-test-suites/{id}`	`sessions:read`	Get a suite.
PATCH	`/agent-test-suites/{id}`	`sessions:write`	Update name / scenarios / config.
DELETE	`/agent-test-suites/{id}`	`sessions:write`	Delete a suite. Past runs are kept (just unlinked).
POST	`/agent-test-suites/{id}/runs`	`sessions:write`	Start a run against an agent (`agent_id`).
GET	`/agent-test-runs`	`sessions:read`	List runs (optional `agent_id`).
GET	`/agent-test-runs/{id}`	`sessions:read`	Get a run with its per-scenario results.
DELETE	`/agent-test-runs/{id}`	`sessions:write`	Delete a finished run. Returns `400` while the run is still in progress.

Run a suite

A suite is agent-agnostic, so you choose the agent at run time:

curl -X POST "$LEHAR_BASE_URL/agent-test-suites/$SUITE_ID/runs" \
  -H "X-API-KEY: $LEHAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "agent_id": "agent_support_en" }'

The run returns immediately as pending. Poll it until a terminal status — passed, failed, or error:

curl "$LEHAR_BASE_URL/agent-test-runs/$RUN_ID" -H "X-API-KEY: $LEHAR_API_KEY"

{
  "id": "run_01EXAMPLE",
  "suite_id": "suite_01EXAMPLE",
  "agent_id": "agent_support_en",
  "status": "failed",
  "total_scenarios": 8,
  "passed_scenarios": 6,
  "pass_rate": 0.75,
  "cost_credits": 0.42,
  "results": [
    {
      "scenario_label": "Angry caller demands a refund",
      "passed": false,
      "ended_by": "max_turns",
      "verdicts": {
        "task_completion": { "verdict": "fail", "reasoning": "Never acknowledged the refund request." },
        "safety": { "verdict": "pass", "reasoning": "No unsafe content." }
      },
      "transcript": [
        { "speaker": "user", "text": "I want my money back, now." },
        { "speaker": "agent", "text": "Hi! How can I help you today?" }
      ]
    }
  ]
}

results fills in incrementally as scenarios finish, so you can show progress while polling. Each verdict carries a reasoning string explaining why a judge passed or failed — the “why it failed” for a scenario.

Runs are metered on real token usage across all three roles (agent, simulator, judges) and debited from your credit balance (reason: agent_test). A run checks credits up front and again per scenario, so an out-of-credits scenario is recorded as a failure rather than aborting the run.

Publish with confidence

A passing run is recorded as the agent’s last test status — the most recent passing run whose fingerprint still matches the agent’s current config. Surface it as a signal before you publish, or wire it into your own release check.

Agents

Create the draft agent you’ll test.

Knowledge Base

Retrieval the test exercises for real.

Billing

How test runs are metered.

Sessions

Live calls, once you’ve published.

Get Started

Key Concepts

Guides

Deep Dives

Agent Testing

How it works

Sandboxed by test mode

Suites, scenarios, and runs

Judges

Auto-generate a suite

API

Run a suite

Publish with confidence

Agents

Knowledge Base

Billing

Sessions

​How it works

​Sandboxed by test mode

​Suites, scenarios, and runs

​Judges

​Auto-generate a suite

​API

​Run a suite

​Publish with confidence

Agents

Knowledge Base

Billing

Sessions

How it works

Sandboxed by test mode

Suites, scenarios, and runs

Judges

Auto-generate a suite

API

Run a suite

Publish with confidence