Agent Testing Framework

Write agent tests with @khaostest and run them withkhaos test. This page reflects the stable SDK surface in this repo.

Quick Start

tests/test_smoke.py
from khaos.testing import khaostest

@khaostest(agent="my-agent")
def test_smoke(agent):
    result = agent("What is 2+2?")
    assert result.success
    assert "4" in result.text
Terminal
khaos discover
khaos test

Decorator Configuration

@khaostest maps directly to KhaosTestConfig.

FieldTypeDescription
agentstrRegistered agent name to target
faultslist[str | dict]Faults to apply for this test
attackslist[str | dict]Attack payloads or ids to run
inputslist[str | dict]Custom test inputs
timeout_msint | NonePer-test timeout override
expect_blockedbool | NoneOptional expected blocked behavior
min_scorefloat | NoneOptional minimum score hint
tagslist[str]Label tests for filtering/grouping

Imperative vs Declarative

Khaos infers mode from function signature:

  • Imperative: function includes agent parameter.
  • Declarative: function has no agent parameter.
Python
from khaos.testing import khaostest

@khaostest(agent="my-agent", tags=["declarative"])
def test_declarative():
    # Khaos executes from decorator config
    pass

@khaostest(agent="my-agent", tags=["imperative"])
def test_imperative(agent):
    result = agent("hello")
    assert result.success

    # You can inspect prior calls
    assert len(agent.call_history) >= 1

AgentResponse Fields

Python
result.success     # bool
result.text        # normalized text output
result.raw         # full raw response payload
result.latency_ms  # float
result.tokens_in   # int
result.tokens_out  # int
result.error       # str | None

CLI Usage

Terminal
# Run all tests (default path: tests/)
khaos test

# Run specific path(s)
khaos test tests/test_security.py
khaos test tests/security tests/resilience

# Filtering / control
khaos test -k "security"
khaos test -x
khaos test --timeout 60000

# Output formats
khaos test --format junit -o results.xml
khaos test --format json -o results.json
khaos test --format markdown -o results.md
khaos test --format all -o khaos-tests
CI Behavior
khaos test exits non-zero on failing tests, so it can be used directly as a CI gate.

CI Example

.github/workflows/khaos-tests.yml
name: Khaos Tests

on: [push, pull_request]

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install khaos-agent
      - run: khaos discover
      - run: khaos test --format junit -o khaos-results.xml

Current API Boundaries

Stable Testing Surface
The stable SDK currently exports only khaostest,KhaosTestConfig, AgentTestClient, andAgentResponse from khaos.testing.

For exact exported symbols, see Testing API Surface (Current).

Related Docs