Sandboxing

Khaos evaluations can execute arbitrary agent code, including tool calls that interact with the file system, network, and other system resources. Sandboxing isolates this execution so that untrusted or adversarial payloads cannot damage the host environment.

Why Sandbox?

Agent evaluations are inherently adversarial. Scenarios inject malicious prompts, tool outputs, and payloads designed to trick agents into executing harmful actions. Without sandboxing:

Security — a prompt injection could cause an agent to run destructive shell commands on the host
Isolation — one evaluation run could interfere with another, corrupting results
Reproducibility — file system state from previous runs could leak into subsequent ones, making results non-deterministic

CI Environments

Always enable sandboxing when running evaluations in CI pipelines. Shared runners are particularly vulnerable to cross-contamination between jobs.

SandboxMode

The SandboxMode enum controls how agent code is isolated:

Python

from khaos.sandbox import SandboxMode

class SandboxMode(str, Enum):
    DISABLED = "disabled"      # No sandboxing — run on host directly
    DOCKER = "docker"          # Isolated Docker container per run
    SUBPROCESS = "subprocess"  # OS-level subprocess with restricted permissions

Mode	Isolation Level	Requirements	Use Case
`DISABLED`	None	None	Local development, trusted agents
`DOCKER`	Full container	Docker daemon	CI, adversarial testing, production
`SUBPROCESS`	Process-level	Unix-like OS	Lightweight isolation without Docker

NetworkMode

The NetworkMode enum controls network access from within the sandbox:

Python

from khaos.sandbox import NetworkMode

class NetworkMode(str, Enum):
    NONE = "none"            # No network access at all
    HOST = "host"            # Full host network access
    BRIDGE = "bridge"        # Docker bridge network (container-to-container)
    ALLOWLIST = "allowlist"  # Only allow connections to specified hosts

Security Testing

Use NetworkMode.NONE when running security-focused evaluations. This prevents agents from exfiltrating data even if they are successfully exploited by a scenario.

SandboxConfig

The SandboxConfig dataclass holds all sandbox configuration. Every field has a sensible default so you only need to override what matters for your use case.

Python

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class SandboxConfig:
    enabled: bool = False
    mode: SandboxMode = SandboxMode.DOCKER
    network_mode: NetworkMode = NetworkMode.NONE
    memory_limit: str = "512m"          # Docker memory limit
    cpu_limit: float = 1.0              # Number of CPUs
    timeout_seconds: int = 300          # Max execution time per run
    read_only_root: bool = True         # Mount root filesystem as read-only
    working_dir: str = "/workspace"     # Working directory inside sandbox
    mount_paths: list[str] = field(default_factory=list)
    env_allowlist: list[str] = field(default_factory=list)
    env_denylist: list[str] = field(
        default_factory=lambda: ["AWS_SECRET_ACCESS_KEY", "GITHUB_TOKEN"]
    )
    allowed_hosts: list[str] = field(default_factory=list)
    docker_image: str = "python:3.12-slim"
    user: Optional[str] = "nobody"
    capabilities_drop: list[str] = field(
        default_factory=lambda: ["ALL"]
    )
    seccomp_profile: Optional[str] = "default"

Field	Type	Default	Description
`enabled`	`bool`	`False`	Whether sandboxing is active
`mode`	`SandboxMode`	`DOCKER`	Isolation strategy
`network_mode`	`NetworkMode`	`NONE`	Network access policy
`memory_limit`	`str`	`512m`	Container memory cap
`cpu_limit`	`float`	`1.0`	CPU core allocation
`timeout_seconds`	`int`	`300`	Maximum execution duration
`read_only_root`	`bool`	`True`	Read-only root filesystem
`working_dir`	`str`	`/workspace`	Container working directory
`mount_paths`	`list[str]`	`[]`	Host paths to mount into container
`env_allowlist`	`list[str]`	`[]`	Environment variables to pass through
`env_denylist`	`list[str]`	Secrets	Variables explicitly blocked
`allowed_hosts`	`list[str]`	`[]`	Hosts reachable in ALLOWLIST mode
`docker_image`	`str`	`python:3.12-slim`	Base Docker image
`user`	`Optional[str]`	`nobody`	User inside the container
`capabilities_drop`	`list[str]`	`["ALL"]`	Linux capabilities to drop
`seccomp_profile`	`Optional[str]`	`default`	Seccomp security profile

DockerSandbox

The DockerSandbox class manages the lifecycle of isolated Docker containers for evaluation runs. It creates a fresh container per execution, runs the agent code, and tears it down afterward.

Python

from khaos.sandbox import DockerSandbox, SandboxConfig, SandboxMode, NetworkMode

config = SandboxConfig(
    enabled=True,
    mode=SandboxMode.DOCKER,
    network_mode=NetworkMode.NONE,
    memory_limit="1g",
    timeout_seconds=120,
)

sandbox = DockerSandbox(config)
result = sandbox.execute("print('Hello from sandbox')")

print(result.exit_code)     # 0
print(result.stdout)        # "Hello from sandbox\n"
print(result.stderr)        # ""
print(result.timed_out)     # False
print(result.container_id)  # "a1b2c3d4e5f6"

SandboxResult

Every sandbox execution returns a SandboxResult with full details about the run:

Python

@dataclass
class SandboxResult:
    exit_code: int           # Process exit code (0 = success)
    stdout: str              # Standard output captured
    stderr: str              # Standard error captured
    timed_out: bool          # Whether the execution hit the timeout
    container_id: str | None # Docker container ID (if applicable)

Check timed_out before inspecting exit_code — a timed-out process may have a non-zero exit code that is not meaningful.

Utility Functions

is_sandbox_available

Check whether a given sandbox mode is available on the current system:

Python

from khaos.sandbox import is_sandbox_available, SandboxMode

if is_sandbox_available(SandboxMode.DOCKER):
    print("Docker is available")
else:
    print("Falling back to subprocess mode")

# Check subprocess mode
is_sandbox_available(SandboxMode.SUBPROCESS)  # True on Unix, False on Windows

get_default_config

Returns a SandboxConfig populated from environment variables and the user's ~/.khaos/config.yaml:

Python

from khaos.sandbox import get_default_config

config = get_default_config()
print(config.mode)           # SandboxMode from config or env
print(config.network_mode)   # NetworkMode from config or env
print(config.timeout_seconds) # 300 unless overridden

Configuration

Sandbox settings can be configured via environment variables or ~/.khaos/config.yaml. See the Configuration page for full details.

YAML

# ~/.khaos/config.yaml
sandbox:
  enabled: true
  mode: docker
  network: none
  memory_limit: 1g
  cpu_limit: 2.0
  timeout_seconds: 600
  docker_image: python:3.12-slim
  read_only_root: true
  capabilities_drop:
    - ALL
  env_denylist:
    - AWS_SECRET_ACCESS_KEY
    - GITHUB_TOKEN
    - OPENAI_API_KEY

Environment variables override config file values:

Terminal

# Enable sandbox via environment
export KHAOS_SANDBOX_ENABLED=true
export KHAOS_SANDBOX_MODE=docker
export KHAOS_SANDBOX_NETWORK=none
export KHAOS_SANDBOX_TIMEOUT=600

Best Practices

Always sandbox in CI — shared CI runners are multi-tenant environments. Use SandboxMode.DOCKER with NetworkMode.NONE to prevent cross-job interference and data exfiltration.
Use NONE network for security tests — security-focused evaluations should never allow network access. A successfully exploited agent could phone home.
Keep timeouts reasonable — set timeout_seconds to the minimum needed for your agent. Long timeouts waste CI minutes and delay feedback.
Drop all capabilities — the default capabilities_drop: ["ALL"] is correct for most use cases. Only add capabilities back if your agent genuinely needs them.
Denylist secrets — add any API keys or tokens to env_denylist so they are never available inside the sandbox, even if set on the host.

Architecture Context

Sandboxing fits into the broader Khaos execution pipeline described in the Architecture Overview. The sandbox layer sits between the scenario runner and the agent adapter.

Transport Layer

Decision Records