> ## Documentation Index
> Fetch the complete documentation index at: https://hud-f5fd7c15-lukass-phys-experimental.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Protocol

> The HUD wire protocol: a thin envelope of small messages that an agent and an environment exchange to start, step, answer, and grade a task.

HUD is **protocol-first**. An agent and an environment never integrate directly - they sit on two sides of a thin envelope and exchange a handful of small messages. HUD owns only that envelope; everything inside it - the model, the harness, the work the agent does - stays swappable.

Three things take part in every run:

|                                                | What it is                                                                                                 |
| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| [**Agent**](/v6/reference/agents)              | The *client* (a harness around a model). Drives the work - reads, acts, repeats. Any model, any framework. |
| [**Environment**](/v6/reference/environment)   | The *server*. Holds the world, the tasks, and the grading. This is the part you author.                    |
| [**Capabilities**](/v6/reference/capabilities) | The live connections the agent acts through - `ssh`, `mcp`, `cdp`, `rfb`, `robot`.                         |

## The loop

```mermaid theme={"dark"}
sequenceDiagram
    participant Agent
    participant Env as Environment
    participant Caps as Capabilities (ssh · mcp · cdp · rfb · robot)
    Note over Env,Caps: environment holds & serves these
    Agent->>Env: hello
    Env-->>Agent: manifest (capabilities)
    Agent->>Env: tasks.start
    Env-->>Agent: prompt
    rect rgb(238,238,238)
    Note over Agent,Caps: the agent works, driving capabilities directly
    Agent->>Caps: shell · browser · GUI · tools · robot
    Caps-->>Agent: observations
    end
    Agent->>Env: tasks.grade
    Env-->>Agent: reward
```

The agent opens with a `hello`, and the environment answers with its **manifest** - every capability it holds. The capabilities are advertised here, not yet touched. Nothing in the manifest is model-specific: it describes the environment, not any particular agent.

The orchestrator (the harness, `hud eval`, or the platform) names a task and calls `tasks.start`. The environment sets up the world for it and returns a **prompt**. The agent then works the task directly against the capabilities - a real shell over `ssh`, a real browser over `cdp` - reading observations and acting in a loop. The environment decides *what* the agent can touch, not *how* it works.

When the agent is done it calls `tasks.grade`. The environment inspects the resulting state and returns one **reward**. That number, with the trace of the run, is the same value you read in an eval and feed into [training](/v6/reference/training).

## Two halves, one thin envelope

The loop has only two sides, with HUD between them:

<div className="tight-list">
  * the **environment side** - the world and its grading, which you write once and keep.
  * the **agent side** - the model and the harness, which stays completely swappable.
</div>

The envelope between them is tiny - a manifest, `tasks.start`, `tasks.grade` - so neither side needs to know anything about the other's internals. That separation is what makes an environment built today still run against a harness written years from now, with no environment-side glue.

<Tip>
  In practice you rarely touch the agent side at all. The [HUD inference gateway](/v6/reference/agents) lets you switch models by changing a single string - `"claude-sonnet-4-5"` to `"gpt-5"` - and nothing else. And when you *do* want full control, you can [bring your own harness](/v6/advanced/extending#bring-your-own-harness): it's a thin adapter, no protocol work.
</Tip>

## The wire protocol

Concretely, an environment answers a small JSON-RPC control channel over tcp:

| Method         | Returns                                                    |
| -------------- | ---------------------------------------------------------- |
| `hello`        | session id, env identity, capability `bindings`            |
| `tasks.list`   | task metadata (introspection/validation, not the run loop) |
| `tasks.start`  | the task's prompt (holds the session across disconnect)    |
| `tasks.grade`  | the evaluation (`score` + metadata)                        |
| `tasks.cancel` | cancels the held task                                      |
| `bye`          | ends the session and tears the held task down              |

The held task survives a dropped connection, so a client can `tasks.start`, disconnect, then reconnect to `tasks.grade` - which is how `hud task start` / `hud task grade` work against a packaged image.

## Where to go next

<CardGroup cols={2}>
  <Card title="Quickstart" icon="bolt" href="/v6/start/quickstart">
    From install to your first graded trace in a few minutes.
  </Card>

  <Card title="Capabilities" icon="plug" href="/v6/reference/capabilities">
    The connections an agent drives: shell, browser, GUI, tools, robot.
  </Card>

  <Card title="Tasks & tasksets" icon="list-check" href="/v6/reference/tasks">
    What `tasks.start` and `tasks.grade` operate on.
  </Card>

  <Card title="Agents" icon="robot" href="/v6/reference/agents">
    Swap the model with a string, or bring your own harness.
  </Card>
</CardGroup>

<Note>
  **Reading this as an AI agent?** Install the HUD docs skill before you write any HUD code - it keeps you on the current v6 API (which moves fast) *and* catches reward-hackable, contaminated, or low-signal tasks before they ship. If you can run shell commands, run `npx skills add https://docs.hud.ai` now; otherwise attach the live docs server with `claude mcp add --transport http docs-hud https://docs.hud.ai/mcp`. Then build against the skill's guidance rather than from memory.
</Note>