Skip to main content
relay.Policy connects to a policy server and works like OpenPI’s WebsocketClientPolicy: you send an observation dict and get a dict back with the actions in it.
Use one Policy for the whole run, and close it when you’re done, or use it in a with block:

The observation

dict[str, image]
required
One frame per camera, by the model’s camera names: for π0.5, base_0_rgb, left_wrist_0_rgb and right_wrist_0_rgb. A frame is an (H, W, 3) uint8 RGB array, of any size: the server resizes it to 224×224, keeping its aspect ratio and padding the short side. Cameras you leave out are masked out.
float array
required
The robot’s state, (32,) for π0.5, as the model was trained on it (normalized, padded with zeros).
str
required
The task in plain language. The server tokenizes it as the model was trained. Code that sends OpenPI’s tokenized_prompt and tokenized_prompt_mask instead still works.
dict[str, bool]
Which cameras are present. By default, the ones in image.
int
Denoising steps for this request. Fewer are faster and slightly less precise. By default, the server’s (get_server_metadata()["num_steps"]).
str
A LoRA adapter the server holds, for this request. By default, the server’s.

The reply

np.ndarray
(action_horizon, action_dim) float32: the next actions, in the model’s units (as its training data was normalized). actions[0] is the one to take now.
dict
infer_ms: time in the server’s engine, queued and run.
dict
infer_ms, parse_ms (unpacking and decoding the request) and, from the second request on, prev_total_ms (the last round trip as the server saw it).

What the policy takes

get_server_metadata() returns what the server sent when you connected:

Sending images

Frames are compressed before they’re sent. Pick the codec when you connect:
JPEG frames are smaller, so they upload faster on a slow connection; the server decodes them on its GPU. If your camera already gives you JPEG bytes, put them in image as they are, with a JPEG codec.

Errors

If the server fails a request (a wrong state length, say), infer raises RuntimeError with the server’s traceback, and the server closes the connection. Connect again to go on. A call on a closed connection raises websockets.exceptions.ConnectionClosed.

Options

relay.Policy(url, codec=Codec.JPEG_Q90, headers=None)
str
required
The policy server’s websocket URL, wss://....
relay.Codec
default:"Codec.JPEG_Q90"
How frames are sent.
dict[str, str]
Extra headers for the connection, such as a proxy’s credentials.
Connecting waits as long as the server takes to start, which can be minutes on a cold one.