Where training goes

π0.5 has three parts: SigLIP reads the cameras (vision), PaliGemma reads the images and the prompt (language), and the action expert turns PaliGemma’s keys and values into actions (action_expert). Both training APIs take targets, which say where training goes, and a method:

Method	Trains
LoRA (`finetune="lora"`)	Low-rank adapters on every module a target names, and the action heads. The base weights stay frozen.
Full (`finetune="full"`)	Every weight of the targets.

Targets

Target	LoRA adapts	Full fine-tuning trains
`"all"`	every module below	every weight of the model
`"vision"`	`vision.attn`, `vision.mlp`	all of SigLIP
`"language"`	`language.attn`, `language.mlp`	all of PaliGemma, its embeddings too
`"action_expert"`	the expert’s `q_proj`, `k_proj`, `v_proj`, `o_proj`, `mlp`	the expert and its action heads
`"vision.attn"`, `"vision.mlp"`	every SigLIP layer’s attention, or MLP	those weights
`"language.attn"`, `"language.mlp"`	every PaliGemma layer’s	those weights
`"action_expert.attn"`	the expert’s `q_proj`, `k_proj`, `v_proj`, `o_proj`	those weights
`"action_expert.q_proj"`, …, `"action_expert.mlp"`	that module	its weights

Combine them freely: ["language.mlp", "action_expert"] puts LoRA on PaliGemma’s MLPs and the whole expert. In Python they’re relay.sft.TARGETS.

Examples

from relay.sft.config import SFTConfig

SFTConfig(model="pi05", dataset_name=..., finetune="full")                                   # the whole model
SFTConfig(model="pi05", dataset_name=..., finetune="full", targets=["action_expert"])        # one part
SFTConfig(model="pi05", dataset_name=..., targets=["all"], lora_rank=16)                     # LoRA everywhere
SFTConfig(model="pi05", dataset_name=..., targets=["language.mlp", "action_expert.attn"])    # LoRA at select places

from relay.training import ServiceClient

service = ServiceClient()
service.create_full_training_client("pi05")                                                   # the whole model
service.create_full_training_client("pi05", targets=["action_expert"])                        # one part
service.create_lora_training_client("pi05", rank=16, targets=["all"])                         # LoRA everywhere
service.create_lora_training_client("pi05", targets=["language.mlp", "action_expert.attn"])   # LoRA at select places

Defaults

LoRA goes on the action expert (["action_expert"]), as π0.5’s reference fine-tuning does; full fine-tuning trains the whole model (["all"]).

Sizes

Measured on one H100, batch of 4, one camera:

Training	Trainable parameters	A step
LoRA rank 16 at `["language.mlp", "action_expert.attn"]`	20M	0.3 s
LoRA rank 16 at `["all"]`	35M	0.5 s
Full, `["all"]`	3.35B	0.4 s

Training anything in vision or language runs SigLIP and PaliGemma with gradients, recomputing each layer in the backward pass, so those steps use more memory and time than training the expert alone. Lower batch_size if a large one runs out of memory. Trained weights are exported with the adapters merged in, so a LoRA fine-tune serves like any other checkpoint.

Get started

Inference

Training

Reference

Where training goes

Targets

Examples

Defaults

Sizes