Why Robots Borrow From Language Models
Vision-language models (VLMs) are trained on web-scale data, so they carry broad world knowledge and can reason about what they see. The idea behind a vision-language-action (VLA) model is to start from such a VLM and teach it to control a robot. The robot then inherits semantic understanding, such as recognizing objects it never saw in robot training data.
The catch is that a VLM only produces discrete tokens (text), while a robot needs precise, continuous motor commands. The authors use a brain analogy. The VLM’s vision encoder is like a visual cortex, its language model is like a prefrontal cortex, and a VLA needs a third piece, a “motor cortex.” In AI the order is backwards compared with biology. We start with language, add vision, and only then bolt on motor control. The question is how to attach it without damaging the capabilities already there.
Two Generations of VLAs
First generation (RT-2, OpenVLA). These models output actions as tokens. Each robot joint angle is split into fixed-size bins, and each bin gets a token, so predicting an action looks like answering a question with numbers. This works for basic tasks such as picking up objects. The authors report that it is poorly suited to precise, fluent, high-frequency motion, and that it is slow at runtime. Their analogy is controlling your arm by verbally naming which muscles to contract. The paper also notes that autoregressive VLAs of this type typically run at control frequencies below 2 Hz.
Second generation (π0 and similar). These models add a new neural module that produces continuous actions, usually with diffusion or flow matching. The module is called an action expert or an action head. It can read the VLM backbone’s internal representations but specializes in motor control. This enables much more complex tasks, such as folding laundry or making a bed.
Actions as tokens
Continuous actions with an action expert
Protect backbone knowledge using stop-gradient
Knowledge Insulation
When you attach an action expert to a pretrained VLM and train both end to end, the backbone receives two kinds of gradients. One comes from whatever language or token loss you keep. The other comes from the expert’s continuous-action loss, and that one is the problem. The expert’s weights are newly added and untrained, and the authors report that their gradients create unfavorable learning dynamics in a backbone whose representations were tuned on web-scale data.
The authors report two primary effects:
Their illustration is a spoon-into-dish-container task, where the naive model grasps the trash instead. They offer a hypothesis, not a proven mechanism. The pretrained backbone attends to language well, the expert’s gradients interfere with that, and the model picks up other correlations in the robot data first.
Three Mechanisms
Knowledge insulation (KI) combines three changes. The first two form a chain: stopping the gradient protects the backbone but cuts off its robot signal, and a token loss supplies that signal. The third adds more training data to reinforce the backbone’s knowledge.
4.1 Stop the gradient.
The expert still reads the backbone’s representations, but its loss cannot update the backbone. This fully insulates the backbone. By itself it is not enough. The backbone now never sees robot data, so its representations are not shaped for motor control, and the expert has a harder time producing correct motions. The authors show this in their comparisons.
4.2 Give the backbone its own robot-relevant objective.
The backbone is trained with next-token prediction on discretized actions in the FAST format. The expert’s loss no longer carries robot signal into the backbone, and this loss takes over that job. The authors report that FAST tokens make the model acquire motor representations significantly faster, and that a cross-entropy loss interferes much less with pretrained knowledge than the expert’s gradients do.
FAST is a compression-based tokenization scheme for robot actions, built on the discrete cosine transform (see the FAST paper in Sources). Its tokens give a better learning signal than simply binning each joint value.
4.3 Co-train on other next-token data.
The backbone already has a token loss, so more token tasks can be added without changing the architecture. The model is trained on the full π0.5 mixture, which includes general vision-language data from the web and high-level robot commands (planning data). The goal is to keep reinforcing the semantic knowledge in the backbone.
| Data Stream | Loss Type | Parameters Receiving Gradient |
|---|---|---|
| Robot trajectories, as FAST tokens | Cross-entropy (next token) | Backbone |
| Web vision-language data | Cross-entropy (next token) | Backbone |
| High-level robot commands (planning) | Cross-entropy (next token) | Backbone |
| Robot trajectories, as continuous actions | Flow matching | Action expert only |
4.4 One model, two paths.
Results: Training Speed, Execution Time & Generalization
Training speed.
On the bussing task with a generalist model, π0.5 + KI needed 7.5 times fewer training steps than π0 to reach similar performance, and it trained about as fast as π0-FAST.
Execution time.
On the single-embodiment bussing task, π0-FAST needed about twice the time of π0.5 + KI to finish, because of autoregressive inference. π0.5 + KI had the highest performance and followed language well.
| Variant | How it differs from π0.5 + KI |
|---|---|
| π0 | Naive action expert; gradients flow into the backbone |
| π0-FAST | No action expert; actions are autoregressive tokens |
| Joint training | Token and expert losses together, with no stop-gradient |
| Frozen backbone | The backbone gets no robot training |
| π0.5 + KI without VLM data | Same recipe, without co-training on VLM (web) data |
Manipulation tasks & Object Generalization
Shirt folding: π0.5 + KI had the highest average success. π0-FAST was close behind, and its printed p-value (p=0.782) shows no meaningful difference from π0.5 + KI. π0 (p=0.002) and the frozen backbone (p=0.001) were clearly lower. Freezing the backbone is not workable for this task.
Items in drawer (static robot, unseen environment): π0.5 + KI had the highest task completion. The printed p-values were 0.049 for joint training and 0.030 for π0-FAST. For π0 and the frozen backbone they were below 0.001.
Practical Takeaways for Engineers
These are interpretations, not claims from the paper.
Limitations
- The results come from specific robot tasks (bussing, shirt folding, items in drawer, object grasping). Do not generalize them to all manipulation.
- The explanation for the interference is stated as a hypothesis.
- This article covers the May 2025 work as of November 2026. It does not cover later models.
Where This Is Going
The authors expect the next generation of VLAs to integrate web-scale knowledge with motor control more deeply, with more sophisticated sequential reasoning, planning, and deliberate, goal-directed task execution. They describe VLAs as among the most multimodal models in existence, because of the variety of robot sensors and actuators, and they treat training them as a branch of multimodal machine learning. They point to a better understanding of learning dynamics as a key step toward that.
