World Action Models (WAMs) fail under camera shifts, gripper-position changes, and sensor noise. We find that some WAMs (Cosmos-Policy, DiT4DiT) encode these failure modes as simple, low-dimensional linear directions in activation space — while others (LingBot-VA) do not. We use this mechanistic signal to predict which models are steerable, then exploit it with WA-LQR, a training-free, closed-loop LQR controller that nudges WAM activations toward robust behavior at inference time — improving task success rates by up to +41% with no finetuning.
World Action Models (WAMs) enable semantically- and physically-informed control but are brittle under distribution shift. In this work, we use mechanistic interpretability to study how robustness-relevant perturbations are represented in WAM activation space. Comparing activations across successful and unsuccessful rollouts, we find some WAM architectures exhibit low-dimensional linear separability for robustness-critical features, while others do not. This motivates the use of contrastive activation directions for training-free WAM steering. We also show that local linearity in WAM activation dynamics enables efficient feedback steering via model-based optimal control, yielding World-Action Linear Quadratic Regulator (WA-LQR), a minimally-invasive reduced-order LQR controller.
Via mechanistic evaluations, we predict strong steerability in the Cosmos-Policy and DiT4DiT models but weak steerability in LingBot-VA, consistent with steering intervention results. On Cosmos-Policy and DiT4DiT, WA-LQR generalizes contrastive directions to new tasks and improves robustness to camera, gripper, and visual-noise perturbations over unsteered and prompt steering baselines.
Steerability is architecture-dependent
We compare WAM activations from nominal vs. perturbed rollouts (camera pose, gripper position, image noise), project the contrastive directions onto their top-3 principal components, and fit a linear SVM. The hinge loss quantifies separability: 0 means perfectly linearly separable, 1 means random.
Training-free activation steering for WAMs
Given paired rollouts that differ mainly in a robustness-relevant feature (e.g. nominal vs. perturbed camera pose), we compute contrastive activation directions and use them to steer WAM inference — with no finetuning and no weight updates.
Activation Addition (ActAdd)
Average contrastive directions into a single steering vector, then add it at every denoising step with a fixed strength γ. Simple and training-free, but ignores the model's current activation state — it can oversteer or degrade the action.
xl,t,τ ← xl,t,τ + γ al,tWorld-Action LQR (WA-LQR)
Project activations into a low-dimensional contrastive subspace, treat the DiT block sequence as a locally-linear dynamical system, and solve a per-timestep LQR problem to track a target feature setpoint — steering only as much as the realized activation deviates from it.
- Project — zl,t,τ = Pl,t xl,t,τ ∈ ℝdz, dz ≪ dx
- Linearize across transformer blocks via Jacobian-vector products
- Track a feature setpoint with online error αl,t,τ
- Solve LQR offline via Riccati recursion → gains Kl,t,τ
- Apply online: u*l,t,τ = ūl,t,τ + Kl,t,τ αl,t,τ vzl,t
Steering improves LIBERO-10 robustness — when the feature is separable
We evaluate WA-LQR and ActAdd against unsteered and prompt-steering baselines on LIBERO-10 tasks under camera-orientation, initial-gripper-position, and Gaussian-noise perturbations (30 trials/task).
Cosmos-Policy 2B
DiT4DiT
LingBot-VA
Unsteered failures vs. WA-LQR steered successes
Paired rollouts from the same task and perturbation, with and without WA-LQR steering. In every pair, the unsteered policy fails and the WA-LQR-steered policy succeeds.
Cosmos-Policy 2B
Camera OrientationDiT4DiT
Initial Gripper PositionLingBot-VA
Initial Gripper PositionNote: LingBot-VA showed weak activation separability, so steering gains are small and inconsistent overall — this pair illustrates a favorable case rather than the typical outcome.
BibTeX
arXiv
@article{hong2026steering,
title = {Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control},
author = {Hong, Jihoon and Skifstad, Julian and Dai, Qiyue and Chan, Alice and Chou, Glen},
journal = {arXiv preprint arXiv:2607.14943},
year = {2026}
}
RSS Robot World Models Workshop
@inproceedings{skifstad2026steering,
title = {Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Control},
author = {Skifstad, Julian and Hong, Jihoon and Dai, Qiyue and Chan, Alice and Chou, Glen},
booktitle = {Robot World Models},
year = {2026},
url = {https://openreview.net/forum?id=K691Pk5J1f}
}