Overview of VISION-SLS

Abstract

We propose VISION-SLS, a method for nonlinear output-feedback control from high-resolution RGB images which provides robust constraint satisfaction guarantees under calibrated uncertainty bounds despite partial observability, sensor noise, and nonlinear dynamics. To enable scalability while retaining guarantees, we propose: (i) a learned low-dimensional observation map from pretrained visual features with state-dependent error bounds, and (ii) a causal affine time-varying output-feedback policy optimized via System Level Synthesis (SLS). We develop a scalable, novel solver for the resulting nonconvex program that leverages sequential convex programming coupled with efficient Riccati recursions. On two simulated visuomotor tasks (a 4D car and a 10D quadrotor) with $\ge 512\times 512$ pixels and a 59D humanoid task with partial observability, our method enables safe, information-gathering behavior that reduces uncertainty while guaranteeing constraint satisfaction with empirically-calibrated error bounds. We also validate our method on hardware, safely controlling a ground vehicle from onboard images, outperforming baselines in safety rate and solve times. Together, these results show that learned visual abstractions coupled with an efficient solver make SLS-based safe visuomotor output-feedback practical at scale.

Video

Method Overview

Overview of VISION-SLS

VISION-SLS compresses high-dimensional RGB observations into low-dimensional measurements using pretrained DINO features followed by a learned projection. The reduced observation model $h^r(x)$ is paired with a calibrated, state-dependent error bound $F^r(x)$ that overbounds sensor noise and representation error. At runtime, these reduced measurements are passed to a dynamic SLS output-feedback controller, which combines the jointly optimized nonlinear nominal trajectory $(\boldsymbol{z}, \boldsymbol{v})$ with feedback gains $K$ to produce safe control inputs under uncertainty.

Results

4D Car with Overhead camera

Overview of VISION-SLS

In the occluded parking-lot task, a 4D car must reach a goal from overhead RGB observations while avoiding obstacles and satisfying state and input limits. VISION-SLS plans through lower-uncertainty regions, tightens constraints using the propagated reachable tube, and keeps closed-loop rollouts collision-free, yielding smaller terminal uncertainty than certainty-equivalent or non-robust baselines.

Humanoid with Partial State Measurement

For the 59-state, 23-input Unitree G1 backflip task, VISION-SLS synthesizes a robust closed-loop plan from partial state measurements. The closed-loop rollouts stay inside the computed reachable tubes and satisfy constraints, while open-loop execution of the nominal trajectory can leave the tubes and violate limits under disturbance.

Hardware

On a physical TurtleBot 4 in an office environment, VISION-SLS uses onboard RGB feedback to track the planned trajectory while respecting input limits and collision-avoidance constraints. Across real-world runs, the robot remains inside the computed reachable tubes, demonstrating safe perception-based output-feedback control on hardware.

BibTeX

@inproceedings{LeemanVISIONSLS,
  title={VISION-SLS: Safe Perception-Based Control from Learned Visual Representations via System Level Synthesis},
  author={Leeman, Antoine P., Zhan, Shuyu, Zeilinger, Melanie N., and Chou, Glen},
  booktitle={Proceedings of Robotics: Science and Systems (RSS)},
  year={2026}
}