Progress Probes (P²)

Vision-Language-Action (VLA) models can perform diverse manipulation tasks directly from visual observations and natural language instructions, but they remain prone to mistakes and unpredictable failures. Deploying these policies without continuous human supervision will require robust runtime monitors that detect when the policy has gone off-course. Existing monitors use external foundation models, such as vision-language models (VLMs) to supervise the policy, introducing significant complexity and computational cost for monitoring at runtime. In this work, we investigate an alternate approach: can a VLA's own internal activations provide the information needed to monitor its execution? Our method trains linear probes on policy activations from demonstrations to estimate progress, then flags executions that deviate from expected progress. In experiments with the open-source VLA π_0.5 and across simulated and real-world manipulation tasks, this approach is competitive with or outperforms VLM-based monitors in task failure detection, as well as methods which require failure data for training. These findings suggest that simple probes of VLA internals can be used to develop reliable runtime monitors, opening a path toward building policies that can act reliably and monitor their execution without external foundation models.

No External Monitor

P² reads task progress from activations the VLA already computes, running 5×–20,000× faster than foundation-model monitors.

Success-Only Training

Progress labels come for free from successful demonstrations; no failure data or human annotation is needed.

Competitive Detection

AUROC of 0.788 on 6 VLABench tasks and 0.78 on 6 real-world tasks, matching or exceeding SAFE and Robometer.

Method

Our key insight is that a VLA's internal activations encode how far the policy has progressed through a task. In successful executions, this encoded progress advances predictably with elapsed time; when the policy fails or stalls, it falls behind. Progress Probes (P²) turn this signal into a runtime monitor in three steps:

Labeling progress. Each timestep of a successful demonstration is labeled with its normalized time remaining, ζ_t = 1 − t/T, which decreases from 1 at the start of the task to 0 at the end. These labels require no human supervision and no failure data.

Linear progress probes. We pass the demonstrations through π_0.5, mean-pool the activations zⁱ at a layer i across tokens, and fit a linear regression ζ̂ = w^⊤zⁱ + b to predict task progress.

Runtime monitoring. At deployment, the probe reads the activations already computed during inference and compares its estimate to the nominal progress 1 − t/μ_T, where μ_T is the mean successful trajectory length. A failure is flagged when the estimate lags behind nominal by more than a threshold δ; finishing ahead of schedule is not penalized.

What Makes a Good Runtime Monitor?

A runtime monitor watches the policy as it executes and raises an alarm when the rollout is unlikely to succeed. On a successful trajectory, the progress estimated by P² closely follows the expected progress and never enters the detection region; on a failed trajectory, it falls behind schedule and crosses the boundary.

We want a monitor that is accurate, fast to react, and practical to deploy alongside the policy:

Reliable discrimination. The monitor should separate successes from failures, firing on failed rollouts while keeping false alarms on successful ones rare. P² only flags a rollout once its estimated progress lags the nominal trajectory by more than a calibrated threshold.

Early detection. A failure flagged at the end of an episode is of little use; the monitor should fire as soon as the execution goes off-course, leaving time to intervene. Because P² scores every timestep, it reacts as soon as progress stalls.

No failure data required. Failures are diverse and costly to collect, so the monitor should be trainable from successful executions alone. P²'s progress labels come directly from successful demonstrations.

Lightweight at runtime. The monitor should keep pace with the policy without adding latency or relying on a large external model. A VLM-based monitor such as Robometer runs a separate 4B-parameter model at every check, whereas P² adds only a 2K-parameter linear readout on activations the VLA already computes during inference.

A 4B-parameter VLM monitor (Robometer) compared with a 2K-parameter linear probe on the VLA's own activations. — **Left:** P² replaces a separate 4B-parameter VLM with a 2K-parameter probe on the VLA's own activations. **Right:** failure-detection AUROC versus inference speed in simulation. P² runs in 0.026 ms per step on an RTX A6000, over 10,000× faster than Robometer, while remaining competitive with the most accurate monitors.

Failure-detection AUROC versus inference speed for runtime monitors; Progress Probe is the fastest while remaining among the most accurate. — **Left:** P² replaces a separate 4B-parameter VLM with a 2K-parameter probe on the VLA's own activations. **Right:** failure-detection AUROC versus inference speed in simulation. P² runs in 0.026 ms per step on an RTX A6000, over 10,000× faster than Robometer, while remaining competitive with the most accurate monitors.

Experiments

We evaluate P² on the π_0.5 VLA across 6 simulated VLABench tasks and 6 real-world manipulation tasks, comparing against runtime monitors that use image inputs, action outputs, VLA internals, and large reward models.

Detection Time

A useful monitor must catch failures early while rarely firing on successful rollouts. We calibrate each detector to a 5% false-positive rate and measure how far into the episode it first flags a failure. P² detects failures on par with state-of-the-art monitors, at 0.60 of the episode on average compared with 0.53 for SAFE-LSTM and 0.56 for Robometer-FT, even though it uses only the current activation and is trained without any failure labels.

Detector	TPR @ 5% FPR by episode fraction ↑				Det. time ↓ (frac. of ep.)
Detector	25%	50%	75%	100%	Det. time ↓ (frac. of ep.)
Progress Probe (P²)	0.00	0.19	0.54	0.64	0.60
SAFE-MLP	0.00	0.08	0.41	0.77	0.75
SAFE-LSTM	0.11	0.37	0.59	0.72	0.53
Robometer (progress)	0.01	0.08	0.21	0.33	0.67
Robometer FT (progress)	0.00	0.34	0.62	0.72	0.56

Simulation detection-time results. TPR is the fraction of failures caught by each point in the episode at a 5% false-positive rate; detection time is the average fraction of the episode elapsed before the first alarm on failed rollouts.

Discrimination

Beyond firing early, a monitor must reliably tell successful rollouts apart from failed ones. We report failure-detection AUROC at two levels: episode, which classifies a whole rollout from its maximum score, and state, which classifies every timestep. On hardware, P² achieves the highest episode- and state-level AUROC of all monitors (0.65 and 0.78), including SAFE and Robometer-FT, which are trained with additional failure data. In simulation, it remains competitive with these supervised baselines without using any failure labels.

We also test generalization by training each monitor on 3 of the 6 tasks and evaluating on the 3 held-out tasks, across all 20 possible splits. P² transfers well to unseen tasks, reaching a state-level AUROC of 0.81 on hardware, the best of all monitors, and 0.794 in simulation.

A failed real-world rollout. The P² failure score rises well before the labeled failure moment and stays high, while SAFE and Robometer-FT fluctuate.

Hardware episode-level and state-level failure-detection AUROC across all tasks; Progress Probe is highest at both levels. — Hardware episode- and state-level AUROC aggregated across all six tasks.

Per-task breakdown

Method	Block Stack	Knife Put	Stacked Utensil	Candy Bag	Coffee Bag	Coffee Bean	Average AUROC
FAIL-Detect (lopO)	0.40	0.46	0.59	0.67	0.49	0.42	0.54
FIPER (RND-OE)	0.61	0.82	0.74	0.50	0.60	0.52	0.61
FIPER (ACE)	0.61	0.68	0.62	0.50	0.60	0.50	0.57
STAC	0.59	0.57	0.37	0.31	0.57	0.34	0.45
SAFE-MLP	0.60	0.80	0.60	0.72	0.39	0.38	0.73
SAFE-LSTM	0.70	0.70	0.57	0.76	0.50	0.33	0.58
FAIL-Detect (logpZO)	0.47	0.94	0.79	0.90	0.72	0.74	0.69
Robometer (progress)	0.53	0.51	0.73	0.94	0.61	0.66	0.70
Robometer (success)	0.33	0.33	0.30	0.46	0.24	0.25	0.41
Robometer FT (progress)	0.63	0.56	0.88	0.95	0.66	0.53	0.75
Robometer FT (success)	0.32	0.43	0.28	0.39	0.20	0.18	0.34
Progress Probe (P²)	0.87	0.80	0.81	0.92	0.66	0.26	0.78

Hardware, all tasks: state-level failure-detection AUROC for each task. The rightmost column is AUROC across all six tasks.

Method	Block Stack	Knife Put	Stacked Utensil	Candy Bag	Coffee Bag	Coffee Bean	Average AUROC
FAIL-Detect (lopO)	0.51	0.66	0.54	0.55	0.60	0.49	0.56
FIPER (RND-OE)	0.40	0.83	0.76	0.62	0.51	0.56	0.61
FIPER (ACE)	0.61	0.68	0.62	0.50	0.60	0.50	0.59
STAC	0.59	0.57	0.37	0.31	0.57	0.34	0.46
SAFE-MLP	0.60	0.78	0.64	0.75	0.53	0.31	0.60
SAFE-LSTM	0.54	0.64	0.56	0.70	0.51	0.14	0.51
FAIL-Detect (logpZO)	0.48	0.80	0.77	0.83	0.50	0.75	0.69
Progress Probe (P²)	0.87	0.75	0.84	0.94	0.60	0.84	0.81

Hardware, unseen tasks: state-level failure-detection AUROC on held-out tasks. Robometer is excluded since it is used zero-shot, so every task is already unseen.

Method	Episode ↑	State ↑
FAIL-Detect (lopO)	0.881	0.857
FIPER (RND-OE)	0.874	0.850
FIPER (ACE)	0.793	0.790
STAC	0.477	0.442
SAFE-MLP	0.930	0.894
SAFE-LSTM	0.923	0.851
FAIL-Detect (logpZO)	0.904	0.888
Robometer (progress)	0.779	0.666
Robometer (success)	0.514	0.316
Robometer FT (progress)	0.922	0.830
Robometer FT (success)	0.505	0.407
Progress Probe (P²)	0.909	0.788

Simulation, all tasks: episode- and state-level failure-detection AUROC over 600 rollouts on 6 VLABench tasks.

Method	Episode ↑	State ↑
FAIL-Detect (lopO)	0.667	0.656
FIPER (RND-OE)	0.661	0.557
FIPER (ACE)	0.843	0.800
STAC	0.482	0.434
SAFE-MLP	0.896	0.866
SAFE-LSTM	0.859	0.803
FAIL-Detect (logpZO)	0.634	0.571
Progress Probe (P²)	0.900	0.794

Simulation, unseen tasks: AUROC on held-out tasks, averaged over all 20 seen/unseen splits.

Does Action Data Matter?

Unlike VLMs, which are trained only on images and text, VLAs are additionally trained on robot actions. To test whether this action-grounded training produces representations that better encode task progress, we train the same progress probe on the activations of four models: PaliGemma (the 3B VLM backbone of π_0.5), Robometer (a 4B reward model trained on robot videos) both zero-shot and fine-tuned, and the π_0.5 VLA itself.

Action data makes the difference. Probing π_0.5 gives the most accurate progress prediction (R² = 0.83, vs. 0.69–0.76 for the VLMs) and the strongest failure detection (AUROC 0.93). Seeing robots in training is not enough on its own: zero-shot Robometer, despite its robot-video training, does no better than PaliGemma. Learning to predict actions appears to shape internal representations that better capture whether the policy is on track.

Progress predictions and summary bars for probes trained on PaliGemma, Robometer, Robometer (FT), and the VLA; the VLA probe has the highest R-squared (0.83) and failure-detection AUROC (0.93). — Top: progress predicted by probes on each model's activations over successful simulation rollouts (grey) against nominal progress (dashed). Bottom: progress-prediction R² and failure-detection AUROC.

A successful rollout. The probe on π_0.5 follows nominal progress most closely, while the VLM probes drift and fluctuate.

BibTeX

@misc{bhardwaj2026progressprobesp2lightweight,
  title={Progress Probes $P^2$: Lightweight Runtime Monitoring for Vision-Language-Action Models},
  author={Atiksh Bhardwaj and Edward W. Duan and Junwon Seo and Prithwish Dan and Wei-Chiu Ma and Andrea Bajcsy and Preston Culbertson},
  year={2026},
  eprint={2610.31613},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2610.31613},
}