RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures

Hanan Gani, Tejal Kulkarni, Madhoolika Chodavarapu, Nicklas Hansen, Manmohan Chandraker
University of California, San Diego
ECCV 2026
📄 Paper arXiv ⌘ Code ⬇ Checkpoints
RoboTALES overview: a planner guides future video imagination, a critic scores alignment, and an action policy controls the robot.

RoboTALES converts language instructions into task-aligned simulated futures, then learns actions from those futures through joint world-model and policy training.

Abstract

Pretrained video generative models are promising backbones for visuomotor control, but their imagined futures often drift from task intent and are not reliably action-conditional. RoboTALES learns task-aligned simulated futures for robot policy learning by combining three signals: a hierarchical LLM planner that decomposes instructions into subgoals, a frozen vision-language critic that scores imagined rollouts, and a single-stage training objective that jointly optimizes the video generator and action diffusion policy.

By aligning imagination with both language-level reasoning and action-level feedback, RoboTALES improves long-horizon manipulation on RoboCasa and LIBERO-10, achieving strong performance with only 50 demonstrations per task.

Highlights

  1. We introduce a reasoning-guided policy learning framework that uses an LLM planner to decompose long-horizon instructions into subgoals for video-conditioned control.
  2. We steer imagined futures with a VLM-based critic, keeping generated rollouts aligned with the task instruction rather than drifting into plausible but irrelevant futures.
  3. We jointly optimize the video generator and action policy in a single stage, allowing action gradients to improve the representations used for future prediction.
  4. RoboTALES achieves strong manipulation results on RoboCasa and LIBERO-10, including 64% average RoboCasa success and 97% average LIBERO10 success under the paper protocol.

RoboTALES Architecture

RoboTALES couples a hierarchical planner, Stable Video Diffusion world model, frozen VLM reward critic, and action diffusion policy. The planner provides subgoal-conditioned task structure, the video generator imagines short-horizon futures, the critic rewards task-aligned predictions, and the action policy converts video-model features into executable controls.

RoboTALES architecture figure

Qualitative Rollouts

RoboTALES produces task-aligned rollouts across pick-and-place, appliance, drawer, and coffee-making tasks in RoboCasa.

Move the object from the counter to the microwave.
Turn off the sink faucet.
Move the object from the stove to the counter.
Place the mug under the coffee machine spout.
Set up the mug for coffee preparation.
Open the drawer.

Quantitative Results

RoboTALES improves long-horizon robot manipulation by aligning future prediction with task reasoning and control.

64% RoboCasa average success
97% LIBERO10 average success
48% RoboCasa pick-and-place success
Impact of the planner on task success.
Impact of the planner on structured subgoal reasoning.
Demonstration efficiency results.
Performance as the number of demonstrations changes.

Released Checkpoints

Trained RoboCasa and LIBERO10 checkpoints are available on Hugging Face.

huggingface.co/hanangani/robotales-ckpts

BibTeX

@inproceedings{gani2026robotales,
  title={RoboTALES: Learning Reasoning-Guided Robot Policies via Task-Aligned Simulated Futures},
  author={Gani, Hanan and Kulkarni, Tejal and Chodavarapu, Madhoolika and Hansen, Nicklas and Chandraker, Manmohan},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026}
}