Method overview
FRED builds on recent WAMs that keep the pre-trained video diffusion model (VDM) $\mathbf{v}_\theta$ unchanged and use a separate diffusion-based action decoder $\pi_\phi(\mathbf{a}_t \mid \mathbf{q}_t, \mathbf{h})$, where $\mathbf{h}$ is an intermediate hidden state of $\mathbf{v}_\theta$ obtained through a single forward pass from Gaussian noise $\boldsymbol{\epsilon}$. This avoids iterative video denoising at inference, but policy performance now depends on how well $\mathbf{h}$ captures the future.
Observation. Video model learns future representations during WAM training. Using DiT4DiT as the base WAM on RoboCasa Kitchen, we probe $\mathbf{h}$ generated from noise with a lightweight attentive probe that classifies future motion primitives, and measure its structural similarity to representations of clean future observations with CKNNA. Both increase throughout WAM training, showing that the video model progressively learns to generate future representations from noise, but a clear gap to clean-future representations remains.
We therefore interpret the video diffusion model as a future representation generator, $\mathbf{h} = \mathbf{f}_\theta(\boldsymbol{\epsilon} \mid \mathbf{o}_t, \mathbf{c})$, where $\mathbf{f}_\theta$ denotes the first layers of $\mathbf{v}_\theta$ that output $\mathbf{h}$ given the observation $\mathbf{o}_t$ and instruction $\mathbf{c}$, and FRED improves $\mathbf{h}$ with two components: self-distillation from noisy future observations, and a short observation history as temporal context. As shown above, FRED reduces the probing gap and further improves the structural similarity.
Self-distillation from noisy futures. The student $\mathbf{f}_\theta$ generates $\mathbf{h}$ from noise, while a momentum teacher $\mathbf{f}_{\theta^{-}}$ processes a lightly corrupted ground-truth future observation $\mathbf{o}^\tau_{t+1} = (1-\tau)\,\mathbf{o}_{t+1} + \tau\,\boldsymbol{\epsilon}$ under the same conditioning to produce the target $\mathbf{h}^\tau$:
$\mathbf{h} = \mathbf{f}_\theta(\boldsymbol{\epsilon} \mid \mathbf{o}_{t-H:t}, \mathbf{c}) \quad \text{and} \quad \mathbf{h}^\tau = \mathbf{f}_{\theta^{-}}(\mathbf{o}^\tau_{t+1} \mid \mathbf{o}_{t-H:t}, \mathbf{c})$
Patch-wise projection heads $\mathbf{g}_\theta, \mathbf{g}_{\theta^{-}}$ map both representations to distributions; the teacher logits are centered by the mean teacher logits $\mathbf{m}$ and sharpened with a lower temperature $\gamma_2$ to prevent representation collapse. The self-distillation loss is a per-patch KL divergence:
$\mathbf{p} = \mathrm{softmax}\!\left(\mathbf{g}_\theta(\mathbf{h}) / \gamma_1\right), \quad \mathbf{p}^\tau = \mathrm{softmax}\!\left((\mathbf{g}_{\theta^{-}}(\mathbf{h}^\tau) - \mathbf{m}) / \gamma_2\right), \quad \mathcal{L}_{\mathrm{SD}}(\theta) = \frac{1}{N} \sum_{i=1}^{N} \mathrm{KL}\!\left(\mathbf{p}^\tau_i \,\|\, \mathbf{p}_i\right)$
History as temporal context. Both student and teacher are conditioned on a short history $\mathbf{o}_{t-H:t}$ rather than only the current frame, which provides motion cues (e.g., where the arm is heading) that disambiguate the future.
Training objective. The video model is trained with the video flow-matching loss plus self-distillation, and the action decoder with its own flow-matching loss. The action loss does not update the video model, so the representation generator $\mathbf{f}_\theta$ is trained only by the video generation and self-distillation objectives; the teacher is the EMA of the student, $\theta^{-} \leftarrow m\theta^{-} + (1-m)\theta$.
$\mathcal{L}_{\text{video}}(\theta) = \mathcal{L}_{\mathrm{FM}}(\theta) + \lambda_{\mathrm{SD}}\,\mathcal{L}_{\mathrm{SD}}(\theta) \quad \text{and} \quad \mathcal{L}_{\text{action}}(\phi) = \mathcal{L}_{\mathrm{FM}}(\phi)$
