Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning

1. The Problem: Reason Before the Future Arrives

Most video understanding benchmarks are retrospective. The evidence needed to answer the question is already somewhere in the clip. Proactive video reasoning is different: the model must make a useful prediction from an incomplete observation.

For proactive systems, being correct is not enough. A prediction is less useful if it comes only after the event has ended.

This latency constraint makes the form of reasoning important. A slow “thinking” procedure can erase the benefit of a more accurate model.

2. Why Not Just Generate the Future?

Predictive Visual Chain-of-Thought follows an intuitive pipeline:

observed video → generated future image → re-encoded visual tokens → textual prediction.

That explicit future frame can carry information that language is bad at compressing: object state, spatial arrangement, motion direction, contact relationships, and fine visual details. But it also adds a dense generation problem to every inference request.

The oracle-frame diagnosis

We run a control experiment here. Instead of asking only whether generated future frames help, we separate two questions:

1. Is future visual information useful?
2. Can a generated image deliver that information faithfully and cheaply enough?

To answer the first question, we replace the generated future frame with the ground-truth future frame at the same checkpoint. This “oracle” is impossible at deployment, but diagnostically clean.

This suggests a different use for the future: do not sample it at inference; use it as a training target that shapes the representation from which the language model answers.

3. What Makes Predictive Supervision Work?

3.1 What should the model predict?

We compare four target spaces:

• Flux-VAE latents: reconstruction-oriented; retain fine-grained appearance and spatial information.
• DINOv2 features: semantic, higher-level visual features.
• SigLIP2-adaptive: semantic target encoder that can be jointly optimized through the text objective.
• SigLIP2-frozen: a stationary semantic target space.

The result is notable: reconstruction-oriented VAE targets beat the more semantic alternatives. DINOv2 still helps, so high-level semantic prediction is useful.

3.2 Predictive supervision cannot be an occasional regularizer

Next, we vary how frequently the model sees language-understanding versus future-prediction examples. Ratios are understanding:prediction = 1:1, 3:1, and 5:1.

The 3:1 and 5:1 mixtures look better early. This is easy to understand: more answer-supervision updates accelerate adaptation to the downstream text target. But by roughly 12–18k steps they saturate or decline. The 1:1 mixture starts slower and finishes much stronger.

At 20k steps, the balanced mixture reaches ROUGE-L 34.5, BERTScore 36.5, METEOR 27.4, and CIDEr 2.02 on the Ego-Exo4D early-event ablation.

This suggests that future prediction is not behaving like a tiny regularization term. It needs enough optimization mass to keep influencing the shared representation instead of being overwritten by task-specific text supervision.

3.3 The best predictive loss depends on the target space

For DINOv2, direct regression consistently dominates flow matching. For Flux-VAE, the gap is much smaller: regression often leads during training, while flow matching catches up on some metrics at the final checkpoint.

The practical lesson is that representation and objective should be selected jointly. A target inherited from a generative model does not automatically imply that its original generative training objective is best for transferring predictive structure into a reasoner.

3.4 Joint training beats “predict first, reason later”

A natural alternative is a two-stage curriculum:

Stage 1: learn next-embedding prediction → Stage 2: fine-tune only on text answers.

It fails surprisingly badly.

The two-stage model is worse than never doing predictive training at all. This is a useful negative result: the benefit is not merely “better pretraining.” It comes from continuously forcing the representation used for answer generation to also carry predictive visual structure.

3.5 Shared decoder or separate experts?

We study two ways to couple the objectives:

• Dense: understanding tokens and predictive tokens update the same decoder parameters.
• MoE: the two token types use separate feed-forward experts, selected deterministically by token type.

Dense wins most clearly when the future target is short and relatively constrained. At H=2, for example, Dense reaches ROUGE-L 37.4 versus 33.8 for MoE. But Dense degrades as more future frames are predicted, while MoE is less sensitive and tends to peak around H=3.

The key takeaways are:

A shared decoder allows gradients from next-embedding prediction to directly shape the representations used for answer generation, which is consistent with its clear advantage at H=1 and H=2. As the horizon increases, however, future prediction becomes more ambiguous, and tightly coupling this objective to the language decoder may introduce less task-aligned supervision. Separating the objectives through MoE appears to reduce this sensitivity, although it does not match the strongest short-horizon Dense configuration.

4. Main Results: Quality Without the Inference Rollout

We use the same BAGEL-7B-based initialization and matching training setup for Answer-Only SFT, Text CoT, Visual CoT, and IVT. At inference, IVT follows the same direct observed-video → text path as Answer-Only SFT. Across the three datasets and two tasks, IVT improves all four metrics over Answer-Only SFT in every setting.

The pattern is revealing. Explicit Visual CoT remains strong for two early-event settings, where the action is already unfolding and generating a near-future frame may be easier. But on all three next-event datasets—where the target event has not begun—IVT is stronger.

Across the three next-event benchmarks, we report average ROUGE-L increasing from 45.9 for Visual CoT to 49.7 for IVT.

Inference efficiency

The quality result is only half the story. IVT reformulates visual foresight from an explicit inference-time operation into the model’s internal representations.

Measured end-to-end on a single NVIDIA B200 at batch size 1 and bf16, Visual CoT averages 6.56 seconds per sample across the six settings, versus 1.20 seconds for Answer-Only SFT and 1.22 seconds for IVT. The mean P95 latency is 7.92 s, 1.92 s, and 1.77 s, respectively.

5. Limitations and Open Questions

5.1 One future target versus many plausible futures

Directly regressing toward one observed future frame is well behaved when the horizon is short. As the horizon grows, the conditional future distribution becomes multimodal. A deterministic feature target may average incompatible futures, while flow matching or other distributional objectives may become more important.

5.2 What should be internalized?

We study frame-level latent targets. Other possibilities include object-centric state, depth, optical flow, contact maps, segmentation, action-conditioned state transitions, or learned dynamics tokens. The choice of target representation appears to matter substantially, and there is still no clear answer to what information is most useful to internalize for proactive reasoning.

We also experimented with V-JEPA 2.1 features as the predictive target, motivated by their strong video-centric representation learning objective. In our preliminary experiments, however, we did not find V-JEPA 2.1 targets to provide a clear improvement for these proactive reasoning tasks. This negative result reinforces an important point: a representation that is strong for video understanding or predictive pretraining is not automatically the best supervision target for transferring future information into an MLLM's reasoning pathway.

5.3 Does the gain survive stronger base models?

Our controlled comparisons are intentionally centered on matched BAGEL-based systems. We also report external 7B–8B baselines, but those differ in pretraining, backbone, and optimization. A natural next step is to test whether internalized predictive supervision scales with much stronger multimodal foundation models.

5.4 Can latent prediction support test-time reasoning too?

IVT deliberately removes the prediction branch at inference. That is its efficiency advantage. But there may be a continuum between “no rollout” and “full pixel rollout”: a model could perform a small number of compressed latent-state predictions or updates only on hard examples, instead of generating full-resolution future images. Such compressed latent reasoning could provide adaptive test-time compute while retaining much of IVT's efficiency advantage and avoiding expensive image synthesis and re-encoding.

5.5 Evaluation of open-ended futures

Top-5 decoding partially acknowledges future uncertainty, but lexical metrics still compare generated descriptions to finite annotations. More fundamentally, we found that rigorous evaluation of multiple plausible futures is itself difficult. Given the same partial observation, different human annotators can anticipate different next events, and even stronger LLM-based judges may produce different yet reasonable expectations about what happens next.

This makes a single-reference notion of correctness inherently incomplete. Better evaluations may need semantic sets of valid futures, calibrated uncertainty, multiple independent annotations, and decision-oriented metrics that ask whether a prediction is plausible and useful rather than merely textually similar to one annotated continuation.

Longer-horizon anticipation is substantially harder

We also evaluate a more challenging setting in which the model must anticipate an action three hops into the future (hop = 3). In this regime, we do not observe a clear benefit from IVT over the corresponding answer-only baseline.

This result suggests that IVT is most effective for short-horizon anticipation, where future events remain strongly grounded in the current visual evidence—for example, ongoing motion, hand–object interactions, object state, and local scene dynamics. As the prediction horizon grows, future actions depend increasingly on latent goals, higher-level plans, and multiple branching possibilities that may not be recoverable from visual input alone.

In other words, internalizing near-future visual dynamics appears useful when the future is still tightly coupled to what the model can currently see. For more distant events, stronger modeling of intent, procedure, memory, or multimodal world knowledge may be required in addition to visual prediction.

References

1. Zhu, X. et al. (2026). Beyond Visual CoT: Internalized Visual Thinking for Proactive Video Reasoning. arXiv:2608.15869.
2. Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
3. Zhao, H. & Wildes, R. (2021). Review of video predictive understanding: Early action recognition and future action prediction.
4. Deng, X. et al. (2025). BAGEL: a unified multimodal model used as the controlled backbone family in the IVT study.
5. Black Forest Labs (2024). FLUX latent autoencoder representation, used as the reconstruction-oriented future target.
6. Oquab, M. et al. (2023). DINOv2: Learning Robust Visual Features without Supervision.
7. Tschannen, M. et al. (2025). SigLIP2 visual representations.
8. For full benchmark, baseline, and metric citations, see the bibliography in our paper.