J-Lens for video models:
uncovering steerable physical dynamics.
How do video models store information about the physical world?
This lab demonstrates that video transformers can build a compact and interpretable internal models of physics purely from raw video.
We show latent states in video transformers encode position, velocity, collisions and more in geometric representations in their activations. What's more, these representations are "causal" - you can write to them and, in doing so, shape the models' output to form a video game.
This is essentially an application of Anthropic's Jacobian lens to illustrate ideas developed in Interpreting Physics in Video World Models, also inspired by MIRA and Genie 3.
The punchline: steer a video model's hallucinations.
Put the puck in the goal!
The game below is the real-time output of a 3.67M-parameter video transformer running in your browser. Arrow keys write to the model's activations and "steer" the model's hallucinations, using Anthropic's Jacobian lens technique.
Real-time sampling from a video transformer with steering.
Training a toy video prediction model
To study the learned representation of video models, we can train a tiny video transformer on a toy physics simulation. For this exercise, we use a 3.67M parameter pixel transformer, which directly operates on pixels and predicts the next frame, similar to an LLM predicting the next word.
We train it on Blocket League, a toy physics simulation with collision mechanics between and goal scoring. We sample rollouts and train with a next-frame-prediction objective.
Blocket League
For about $10 of compute, the model cleanly learns the transition function and can “hallucinate” valid game rollouts by recursively sampling frames. The learned dynamics are pretty spot on.
Loading hallucinations…
Generalizing to unseen scenarios
Is this model merely memorizing the training set, or is it able to learn some higher-level representation that generalizes beyond seen examples?
If you retrain this model with certain classes of samples entirely removed - for example, collisions in the top-right quadrant or an entire wedge of movement directions - it turns out the model can still accurately predict what will happen. This is fairly significant and implies the model has learned some higher-level representation that captures entities and their relationships, as opposed to merely interpolating between previously-seen samples.
Loading hallucinations…
Against a matched control trained everywhere, the penalty in the unseen quadrant is only 0.8%.
Decoding the learned representation
How accessible is this abstract learned representation?
Some physical properties are easily accessible within the transformer's hidden state. A simple linear regression from block 5 activations, for example, can neatly predict X/Y coordinates of both the orange and the blue ball. See the cross-hairs below for decoded X/Y coordinates, overlaid on next frame prediction samples
Similarly, we can see this same layer will "anticipate" a collision up to eight frames before contact.
Loading coordinate readouts…
The same fixed-token decoder keeps tracking both objects even when the model is feeding its own predicted pixels back as input.
See the full layer-by-layer and random-weight controls in Appendix D ↓Motion direction forms a ring in activation space
Motion direction itself has a ring-like topology in the transformer's block-5 activation space.
Average the block-5 activations by motion direction and project them with PCA: they wrap into a circular manifold, with nearby directions in the world remaining neighbors inside the model.
The geometry mirrors what it represents: rotate the physical direction smoothly, and the hidden state moves smoothly around its internal ring.
24 observed motion-direction bins projected from 768 activation dimensions to two principal components.
Finding steerable directions with the Jacobian lens.
In NN activations space, readable is not necessarily writable. "Writing" to the x/y activation directions from above does not properly steer model output; this implies it is not causal, and just a reflection on the proverbial cave wall of what the model has actually learned.
Using Anthropic's recently-published Jacobian lens, however, we can uncover a causal mechanism in the model's computation that controls ball velocity, then manipulate this to change the transformer's output.
Concretely, we can trace how block-5 activations affect the next frame, then average those gradients into reusable motion directions. See the implementation.
Run the frozen model on many randomized motion histories.
Backpropagate the player disc's next-frame x/y position to its block-5 hidden state.
Average across worlds to get a reusable vector we can write into the residual-stream activations.
These variables are causal. Write to them and the hallucination changes.
Write the recovered +x direction for four frames, then stop. By frame 12, the player disc is 3.51 pixels farther right on average across 256 unseen worlds, and 85.9% move in the intended direction. A random activation direction has almost no effect.
Loading passive-pixel study…
Making a video game via transformer "brain surgery"
If you map your keyboard to these causal direction probes, it becomes a simple video game. Note that the model has exclusively trained on raw pixels - we are not manipulating the model's input, but rather it's own internal representation for what it has "discovered" about motion.
This demo (repeated from above) loads the full 3.67M-parameter model into your browser.
Real-time sampling from a video transformer with steering.
Conclusion
Even this toy model builds a higher-level internal state we can read, that generalizes beyond its training examples, and that interpretability techniques available in 2026 let us causally influence.
I think this scales. In the spirit of Anthropic's global-workspace view, models trained over trillions of tokens seem to develop detailed internal worlds that are isomorphic to the reality they observe. Mapping these worlds and learning how they shape behavior may be one of the most exciting and consequential scientific projects of the AI era.
Appendix A: How small can the world model get?
We compressed the pixel transformer from 3.67 million parameters to 377,136—about one tenth the size—and retrained it for 12,000 steps. The 1.54 MB model learns the arena and coarse motion, but its entities fragment during long autoregressive rollouts: 64-frame position error rises from 6.53 px to 19.10 px. For this recipe, 377,136 parameters is below the useful capacity floor.
The 1.5 MB capacity cliff
The model learns coarse motion and the static arena, but its entities fragment under autoregressive rollout. Tenfold compression crosses the useful capacity threshold for this recipe.
- 12-frame error
- 4.98 px
- 64-frame error
- 19.10 px
- FP32 checkpoint
- 1.54 MB
Appendix B: Does the model generalize to a direction it never saw?
We trained two new transformers from scratch. For one, we rejected every 24-frame training world in which the puck ever moved within ±30° of due east; the matched control saw all angles. Then we forced both models to begin in eight controlled direction bins and rolled each one forward from pixels.
Mean rendered-pixel puck error over 12 autoregressive frames · 32 unseen worlds per direction bin
The held-out model reaches 0.95 px error due east—almost identical to its 0.99 px average on directions present in training. There is no catastrophic hole at the missing angle. Yet the all-angle control reaches 0.56 px on the same due-east worlds, so removing that 60° wedge costs 71% in accuracy. The transformer has learned a direction-general transition rule, while direct experience still sharpens it.
Appendix C: Does collision physics transfer across the arena?
We rejected every 24-frame training world containing a player–puck collision in the upper-right quadrant, then tested collisions centered in all four quadrants. The model could still observe free motion in the missing region; only collision outcomes were removed.
Mean puck-position error over 12 autoregressive frames · 32 collision worlds per quadrant
On 32 upper-right collision worlds, the held-out model reaches 1.257 px puck error versus 1.247 px for the matched all-location control: just a 0.8% penalty. Its error in the unseen quadrant is only 7.5% above its own three-quadrant average. In contrast to the direction holdout, the transformer appears to have learned collision dynamics as a location-independent rule.
Appendix D: Does the model construct Cartesian coordinates?
Earlier position probes selected the token containing the rendered entity, so they could partly recover absolute patch identity. Here the decoder receives only one fixed hidden state: the bottom-right token after the eighth observed frame. We ask it to report the player and puck's x/y coordinates, then exclude the upper-right quadrant from probe fitting to test whether horizontal and vertical position recombine.
Player and puck x/y from one fixed token.
Velocity direction, with speed at 0.89 R².
Collision prediction eight frames before contact.
Linear x/y decoding from the literal bottom-right token after the final observed frame. No entity token is selected.
Readable is not automatically writable.
Adding the linear probe's own x/y vector barely changes the rendered puck. A probe identifies correlation; it does not identify the downstream route that controls pixels. Reorient the write using the Jacobian of the rendered puck centroid, however, and all four directions become causal.
Spatial attention broadcasts both objects' locations into a fixed token, and a linear transform recovers separate x/y axes. But adding that regression vector back into the residual stream does not teleport the puck. Linear decodability alone is correlational. When we instead orient the write with the downstream Jacobian of the rendered puck centroid, the same position variable becomes steerable: one brief write moves the puck in the intended direction in 84–90% of worlds immediately, and the altered trajectory continues separating after the write stops.
This still does not prove that a collision circuit literally consumes this particular coordinate readout. The next decisive test is geometric mediation: decompose a collide-versus-miss activation patch into the x/y-and-velocity subspace and its orthogonal remainder, then ask which component causes the model to apply a collision-like change in post-contact velocity.
Appendix E: Can the model see collisions coming?
Yes. A linear probe at block 5 predicts a collision up to eight frames—400 ms—before contact, even when collide and miss examples have matched positions and speeds.
Held-out AUROC · 1,024 matched fit pairs and 512 disjoint test pairs per horizon
At block 5, collisions remain predictable eight frames, or 400 ms, ahead. Held-out AUROC reaches 0.882 at that horizon, compared with 0.483 for a linear readout of the explicit pixel trajectory.
An untrained transformer with the same architecture reaches only 0.567 at block 5, averaged across three random initializations. Shuffled-label controls remain near chance. Training therefore organizes a substantially more predictive collision representation than either raw linear pixels or random transformer features.
Appendix F: Registered experiments
Each run links to its checkpoint metadata, metrics, and held-out sample rollouts.