We Rebuilt Yann LeCun's JEPA World Model on a 2017 Laptop. Every Metric Said It Worked. It Didn't.

What we were actually testing, and why

Start with the problem, because the architecture only makes sense once the problem is clear.

A large language model learns by predicting the next word. An image generator learns by predicting pixels. Both work remarkably well, and both share a weakness: they spend enormous effort predicting detail that does not matter. If you ask a model to predict the next frame of a video of a tree, it must predict the position of every leaf. Almost none of that is knowable, and almost none of it is useful.

A human watching that tree does not predict leaves. You predict something like "the branch will keep swaying, nothing is going to fall." You predict at the level of an idea, not a pixel. And that compact internal model of how things behave is what lets you plan — reach for a cup, cross a room, catch a ball — without simulating the universe in detail.

JEPA — Joint-Embedding Predictive Architecture — is Yann LeCun's proposal for building that. He laid it out in his 2022 position paper A Path Towards Autonomous Machine Intelligence, and it is the centrepiece of his argument that scaling next-token prediction will not, on its own, produce machines that understand how the world works. The mechanism is a deliberate act of omission: instead of predicting the next observation, the system predicts the embedding of the next observation — its own compressed summary. Because the summary is learned rather than fixed, the system is free to discard everything it cannot predict and keep only what matters.

That freedom is the whole idea, and it is also the whole danger. A system allowed to discard the unpredictable can discard everything and predict a constant. This is called representation collapse, and preventing it is the central engineering problem of the entire approach.

Why we ran this test

JEPA is a claim about how machine intelligence should be built. Claims are what our research integrity standard exists to examine. Three questions in particular:

• Does the core idea work without scale? Published JEPA results come from well-resourced labs. If the mechanism is sound, a stripped-down version should work on commodity hardware. If it only works at scale, that is worth knowing too.
• What actually makes or breaks it in practice? Papers report the version that worked. We wanted a full record of the versions that did not.
• Would we notice if it were broken? This turned out to be the important question.

We deliberately used a nine-year-old dual-core Intel laptop with no GPU. Constrained hardware is a useful discipline: it makes every failed run expensive, which forces you to measure instead of guess.

The setup

An agent is a dot in a small walled arena. It sees only a 48×48 pixel image — it is never told its own coordinates. It can nudge itself in any direction. There are walls arranged so that the straight line to most destinations is blocked.

Training is pure aimless wandering. The agent takes random actions for a while and watches what happens. No rewards. No labels. No goals. No map. The only training signal is: given what I see now and the action I am about to take, can I predict my own summary of what I will see next?

Then we test it on something it was never trained to do. We hand it a photograph of a location and ask it to get there. It plans by imagining: it dreams up hundreds of possible action sequences, rolls each one forward through its learned model of how the world moves, and picks whichever imagined future looks most like the photograph. It executes one step, looks again, and re-plans. This is the same shape as what LeCun proposed — an agent that plans by searching for actions leading to desired predicted states.

The result

100 tasks per condition, replicated across four independent task seeds — 400 episodes, zero failures. The model has 0.10 million parameters, which is roughly one-millionth the size of a frontier language model, and it trained in about 40 minutes on a CPU from 2017.

The representation learned to encode the agent's position almost perfectly — a straight-line readout of its coordinates from the internal state scores R² = 0.9997 — despite never being shown a coordinate.

That is the good news. Now the useful part.

Four versions failed, and every metric said they were fine

The first working version scored 12% against an 8% random baseline. Its planner ended up further from goals on average than random wandering did. Throughout, the loss curve fell smoothly. The collapse detector sat exactly where it should. The standard quality check for a learned representation — can you read the agent's position out of it? — read 0.97 out of 1.0.

Every dashboard was green. The agent was useless. Here is what was actually wrong.

Failure 1: the renderer had quantized the world

We drew the agent as a hard-edged circle — each pixel was either agent or background. That sounds harmless. It meant the image was a staircase: the agent could move a fraction of a pixel and the picture would not change at all, then a threshold would tip and several pixels would flip at once.

Sweeping the agent smoothly across the arena and measuring how much its internal representation moved at each step:

mean 0.0715   min 0.0000   max 1.4701   max/mean 20.6

min 0.0000 is the tell. Some movements produced no change whatsoever. You cannot learn smooth dynamics from a staircase — there is no consistent relationship between "how hard I pushed" and "how much the world changed." Anti-aliasing the circle, so its edge fades over one pixel, brought that ratio to 1.04.

This bug had nothing to do with machine learning. It was in the drawing code.

Failure 2: the encoder threw away spatial structure

Our vision network flattened the image into a long list of numbers and fed it to a standard neural layer. That forces the network to rediscover the concept of "where" from scratch, and it settled on a jumpy solution: a 4% change in position moved the internal representation by 38% of its total range.

The fix was to use a spatial soft-argmax — a layer that reads out where each visual feature is located as an explicit coordinate, rather than making the network infer it. This is not our idea; it comes from Chelsea Finn, Sergey Levine and colleagues' 2016 work on deep spatial autoencoders for robot learning. It made the representation smooth by construction, and the model's sensitivity to its own actions jumped from 0.029 to 0.156 — against a true signal of 0.150.

Failure 3: our own normalization made two networks disagree

This one we introduced ourselves while fixing the others. JEPA uses two copies of the vision network: a live one and a slowly-updated "target" copy that provides the prediction goal. We normalized using batch statistics — and the two copies process different batches. The result was that the same image encoded to two different vectors:

same frame, two encoders : 0.1448   <- error floor
one real step of motion  : 0.1500   <- the signal to predict

The disagreement was as large as the entire quantity being predicted. No amount of training could beat that floor. Switching to normalization from running statistics, shared between both copies, dropped it to 0.0034 — a 40-fold improvement.

What tied all three together

Through every one of these failures, the standard representation-quality check kept improving: 0.81, then 0.97, then 0.9964, then 0.9997. The agent stayed useless the whole time. The position signal was surviving inside a tiny corner of the representation while noise dominated everything else — enough for a readout to find, nowhere near enough to steer by.

A representation can be perfectly decodable and still useless for control. Probe accuracy is necessary, not sufficient.

What caught all three was an embarrassingly cheap comparison we now recommend to anyone building a world model: is your model better than assuming nothing happens? Feed it a state, have it predict the next state, and compare against simply guessing that nothing changed. A model that cannot beat "nothing happened" has learned nothing about actions — no matter how good the loss curve looks. All four broken versions failed that test. It costs nothing to run.

A solution we built, tested, and threw away

With the model fixed, the agent reached 82%. The remaining failures were long detours — goals on the far side of a wall.

The diagnosis seemed obvious. The planner scores imagined futures by straight-line distance to the goal, which ignores walls, so it should get stuck driving into the barrier nearest the target. The textbook fix is to learn a smarter notion of distance: train a network to predict how many steps of wandering separated any two states. Walls should fall out for free, because going around one takes time.

It failed twice. First it learned to predict a constant — obvious in hindsight, since a random walk that dawdles takes 55 steps to cover ground it once covered in 5, so the average is nearly meaningless. We switched to predicting the shortest observed transit instead, which fixed that, and it still did not help the agent.

Then we checked the learned distance against ground truth, which we should have done first:

correlation with true walk-around distance:
  our learned distance : +0.314
  plain straight-line  : +0.980

Straight-line distance was already a near-perfect proxy in this arena — the walls are thin enough that going around barely changes the ordering. Our premise was wrong. Worse, the learned network was backwards exactly where it mattered: for two points equally far from the goal, it rated the one on the wrong side of the wall as closer.

That validation took seconds. It would have saved two full training-and-evaluation cycles had we run it before building the policy on top. The failed approach stays in our codebase, clearly marked and switched off by default, because a documented dead end is cheaper for the next person than rediscovering it. The full code, the diagnostics behind every figure in this post, and the failed approach are public: github.com/cortexresearch/jepa-crg-ris-study.

What actually worked — and who invented it

The real problem was not the distance metric. It was that the planner imagined walking through walls, got blocked, and re-planned the identical route forever.

The fix: stop asking a network to generalize a notion of distance, and instead search over places the agent has actually been. We take 2,500 remembered states as nodes in a graph, connect ones that are near each other, and plan by finding a path through that graph — steering toward a waypoint a few hops ahead rather than at the goal directly.

The critical detail is which connections are allowed. Connecting states purely because they look similar reproduces the original bug — two points either side of a thin wall look almost identical. So every candidate connection is checked against the world model: it survives only if some action actually moves the agent from one to the other. Because the model had learned about collisions, it refuses the impossible shortcuts:

graph: 2500 nodes, 59243 connections
connections passing through a wall: 0 of 67320  (0.00%)

Zero. That took the agent from 82% to 100%.

And this is not a new idea. We reinvented, with some embarrassment, a method published eight years ago: Semi-Parametric Topological Memory by Nikolay Savinov, Alexey Dosovitskiy and Vladlen Koltun (ICLR 2018), which builds exactly this — a non-parametric graph of remembered locations that emits a waypoint toward the goal. Ben Eysenbach, Ruslan Salakhutdinov and Sergey Levine extended the idea in Search on the Replay Buffer (NeurIPS 2019). Our contribution is narrow: using the JEPA world model itself as the edge validator. The architecture is theirs.

Where this could go

We want to be careful here, because our result is one fixed maze on one laptop, and the gap between that and the real world is exactly where confident predictions go to die. The following is clearly marked as informed speculation rather than a finding.

The pattern that generalizes is the training recipe: collect cheap, unlabeled observation of a system doing more or less nothing in particular, learn a predictive model of it, and then use that model to pursue goals it was never trained on. This matters wherever rewards are expensive but observation is cheap — which is most of the physical world. Designing a reward function for a robot is difficult, hand-labeled demonstrations are expensive, and both must be redone for every new task. Passive video is nearly free.

This is not hypothetical at the top end. Meta's V-JEPA 2 (2025) post-trained an action-conditioned world model on under 62 hours of unlabeled robot video, then deployed it zero-shot on Franka robot arms in labs it had never seen — picking and placing objects specified by a goal image, with no task-specific training and no reward. That is the same shape as what our dot in a maze does, several orders of magnitude up.

Places this recipe plausibly fits:

• Robotics in unstructured spaces — warehouse floors, construction sites, agricultural rows, where the layout changes weekly and re-engineering a reward function for every change is impractical.
• Inspection and mapping — drones or ground robots that learn a facility's dynamics by patrolling it, then navigate to a photographed location on request.
• Assistive and mobility devices — systems that must adapt to one specific home or one specific user's environment, where no pre-existing dataset exists and passive observation is the only affordable data.
• Industrial process control — settings with abundant sensor logs and no reward signal, where "reach this state" is the natural way to express an objective.

The transferable methodology may be the more immediately useful output. Three checks caught everything that went wrong here, none of them requiring a training run:

1. Compare against "nothing happened." If your model cannot beat the assumption that the world is static, it has learned nothing about actions, whatever the loss says.
2. Compare against a perfect model. Swap in ground-truth dynamics. Whatever that scores is your planner's ceiling. If your learned model already sits at it, more model work is wasted — this saved us from optimizing a component that was already good enough.
3. Validate any learned metric against ground truth before you build on it. Correlate it with the real quantity. This killed a solution we had already implemented.

Those apply to any world-model project, at any scale.

What we did not establish, and will not claim: the maze is identical during training and testing, so this says nothing about generalizing to new layouts. The environment is deterministic and reversible; the real world is neither. And a graph of remembered states grows with the size of the space — the approach as built does not obviously scale to large or open environments without a smarter index. Each of those is a real open question, not a formality.

Credit where it belongs

Almost nothing in this project is ours. The value we added was testing carefully and reporting the failures.

• JEPA and the world-model argument — Yann LeCun, A Path Towards Autonomous Machine Intelligence (2022). The core thesis that prediction should happen in representation space, and that this is a prerequisite for machines that understand how the world works.
• I-JEPA — Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun and Nicolas Ballas, CVPR 2023. The first large-scale demonstration on images.
• V-JEPA 2 — Assran, Bardes, Fan, Garrido, Ballas, Rabbat, LeCun and colleagues (2025). Video world models used for zero-shot robot planning from image goals.
• Collapse prevention — VICReg, Adrien Bardes, Jean Ponce and Yann LeCun (ICLR 2022); and the momentum-target technique from BYOL, Jean-Bastien Grill and colleagues (NeurIPS 2020).
• Spatial soft-argmax — Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine and Pieter Abbeel, Deep Spatial Autoencoders for Visuomotor Learning (ICRA 2016). Fixed our second failure.
• Graph-based planning over remembered states — Nikolay Savinov, Alexey Dosovitskiy and Vladlen Koltun, Semi-Parametric Topological Memory for Navigation (ICLR 2018); and Ben Eysenbach, Ruslan Salakhutdinov and Sergey Levine, Search on the Replay Buffer (NeurIPS 2019). This is the method that took us to 100%.

Scored against our own standard

The Cortex Research Group Research Integrity Standard asks that work be scored on ten criteria, three points each. We scored ourselves conservatively where the evidence is thin rather than where the story is strong.

Not Robust. That would require independent replication, measured variation across training runs, and clean isolation of each of the three fixes. We applied them in sequence alongside other changes, which means our causal story is better supported than a guess and worse supported than a controlled experiment. Saying so is the point of the exercise.

Our own predictions, recorded before testing, went 1 for 5. We were wrong that the planner's sampling was the bottleneck, wrong that wall detours were the problem at the 50% stage, wrong that collision modelling was the remaining gap, and wrong that a learned distance would help. We were right that graph search would close it. A scorecard like that is not a confession — it is the reason the final number is worth anything.

The takeaway

The headline is that a 0.10-million-parameter world model, trained on nothing but aimless wandering on a nine-year-old laptop, learned enough about a space to navigate it perfectly from a photograph. LeCun's core mechanism holds up at a scale where nothing can hide behind compute.

But the finding we would actually hand to another team is smaller and more practical. Four broken versions of this system produced healthy loss curves, healthy collapse metrics, and a representation-quality score of 0.97 out of 1.0. If we had trusted the dashboard, we would have published a system that performed worse than random wandering and never known.

The checks that caught it took seconds and no training runs at all.

The code, the diagnostic scripts, the failed approach and the full methods report are all at github.com/cortexresearch/jepa-crg-ris-study. If you reproduce it and get something different, we would like to know.

Evidence Over Assumption.