AI pioneer Yan LeCun is championing a billion-dollar alternative to the dominant Large Language Model (LLM) paradigm known as JEPA (Joint Embedding Predictive Architecture). Unlike generative models that spit out text, images, or video by predicting the next token or pixel, JEPA is a non-generative framework designed to build “world models.” LeCun argues that while LLMs excel at manipulating language, they lack the common-sense reasoning required to understand the physical world. JEPA aims to solve this by learning abstract representations of the world, mimicking the highly efficient way animals and humans learn to predict the consequences of their actions.

The Limitations of Generative Vision Models

The self-supervised, generative approach of predicting the “next token” works brilliantly for language because text relies on fixed-size vocabularies. However, applying this exact same auto-regressive approach to video fails miserably. Video data is incredibly complex and continuous. When a generative model is asked to predict the exact pixels of an uncertain future video frame (like a bouncing ball that could go left or right), it averages the possible outcomes. This results in compoundingly blurry, washed-out, and useless predictions. To achieve true machine intelligence, AI needs a different proxy for learning other than pixel-perfect reconstruction.

Solving Representation Collapse with Joint Embeddings

To bypass the blurry video problem, researchers revisited Joint Embedding Architectures (like Siamese networks). Instead of generating images, these networks compress images into mathematical vectors (embeddings). By feeding a network an image and a distorted version of that same image, the AI is trained to recognize that both represent the same semantic concept. Historically, this method suffered from “representation collapse”—a glitch where the network lazily outputted the exact same generic vector for every single image just to satisfy the training rules. Researchers solved this collapse using a breakthrough called “Barlow Twins,” which forces the network to minimize redundant information across its artificial neurons. This unlocked powerful self-supervised vision models (like DINO) that rival fully supervised systems.

How JEPA Builds World Models

JEPA takes these highly accurate embedding architectures and transforms them into predictive “world models.” Instead of trying to guess the exact pixels of a future state, JEPA runs a current observation through an encoder to get an embedding. A separate “predictor” model then attempts to forecast the embedding of the next state, often conditioned on a specific action (like a robot arm moving). This is a game-changer: it frees the AI from wasting computational power predicting irrelevant, chaotic background details—like leaves rustling in the wind—allowing it to focus strictly on the salient, meaningful features of an environment.

The Future of Autonomous Agents

LeCun’s ultimate conclusion is that reliable, agentic AI cannot be built on auto-regressive text predictors. True autonomous intelligence requires the ability to plan. By utilizing machine-learned world models like JEPA, an AI agent can hypothesize different actions, predict their resulting states in an abstract embedding space, and choose the optimal sequence to achieve a goal. In this future, AI inference shifts from simple text auto-completion to active search, planning, and safe physical-world navigation.

Mentoring question

How might the shift from auto-regressive generative AI to predictive ‘world models’ alter the way your organization develops and deploys autonomous AI agents for complex, real-world tasks?

Source: https://youtube.com/watch?v=kYkIdXwW2AE&is=3qGyjM3xQv8_P5vk