The Decade of Agents

Karpathy explains his statement that this will be "the decade of agents" rather than "the year of agents," which he describes as a reaction to over-prediction in the industry. While early agents like Claude and Codex are impressive and used daily, substantial work remains. Agents should function like employees or interns, but currently they lack sufficient intelligence, multimodality, computer use capabilities, and continual learning. These cognitive limitations mean it will take approximately a decade to work through all necessary improvements.

When asked why a decade rather than one or fifty years, Karpathy cites his nearly fifteen years of AI experience, watching predictions and their outcomes, and working in both research and industry. He believes the problems are tractable and surmountable but still difficult, leading to his decade estimate based on intuition from this experience.

Three Major Seismic Shifts in AI

Karpathy identifies living through two or three major seismic shifts in AI, with more likely coming. His career began with deep learning at the University of Toronto next to Geoff Hinton, when neural networks were a niche subject. The first dramatic shift came with AlexNet, which reoriented the field toward training neural networks, though initially focused on per-task applications like image classifiers or machine translation.

The second shift involved agents, exemplified by Atari deep reinforcement learning around 2013. This attempted to create agents that perceive and interact with environments, not just process information. However, Karpathy considers this a misstep that early OpenAI adopted. For two to three years, the field focused on reinforcement learning on games, which he was always suspicious of as a path to AGI.

At OpenAI, Karpathy worked on the Universe project with an agent using keyboard and mouse to operate web pages, aiming for something that could do knowledge work. This was extremely early and burned through computing resources without success because it lacked the necessary representation power in neural networks. The key insight: you need the language model and representations first through pre-training before building computer-using agents on top. People kept trying to achieve the full agent too early.

LLMs as a Different Kind of Intelligence

Karpathy argues that LLMs represent a fundamentally different kind of intelligence. They process everything simultaneously through context windows, unlike humans who think sequentially. When asked about the intelligence of a 1950s physicist versus an LLM, he suggests they're different types: humans are slow sequential thinkers who understand the world deeply through embodiment, while LLMs are fast parallel processors trained on massive text data.

LLMs lack human experiences like hunger, thirst, or social interactions with other intelligent beings. They've never even seen an apple or held one. While they can recombine text patterns intelligently, they miss entire swaths of human understanding gained through embodied experience. Both types of intelligence have strengths, making direct comparison difficult.

Scaling, Data, and Architecture

On debates about scaling versus new algorithms, Karpathy emphasizes that both matter but scaling is more certain. The transformer architecture itself emerged from algorithmic innovation and became fundamental, but he's uncertain whether current architectural changes will prove equally important. He acknowledges we may hit data walls but notes inference compute offers an alternative scaling dimension that's relatively unexplored.

Regarding synthetic data, Karpathy believes it will become increasingly important. While reinforcement learning with environment interactions works for games and increasingly for robots, synthetic data generated from language models themselves will play a major role in training future models. The key question is whether systems can bootstrap and improve themselves through this approach.

Understanding Through Representation

When asked about understanding, Karpathy notes that LLMs clearly have some form of understanding—they manipulate abstract concepts, reason, and problem-solve. However, the depth of this understanding is debatable, particularly regarding concepts requiring embodied experience. The transformer architecture creates sophisticated representations that enable these capabilities, even if the underlying process differs fundamentally from human cognition.

He explains that neural networks build hierarchical representations: early layers detect edges, middle layers combine these into textures and parts, and later layers recognize high-level concepts. This representational hierarchy enables the network to manipulate abstract ideas effectively, even without human-like understanding rooted in physical experience.

Educational Philosophy and Micrograd

Karpathy describes his approach to teaching, exemplified by projects like micrograd and his neural network tutorials. Micrograd demonstrates that neural network training can be understood in just 100 lines of Python—everything else is efficiency optimization. He believes in finding these "small-order terms" and serving them clearly to learners.

Education interests him intellectually because it involves untangling complex understanding and creating a learning ramp where everything depends only on what came before. His transformer tutorial begins with bigrams (simple lookup tables) and progressively adds complexity, presenting problems before solutions so students appreciate why each component is necessary. This approach maximizes knowledge gained per new fact added.

He notes that genuine experts often struggle to explain their fields due to the curse of knowledge—they can't easily put themselves in beginners' shoes. He finds it helpful when students share their "dumb questions" with ChatGPT, as this reveals where explanations need improvement. Additionally, he observes that informal explanations over lunch often capture ideas more clearly and accurately than formal papers or abstracts.

Learning Strategies

Karpathy advocates for alternating between depth-wise learning (on demand, project-driven) and breadth-wise learning (foundational courses teaching things you'll need later). He particularly values learning driven by projects that provide immediate rewards, though acknowledges the need for traditional breadth-wise education as well.

He emphasizes that explaining things to others is one of the most effective ways to learn deeply. When you can't explain something well, it reveals gaps in your own understanding. This forces you to manipulate knowledge and ensures you truly understand what you're discussing. He encourages people to re-explain concepts more frequently as a learning tool.