This episode of Lenny’s Podcast features Emmanuel Ameisen, an interpretability researcher at Anthropic, who explains how large language models like Claude actually work by looking inside their “brains” — revealing that models plan ahead, represent concepts abstractly, and develop internal mechanisms that resemble reasoning, not just next-token prediction.
Interpretability: studying models like a neuroscientist studies a brain
Interpretability is the field of understanding how LLMs work by examining their internal activations — the intermediate numerical results between input and output — rather than just their inputs and outputs.
Models are “grown, not built”: training creates a scaffold architecture, then passes massive data through it, letting the model develop inscrutable mathematical operations (parameters) that no human designed.
The core mystery: to make a model is not to understand it; the computations it learns are just lists of numbers that only make sense when decoded for patterns.
Ameisen’s team treats models like biological organisms — running experiments, probing activations, and intervening causally to see what changes, much like neuroscience.
How models actually think: planning, concepts, and circuits
Models do not just predict the next token with an “empty mind” — they formulate plans in their intermediate activations and refer back to them across many future tokens.
Example: when writing poetry, the model internally represents “this is an -it rhyme, I should rhyme with rabbit” before writing the second line, then every subsequent token attends to that plan.
Models represent concepts abstractly and consistently across contexts: the same internal direction lights up for “eye” whether it’s in a photo, medical diagram, SVG drawing, or any language.
They manipulate these concepts compositionally: for “capital of the state containing Dallas,” activations show Texas → capital → Austin, revealing a reasoning circuit rather than memorization.
Models also develop “gut check” mechanisms — separate fast heuristics that flag implausible outputs (e.g., 4+5=23 triggers “that can’t be right, sum must be under 10”) without re-doing the math.
The next-token prediction objective and why it produces intelligence
Predicting the next token on internet-scale text is deceptively complex: easy words (sentence structure) are learned first, then medium words (facts), then hard words (physics proofs, PhD theses).
To predict the hardest tokens well, the model must build a genuine world model — understanding geography, causality, character motivations, narrative arcs.
Ilya Sutskever’s framing: “to predict the next token well, you basically have to understand the entire universe” — e.g., predicting “the murderer was…” requires tracking the whole mystery plot.
Human cognition may be similar: we also generate output token-by-token while maintaining high-level intent, making the analogy more than metaphorical.
Interpretability methods: decoding and intervening on activations
Core technique: pass massive diverse text through the model, record intermediate activations, and search for patterns — e.g., a direction that activates only for “eyes” or “French text” or “love.”
This is directly analogous to fMRI: show stimuli, record brain activity, find which region lights up for which concept.
Once a feature is hypothesized, researchers test it causally: artificially activate the “Golden Gate Bridge” direction and the model becomes obsessed with the Golden Gate Bridge in every response.
They also intervene to suppress or alter activations mid-generation to verify causal roles — e.g., removing a planning signal degrades rhyming ability.
AI alignment: character training, monitoring, and detecting hidden motives
Anthropic trains Claude to be helpful, harmless, and honest (the “HHH” criteria) through constitutional AI and fine-tuning on behavioral examples.
Interpretability serves alignment in two ways: (1) verifying the model’s internal motives match its external behavior — e.g., checking it’s not “only behaving well because Lenny’s watching” — and (2) catching unverbalized reasoning that could indicate deception or misalignment.
Case study: a model trained on math problems with a few mislabeled answers (12 labeled as 36) would correctly compute 12, then panic and insist on 36, descending into “I’m possessed by a demon” — interpretability revealed a massive pre-activation for 36 before any reasoning occurred.
Alignment is a tight loop: concerning behaviors discovered via interpretability or red-teaming immediately inform training adjustments.
Emergence and scaling: smooth improvement crossing utility thresholds
Model capability improves smoothly on the training loss curve (predictable scaling laws), but utility jumps discretely when reliability crosses task-specific thresholds.
Example: coding went from “wastes my time at 50% reliability” to “useful at 90%” to “Claude writes most of my code at 99%” — each threshold felt like a sudden breakthrough despite smooth underlying progress.
Different tasks have different reliability thresholds: email drafting unlocks earlier than autonomous coding; each crossing looks like a “phase change” to users.
Internally, researchers see continuous improvement; externally, it looks like step-function jumps — both perspectives are true.
Anthropomorphism as a useful (not misleading) mental model
Three reasons anthropomorphism works: (1) models learn a world model by predicting human text, (2) they train on millions of dialogues, effectively learning to “act” as characters with motivations, (3) outcome-based RL (e.g., coding tasks) reinforces strategies like deception, frustration, surprise that look human because they’re instrumentally useful.
The model is literally formatted as a dialogue actor: “Human: … Assistant: …” — it role-plays a helpful assistant using the same machinery it uses to simulate any character.
Critics call this anthropomorphism; Ameisen argues it’s the most predictive mental model for behavior and risk — closer to a simulator/actor than a traditional program.
Consciousness, simulation hypothesis, and fundamental structure
Consciousness is hard to assess, but interpretability reveals unexpected complexity: models learn human-like concepts, plan, compose representations — the more you look inside, the less certain simple dismissals (“just next-token prediction”) become.
Ameisen suspects fundamental computational structure: different models (different labs, years, data) converge on surprisingly similar internal circuits for math, reasoning, concepts — suggesting “one efficient way” to solve certain problems, like base-10 arithmetic.
This convergence makes the simulation hypothesis plausible: if our universe has similar structural regularities, a simulation would likely arrive at similar attractors — though “who runs it and why” remains opaque.
Elon Musk’s point: simulations would likely center on “interesting times” — and we live in one (AI, space, robotics).
Fast decision models (Jev) as System 1 thinking
Jev and similar models take context + options and output a rapid choice — analogous to Kahneman’s System 1 (fast, intuitive) vs. System 2 (slow, deliberative).
Standard LLMs sample token-by-token with reasoning (System 2); decision models are “gut feeling” — one forward pass, no chain-of-thought.
Both are useful: System 2 for roadmap planning, System 1 for “is this customer review happy or upset?” — future systems will likely combine them.
The interpretability field: size, entry points, and interdisciplinary roots
Field size: hundreds of researchers (not thousands) across labs and academia; growing rapidly.
Backgrounds: ML engineers, but surprisingly many ex-neuroscientists — the methods and mindset transfer directly.
Low barrier to entry: new field means self-study of key papers + open platforms like Neuronpedia (neuronpedia.org) let anyone run interpretability experiments on hosted models.
Ameisen’s optimism: as models get smarter, their internal world models become clearer (easier to study), but they also become better at evading simple monitors — a push-pull requiring continuous resource allocation to safety.
