This episode features three AI researchers — Beren Millidge (CTO of Zyphra), John Schulman (Chief Scientist at Thinking Machines, former OpenAI co-founder), and Charlie O’Neill (Head of Model Training at Baseten) — debating how close we are to recursive self-improvement (RSI), the technical bottlenecks that could prevent an intelligence explosion, and concrete timelines for AI automating research and general white-collar work.
The case against recursive self-improvement by 2036
The most likely technical reason 2036 isn’t a superintelligence world: persistent sim-to-real gaps and failure to solve continual learning / meta-learning generalization, not compute or data limits.
Each new model generation catches up on some benchmarks but reveals new bottlenecks in judgment, self-checking, and long-horizon coherence; the “this is AGI” cycle may repeat many more times.
Even if models write 100x more code, researchers don’t become 100x more productive because they’re bottlenecked on experiment design, taste, and deciding what to optimize.
The current paradigm (transformer + RL) may hit an asymptotic curve if reaching the global optimum of “a learner on a chip” requires a discontinuous architectural breakthrough that LLMs can’t discover by scaling RL environments alone.
Moore’s law and LLM scaling laws both required repeated discrete innovations to maintain straight lines; the next discontinuity might be beyond what current methods can find.
If deep learning can’t produce an AI that dominates human R&D (including paradigm invention), RSI stalls — but given progress since 2012, it would be weird if it didn’t eventually reach that point.
Ryan Greenblatt’s “Elo crossing” analogy: AI capabilities rise linearly like chess Elo; economic impact stays low until crossing the human range, then shifts dramatically — the only way this doesn’t happen is if capabilities asymptote just before human-level or regulation intervenes.
A key distinction: “autoresearch” (optimizing a specified objective) vs. open-ended science (specifying the objective itself). Current RL optimizes inner loops; the outer loop of choosing what to optimize may not automate easily.
John Schulman: early OpenAI intuition was that next-token prediction loss wouldn’t capture the important bits; they needed crafted objectives. It turned out to work anyway, but generalization from verifiable to unverifiable tasks remains unpredictable.
Automated theory-building (centuries of AI thinking per experiment) could 10x speed up the current paradigm, but doesn’t solve objective discovery — “just thinking doesn’t buy you the right objective.”
The last human job in AI R&D: defining objectives, deciding what we want, and alignment (specification + optimization). Even with full technical automation, humans must specify constitutions, model specs, and behavioral preferences.
What’s driving Chinese labs’ progress
Distillation is the main anti-centralization force: RL-learned behaviors are few bits, easily distilled from trajectories. Router/proxy services in China collect real-user prompt distributions from frontier models, providing ideal distillation data.
Prompt distribution is critical for distillation; Chinese labs synthesize vast coverage from seed prompts using frontier models, automating data gathering.
Ironically, distillation is easier for followers: they prompt the frontier model for “good politician” behaviors, while the frontier lab had to build the capability from scratch.
Real-world deployment data (realism axis) may matter more than difficult verifiable environments (benchmaxxing axis) for broad capabilities.
Frontier labs push both difficulty and realism; distillers matching only on benchmaxxing tasks lose the realistic, multi-turn, multi-objective behaviors.
Smaller models may generalize worse from narrow tasks to realistic settings; big models generalize better.
Data vendors sell the same datasets globally; Chinese labs buy identical data and distill, making it easy to keep up.
How automated AI researchers will be trained
Training will combine learning from human feedback (absorbing researcher taste) with massive multi-step research environments; each iteration patches the biggest weaknesses.
Labs “diff” the last 3 months of research progress (bugs found, improvements made) into environments — continual learning via distillation of recent human+AI progress.
Rolling back lineage (e.g., pre-GRPO) to rediscover algorithms via self-play is compute-prohibitive; staying at the frontier and distilling incremental gains is more efficient.
Environments can exceed human performance (e.g., “achieve loss 1.3” or “train 100M param model beating complex game”), enabling superhuman optimization on well-specified goals.
But much research is intuition-guided: relax realism to test algorithmic ideas, then mature toward realism. Training will mix verifiable tasks, LLM-as-judge, and human evaluation, hoping for generalization to fuzzy long-horizon tasks.
Whether the loop becomes self-sealing without humans is unclear.
Will long-horizon RL elicit AGI?
The labs’ bet: scale RLVR across millions of diverse environments (coding → finance → PowerPoints → long tail of economy) → emerges persistent, sample-efficient agent functioning as drop-in remote worker over weeks/months.
Two explanations for domain-specific training despite generalization hopes: (1) models aren’t yet good enough at in-context learning, so amortize skills into weights; (2) revenue from commercial deployment funds RSI-focused research.
Transfer occurs: meta-learning “how to figure out what’s important” generalizes across domains; parameter capacity is massive, so learning everything is cheap.
Sim-to-real dominates while sample efficiency is low (thousands of human interactions needed per task); as sample efficiency improves, learning from deployment (off-policy, cartridges, LoRAs) will grow.
Economic pressure favors modular specialization (LoRAs, cartridges, KV caches) over one shared model updating on all deployment data — companies won’t want their proprietary data improving a competitor’s model.
Stages: specialize via modules → generate traces → consolidate into base model every 3 months → accelerate to weekly/daily/hourly → continual learning solved.
Learning from deployment / continual learning bottlenecks
Deployment data already feeds next-gen models (especially Chinese labs via distillation); closed labs claim not to train on user data but likely do via mid-training.
Companies like Cursor (Composer) and Harvey do rapid online RL: daily REINFORCE updates from user accept/reject signals, deploying if benchmarks improve.
Core problem: reward function for natural data is unknown; superficial signals (edit acceptance) reward-hack.
Fundamental bottleneck: catastrophic forgetting during iterative micro-updates on non-stationary data.
SFT/on-policy distillation destroy previous capabilities after hundreds of updates; RL changes weights minimally (small loss valley tweaks) but can’t inject explicit knowledge efficiently.
Not a capacity issue — retraining from scratch with all data works — but a technique issue: plasticity vs. stability trade-off unsolved at micro-update granularity.
Current workaround: continual mid-training on frozen base + RL from later checkpoints, but bases still asymptote and require full retraining eventually.
How much progress is explained by data vs. architecture
Beren’s grid experiment (recipes 2019–now × datasets 2019–now): data explains ~12x compute efficiency gain, architecture ~3.7x at small scale.
Missing ~100x vs. Epoch’s 3x/year (2000x total) likely from scale-dependent effects: architecture unlocks qualitative regimes (e.g., GQA enables 1M context → new data usable).
Data efficiency gains may increase with scale (long-context environments need big models); architecture gains may be one-time regime unlocks.
Pre-training data low-hanging fruit exhausted (internet not growing at same rate); mid/post-training signal must be created via environments, human reasoning traces, or deployment data — world isn’t generating enough novel bits at the capability frontier.
Parameter scaling: active parameters may plateau for a few years (inference efficiency critical for RL rollouts); total parameters depend on hardware (memory bandwidth/VRAM for multi-trillion param serving).
If compute bottleneck eases and data becomes constraint, larger undersaturated models may return for better generalization.
Sparsity poorly understood; may have sweet spot, not unbounded increase.
Why RL is working better than naive theory predicted
Mid-training (synthetic reasoning data) does ~80% of the work; RL only tweaks policy with a few high-signal bits (correct/incorrect), avoiding SFT’s noise of matching exact reasoning tokens.
RL’s signal-to-noise ratio is dramatically higher: objective ignores irrelevant tokens, upweights only correct-answer bits.
With large group sampling, probability of correct answers rises, enabling learning even from sparse rewards.
Key generalization: horizon generalization (models learn to persist longer across tasks), not horizontal reasoning transfer (math RL doesn’t automatically make great coders).
EdgeBench: model work duration doubling every 3 months.
RL hits discrete “quanta” per task (phase transitions from 0.5% to 90% pass rate); averaged across thousands of tasks + horizon gains → qualitative leap.
RL doesn’t necessarily destroy creativity: AlphaGo’s Move 37 used MCTS (more exploration); LLMs already show creative zero-day exploits. Entropy collapse in writing stems from narrow verifiers/judges, not RL itself — monoculture from distilling Claude’s style.
Rapid-fire timeline predictions
Drop-in remote worker (month-long, full computer use, complex projects, human interaction):
Charlie: ~1 year (browser-mandated: ~2 years); 80-90% via org adaptation sooner.
Beren: ~3 years for full generality; long tail of miscellaneous human tasks (e.g., yelling to unblock) remains hard.
John: ~2 years for competent white-collar (not super-creative research); already 10x for coding.
10x AI researcher productivity uplift (breakthrough/month instead of year):
Charlie: 5-10 years (bottleneck: human Bayesian decision-making on next experiment).
John: ~2 years (if AI runs 2-3 clean experiment loops, huge uplift; shifts bottleneck to compute/infrastructure).
Beren: ~2 years (coding already >10x; one experimental feedback loop = massive).
ASI: dominates top human experts across all computer-based fields, including 3-year projects and fast on-the-fly learning:
Charlie: 3-4 years (AI research gets most attention, code/math are strengths; physical/3D domains lag).
John: 5-10 years (long-tail domains with little data, e.g., TSMC engineering; requires solving long-horizon learning).
Beren: 5+ years (labs focus on high-value domains; literal “every single expert” includes obscure fields no one has allocated compute to).
