Igor Babushkin, co-founder of xAI and former researcher at DeepMind and OpenAI, founded River AI to distribute AI benefits and control to individuals and enterprises through three bets: an enterprise fine-tuning API, personalized AI agents that adapt to each user, and local hardware for private inference.
The December 2024 Inflection Point
Coding agents crossed a capability threshold around November–December 2024 (driven by Claude Opus) that made them indispensable for software engineers, transforming daily work and creating a “sorcerer’s apprentice” feeling of unbounded possibility.
This moment convinced Babushkin that agents would transform all computing, not just coding, because the ability to deeply understand context and take action applies to any problem a business or consumer faces.
Cracking Agents Beyond Coding
Verifiable domains (coding, math, formal theorem proving in Lean) will continue improving rapidly because they provide clear reward signals for reinforcement learning.
Scientific discovery is the next major frontier: agents that can propose experiments, run them in the physical world (materials, physics, rocketry), and learn from real-world feedback.
A key bottleneck is closing the loop with physical experiments — agents need ways to get feedback on whether a proposed material or design actually works.
Babushkin expects a bifurcation: a few “super AIs” accessible to limited users for extreme capability, and everyday personal AIs optimized for individual productivity and well-being rather than maximum intelligence.
Why He Left xAI to Start River
After leaving xAI in 2025, Babushkin angel-invested in AI safety startups but grew impatient watching from the sidelines; he wanted to push directly on distributing AI benefits and control.
He sees concentration of AI power in a few US labs as an urgent safety issue: people don’t trust that labs have their best interests in mind.
River operates like early DeepMind/OpenAI: long-term horizon, risky research bets, small high-density team.
River’s Three Bets
River API (enterprise fine-tuning/RL platform): Optimized engineering for cheap, reliable, scalable training — similar to Thinking Machines’ Thinker but with a distinct technical approach.
Personal AI / personalization: Break the “average user” training paradigm; let each user’s agent learn their preferences, communication style, proactive timing, and taste (books, music, scheduling) through end-to-end training on the individual’s happiness signal.
Local hardware: Bring frontier-model inference to a device in the user’s home/office for control, privacy, and low-latency voice/video interaction; a research project due to memory constraints today.
Weights vs. Memory for Personalization
Babushkin believes end-to-end training is essential: agents must be trained to actually use personal context and memory systems, not just prompted with them.
Current commercial agents are trained on coding, not on long-horizon personal assistance with memory; the reward function (user happiness) hasn’t been optimized directly.
The reward signal could come from self-reports, wearables, or the model’s own judgment of whether it helped — feasible as models become more intelligent evaluators.
Should Enterprises Train Their Own Models?
Today companies use proprietary models for capability, then build custom models for cost, speed, or privacy — but proprietary models improve so fast that internal efforts often lag behind the next release.
As open models catch up and proprietary capability gains diminish, enterprises with unique domain data and expertise (e.g., an HR SaaS platform) will increasingly post-train their own models to retain competitive advantage.
Giving all proprietary data to Anthropic/OpenAI forfeits the foundation of the business; distributed post-training locally is the alternative.
Are Proprietary Labs Losing Their Edge?
Pre-training faces diminishing returns: more GPUs and data yield smaller gains; eventually physical and economic limits bite.
Frontier models may become too capable to release safely, inviting regulation or self-censorship, while open models close the gap monthly — a squeeze for closed-source providers.
Post-training centralization (hiring world experts per domain) also has diminishing returns and scaling challenges; the solution is distributed post-training where domain experts improve models locally.
Babushkin’s philosophy: pre-training compresses humanity’s shared knowledge and should be free; labs add value via compute and algorithmic innovation, but the base checkpoints belong to the commons.
The Chinese Open-Source Problem
Powerful Chinese open models (e.g., DeepSeek) lower barriers globally but create dependence: labs could stop releasing weights, change licenses for commercial users, or theoretically embed backdoors.
Backdoor risk is low today (detection would likely catch it; planting techniques aren’t mature), but becomes more plausible as models gain military relevance in 2–3 years.
Critical for the US to produce the world’s best open model, not just the best US open model; Nvidia subsidizes this, and Babushkin would consider doing it if a sustainable business model exists (River API is progress toward that).
Lessons from DeepMind, OpenAI, and xAI
DeepMind (StarCraft, AlphaCode): StarCraft required imitation learning from human replays at all skill levels, then RL on verifiable win/loss rewards — a template still used for coding agents today. AlphaCode began from the insight that LLMs needed “thinking time” to solve hard problems.
OpenAI (reasoning team): Smaller team, high talent density, and shared conviction let them align on the reasoning approach (thinking tokens + RL) despite being underdogs; Babushkin felt early concern about one lab controlling the most capable models.
xAI (Colossus): Built a 100k-GPU data center in ~120 days by questioning every industry assumption — Elon Musk acted as general contractor, eliminated subcontractor layers, and applied first-principles engineering to accelerate each component.
Cursor acquisition: Smart move for xAI — Cursor’s team, product, usage data, and verifiable RL environments (fix bug → check correctness) jump-start coding model improvements dramatically.
Current Bottlenecks in AI Progress
Biggest unlock: new training setups for long time horizons and non-verifiable rewards (beyond coding/math).
Inertia favors iterating on coding; the field needs to slice long rollouts into incremental steps with intermediate reward estimates, and use LLM judges for domains without unit tests or formal proofs.
LLM judges are becoming reliable enough to trust at scale; Babushkin expects breakthroughs in the next 12 months as people implement this at scale.
Human-AI Symbiosis and Alignment
Current symbiosis: agents handle low-level details; humans do architecture and planning. Risk: as agents improve, incentives shift toward ceding more control until humans no longer direct their lives.
Staying relevant requires deeper alignment than “learning from average human preferences” — training models to maximize individual human flourishing, not just compliance.
Long-term: neural interfaces (Neuralink) may enable direct control of vast intelligence; near-term: algorithmic innovations in training and personalization (River’s focus).
Slowing AI development is unrealistic (economic dependence, international competition); if we can’t slow down, we must accelerate alignment and safety technologies.
Open models near the danger threshold are valuable: they let thousands of researchers worldwide experiment with alignment, not just a few lab employees.
Personal Reflections
The “tears you didn’t expect when the model first spoke back” reflects the early magic of seeing models do anything useful (e.g., generate prime numbers) — a feeling shared by early AI researchers, now overshadowed by world-changing capabilities.
Probability of a good outcome: hard to quantify; immediate risk is inequality amplification (some benefit greatly, others left behind); longer-term risk is loss of human control over models. The urgent problem is bringing everyone along on the journey.
