Nebius CTO Dima discusses the company’s evolution from Yandex spinout to independent AI cloud provider, the mechanics of bringing Nvidia’s latest chips to customers, why hyperscalers haven’t dominated the GPU cloud market, and where inference optimization and agentic workloads are heading.
Nebius originated as a Yandex spinout that separated cleanly in 2024, inheriting a legendary infrastructure engineering team but rewriting its entire software platform from scratch
Dima joined before the Nebius name existed; the core hardware operations team (40–50 people) came from Yandex with experience running hundreds of thousands of servers across multiple facilities
The software organization grew to ~200 engineers by end of 2024, most hired after the separation, because the old Yandex cloud stack was discarded and rebuilt as a holistic, software-first platform — not managed bare metal or a patchwork of third-party services
The separation deal finalized in 2024 removed all connections to Russian assets, giving Nebius a clean slate as an independent “AI cloud” (Dima rejects the “neo cloud” label, which now covers 300–500 companies)
The AI cloud market is defined by a structural window: hyperscalers (AWS, GCP, Azure) face chip allocation constraints and massive internal demand, creating space for independent providers to serve external customers at scale
Hyperscalers juggle internal consumption (training, runtime, research) against external rental; chip shortages mean they cannot simultaneously satisfy both at hyperscale
Re-architecting their infrastructure (networking, storage throughput, virtualization) for AI workloads is a massive investment that must be justified by revenue potential — currently a small slice of their business
Dima expects this window to last 2–5 years until chip supply eases, at which point hyperscalers will reorganize and compete aggressively; Nebius’s bet is to grow fast enough to fight as equals
The market will support a long tail of providers (like Hetzner, DigitalOcean alongside AWS today); consolidation will happen but hundreds of players will persist
Bringing new Nvidia generations (Blackwell, GB300, Grace Blackwell) to production is a messy, iterative co-engineering process where Nebius often operates at the bleeding edge
Nvidia shares prototypes and schematics early, but designs shift constantly during compressed development cycles; lab samples often differ from earlier specs
Nebius integrates support into its full stack (virtualization, storage, networking, isolation) and frequently discovers novel issues at cluster scale, reporting patches back to Nvidia
The shift from server-level (H100/B200) to rack-level architectures (GB300/GB200 NVL72) changes maintenance models — you service a rack, not a single server — requiring new software primitives
Despite the chaos, the process has normalized: constant back-and-forth with Nvidia hardware and software teams, daily incremental progress, eventual production readiness
Customer needs segment less by workload type (training vs. inference) and more by organizational maturity and compliance requirements
Frontier labs (seed rounds in billions) prioritize raw GPU, storage throughput, and availability; they often don’t care about data center geography
Enterprise customers have smaller contracts but demand certifications, compliance (multi-jurisdiction), security posture reviews, source code access — “being a real cloud”
Storage throughput has exploded: 2 years ago NAS meant gigabytes/second; now customers routinely pull 500–700 GB/s from a single shared volume across thousands of GPUs
This high baseline lets Nebius serve the broader cohort; the platform is universal, with differentiation handled at the orchestration and model-serving layers, not hardware
Inference optimization remains a vast unsolved surface; disaggregated prefill/decode is talked about but rarely production-ready, while lower-hanging fruit abounds in model compression, kernel tuning, and specialized agents
Running open-weight models “vanilla” yields underwhelming cost/performance; 95% of Nebius customers run modified, fine-tuned models for scoped domains
Selective pruning, compression, and speculative decoding can retain frontier reasoning on a task subset while shrinking model size — critical for enterprise adoption where predictability and latency matter more than open-ended capability
Nebius acquired a model-optimization team (summer 2024) and now runs specialized groups for kernel optimization and model compression; this is core to running inference-as-a-service seriously
Disaggregation (splitting prefill/decode across nodes) is complex to operate with SLAs; promising in prototypes and for largest customers using full racks (Grace Blackwell NVL72), where fast intra-rack interconnect mitigates traditional penalties
Rack-level disaggregation lets Nebius collocate multiple workloads per rack, solving packing problems via managed inference service
Tavily acquisition (web search) was driven by product velocity and complementary infrastructure expertise; it fits a broader agentic tooling strategy alongside sandboxes and other building blocks
Tavily had strong go-to-market motion; Nebius contributed low-level scale expertise (running large web indexes) to accelerate them
Web search is a key primitive for long-running agents that consume significant tokens pulling information
Nebius is building an agentic toolkit: managed inference + search + sandboxes + other early-stage products, exposing primitives customers need to build agents
Data center strategy: Nebius controls everything inside the walls — whether owned, leased, or “asset-light” (third-party capital, Nebius platform) — to guarantee uniform SLAs and stack ownership
No renting pre-built GPU clusters from third parties; internal build-out follows Nebius spec (networking, power, cooling, firmware, server/chassis design)
Software sees zero difference across facilities; hardware is standardized to known, qualified configurations
Bottlenecks are everywhere (GPUs, CPUs, NVMe, transformers with multi-year lead times, skilled labor, facilities, power, transceivers) — the only real moat is execution ability: sourcing, sequencing, flexibility, project management at scale
Workload evolution (agents, post-training, long-running) doesn’t fundamentally change cluster build-out; a high universal baseline absorbs variation, with optimization at packing and orchestration layers
New architectures (e.g., Nvidia LPX from Groq deal) are “0.5 generation” increments — novel interconnect, some restrictions — but don’t demand custom cluster designs
The platform exposes primitives for collocation, fractional allocation, and workload-specific tuning; hardware stays general-purpose
Dima’s reflections: underestimated single-customer scale (30 MW facilities now too small for one client; 700 GB/s storage throughput), conservative on AI coding agents (full autonomy barred from core services, human accountability enforced), and watching outcome-optimization loops for agents over model-scale races
AI coding went from “total slop” to daily tooling in one year; Nebius allows unrestricted token usage but monitors for ineffective patterns, mandates human ownership of all PRs/on-call
Model intelligence gains are accepted as given; the frontier is reliability of multi-turn agentic outcomes — harnesses, tools, quality gates, CPU-heavy orchestration that runs minutes between LLM calls
Nebius runs an internal research group on outcome optimization: increasing success probability without changing the model, via evolving harnesses and toolchains per task
