This episode features Eric, founder and CEO of Modal, discussing the company’s three-year journey from a serverless compute platform to a multi-product AI infrastructure provider now at roughly $1B revenue run rate, covering GPU market dynamics, core technology investments, inference optimization, the rise of coding agents and sandboxes, neocloud economics, and future infrastructure bets.
Modal’s journey and product evolution
Modal started in late 2021 as a general-purpose serverless function platform for data, AI, and ML workloads, well before the ChatGPT moment.
The first major growth driver was Stable Diffusion hosting in 2022–2023, followed by expansion into audio, video, and adjacent generative AI use cases.
Modal deliberately began with a low-level compute primitive (serverless functions) to serve ML engineers and researchers who wanted to write code and move fast without managing infrastructure.
Early investments included a custom file system for fast container image loading and a custom container runtime and scheduler (no Kubernetes or Docker), which became foundational differentiators.
Sandboxes launched in 2023 as an asymmetric bet — low initial investment, high upside — anticipating code-generation use cases; they exploded in 2025 when models became proficient at writing code and now represent ~30% of revenue (~$300M run rate).
A dedicated training product was added recently, and the DeepSeek moment (early 2024) prompted a focused push into LLM inference, which also grew to ~30% of revenue.
Modal now offers a dedicated inference product (single-tenant deployments of state-of-the-art models with one click/CLI) and shared endpoints with token-based pricing, moving up the stack while leveraging its compute/storage foundation.
The long-term vision is a full portfolio of primitives for AI applications: compute, storage, serverless functions, sandboxes, training, inference, batch processing, and higher-level agent tooling.
GPU market dynamics and compute procurement
The GPU market remains highly fragmented and inefficient — “like an open-air drug market” — with deals often sourced through late-night WhatsApp groups, hotel lobby meetings, and personal networks (e.g., a VP’s cousin’s golf buddy).
Modal works with ~25 different cloud providers (including many neoclouds) and executes 2–3 compute deals per week, aggregating demand to manage capacity efficiently.
Compute is not a fungible commodity: it varies by chip type (H100, A100, B200, L40S, RTX Pro 6000, etc.), memory, networking, disk, location, and reliability.
Reliability is a major differentiator; neoclouds (often ex-crypto miners) frequently have failures — GPUs falling off the bus, node outages — and Modal automatically detects and routes around them.
Pricing data from third-party GPU marketplaces is moderately helpful for discovery but not fully trusted; internal pricing diverges from public listings, though external markets may mature over time.
Model-to-chip matching is largely memory-constrained: pick the cheapest GPU that fits the model’s memory requirement; Modal aims to automate this and dynamically shift workloads across GPU types to maximize utilization.
Older chips (Hoppers, A100s, L40S) remain heavily used for non-LLM workloads (audio, bio, smaller models) where they are more cost-efficient; H100s are still the most common GPU at Modal.
Modal is ramping aggressively on Blackwell (GB200) but sees sustained demand across the chip spectrum.
Core technology infrastructure
A massive source of alpha is willingness to build hard infrastructure: custom file system, container runtime, scheduler, and RDMA-based peer-to-peer data transfer between nodes.
This enables extremely fast cold starts (seconds for large models), CPU/GPU snapshotting, and rapid autoscaling — critical for both developer experience and production autoscaling.
The stack uses gVisor for isolation and recently added KVM for specific tasks; almost no off-the-shelf orchestration software.
These investments compounded over years and now yield unique performance (e.g., loading multi-hundred-GB models in seconds via RDMA) that is hard to replicate.
Inference optimization and unsolved problems
Inference remains an empirical science: teams read papers, test kernels, and iterate; Modal has 20–30 engineers focused solely on LLM performance.
Key techniques: disaggregated prefill (prefill on one node set, decode on another, connected via RDMA), KV cache optimization, quantization, mixture-of-experts routing.
The paradigm may shift (e.g., post-transformer architectures), so flexibility is essential; open-source engines (vLLM, SGLang) are now state-of-the-art and Modal contributes heavily to both rather than building proprietary alternatives.
Modal’s alpha lies in the infrastructure layer (autoscaling, scheduling, multi-cloud orchestration), so leaning into open-source inference engines is strategic.
Long-term, Modal wants to automate model-to-chip placement and dynamically shift workloads across heterogeneous GPU fleets to improve utilization and cost efficiency.
Multi-product strategy and customer examples
The durable differentiator is a multi-product platform, not just token serving (which is commoditizing via routers like OpenRouter).
Customers want a toolbox: sandboxes for code execution/RL rollouts, inference, training, batch processing, file storage — all integrated.
Prioritization is ruthless: ~150 engineers, constant trade-offs, pausing projects when higher-ROI opportunities emerge; resource allocation is the CEO’s core job.
The roadmap is extensive; velocity is high but CI/CD and test infrastructure are under pressure as agent-generated code volume grows (est. 95% of merged lines agent-written).
Coding agents, sandboxes, and agent-native infrastructure
Sandboxes became the primitive for coding agents (rollouts, evals, agent orchestration); adoption was driven by senior engineers who treat agents as “10 interns” and use extensive test suites to guide them.
Junior adoption is lower; senior engineers benefit most from agent leverage.
Agents want the same things developers want: fast feedback loops, good errors, great docs — so Modal’s existing DX investments serve both.
Deprecation is harder: agents train on stale code, so Modal extended deprecation windows and runs automated pipelines where LLMs continuously write/run Modal code to catch breakage.
“Agent SEO” matters: ensuring LLMs recommend Modal in code-generation contexts creates a flywheel; incumbency in training data is a new moat.
Future agent infrastructure bets: low-latency real-time voice/video (CDN-like inference fabric), training/RL for mid-size models, full VM support (macOS, Windows, Android, iOS), durable object/workflow engines for agent orchestration — but the 5-year architecture is still unclear.
Neoclouds and GPU market structure
Neoclouds are essentially financing vehicles: they lease GPUs, raise debt against customer contracts, and operate at high leverage.
They are decent business models — GPU lifetimes and residual values have exceeded expectations — but structurally undifferentiated and capital-inefficient.
High cost of capital (vs. hyperscalers backed by massive cash flow) limits their scaling; some will fail and resell GPUs, creating a secondary market (“neo clouds” becoming “neo clouds”).
Implicit leverage: GPU commitments often require only ~30% upfront, creating future payment obligations dependent on continued VC fundraising.
Eric is bullish on AI demand overall but expects consolidation; not all neoclouds become trillion-dollar companies.
Future outlook and predictions
LLM model sizes will keep growing (pushing memory needs) until a paradigm shift; people use every bit of available memory.
Multimodal (voice-to-voice, real-time video) is severely underdeveloped — current voice assistants are “horrible” — and will drive demand for low-latency, globally distributed inference.
Robotics inference will be largely on-device, but training, simulation, and testing will run in the cloud.
Most companies won’t do custom post-training; the largest will (Cognition, Cursor), and they drive disproportionate token volume.
RL/post-training as a service faces a product-vs-services tension; Modal aims to productize it.
Every successful infra company becomes multi-product; Modal will own a large adjacent cluster (compute, storage, sandboxes, inference, training, agent tooling) but not everything (e.g., not model training, not applications).
GP rental prices in 12 months: likely higher, because Anthropic/OpenAI are profitable and have zero incentive to slow capacity acquisition.
Quickfire round insights
Biggest mind-change in the last year: sandbox opportunity size far exceeded expectations.
Hiring philosophy unchanged: agent productivity increases demand for engineers (Jevons paradox); short-term friction for new grads, long-term bullish on CS.
Biggest mistake: sequential thinking — in fast markets, must parallelize unblocked work instead of waiting for dependencies.
Swedish tech success: flywheel effect (success breeds success) + cultural entrepreneurial streak (Vikings to IKEA to Spotify to Lovable); Stockholm’s “delusional global ambition” is a feature.
Years 10–20: Modal as a conglomerate with many products, some moonshots; Eric stays involved in high-level resource allocation, less in day-to-day GPU contracting.
Call to action: pip install modal, build on Modal, reach out — Eric loves talking to customers.
