Sam Newman, author of “Building Microservices” and the new “Building Resilient Distributed Systems,” discusses the realities of distributed systems, why microservices are an architecture of last resort, and how to build resilience through observability, idempotency, and socio-technical practices — while offering a grounded perspective on AI’s role in software development.
Sam’s background and formative experiences
Sam got into tech early via UK home computers (Acorn Electron, Spectrum), studied software engineering at De Montfort University, and did a sandwich year at ALSTOM working on European Space Agency turbine software.
A pivotal early moment: refactoring 8,000 Fortran 77 files by hand vs. learning regex in two days — the “god mode” thrill of automation that still drives him.
Joined Thoughtworks (~400 people) over a bank job because the recruiter said “I don’t really know what you’ll be doing” — worked with Dave Farley, Jez Humble, helped build early CI/CD tooling, and spent 18 months at Google teaching test automation.
Thoughtworks culture: constant reinvention, challenging constraints, “say yes to everything” — but few entrepreneurs emerged; innovation happened within enterprise constraints, not startup product building.
Left Thoughtworks (~4,000 people) to avoid institutionalization; went independent, wrote first microservices book in spare time.
The rise of microservices and what they actually are
Term coined by James Lewis at a 2011-12 architectural symposium in the Lake District; Martin Fowler and James published the first post.
Sam came at it from continuous delivery lens (10 years focus); James from architectural lens — both saw finer-grained SOA with independent deployability.
Sam’s definition (he wrote the book, so “my definition wins”): two strong opinions — (1) services must be independently deployable (unit of deployment = service boundary), (2) default boundary definition around business domain, not technical layers.
Uber/Netflix adoption: Uber mandated “everything new is a service” to escape monolith, creating thousands of services organically; Netflix had fine-grained SOA before the term existed, with teams already scoped to product areas.
Microservices as architecture of last resort: they push aggressive distribution of state; only adopt if you need organizational autonomy (teams acting independently) — it’s a means to that end, not magic.
Enterprise vs. tech companies: tech domains are simple and co-evolve with software; enterprises codify decades-old paper processes, making domain modeling far harder.
Specs vs. code as source of truth
Ongoing industry debate: is the system definition in code, specs, or people’s heads? Charity Majors: “production is truth.”
Spec-driven development requires defining “what good looks like” (functional, operational, business requirements) and verifying output — if you can’t define good, don’t do it.
Tacit knowledge problem: code reviews and pairing shared context; AI-driven solo work erodes shared mental model. Shang (Chang?) dropped human code reviews to spend more time on design docs — acceptable if they’re doing deeper thinking, not just ignoring code.
Collaboration remains essential: whether 15 people or 5, collective model-building must happen somewhere (specs, design docs, pairing).
Building Resilient Distributed Systems — the new book
Origin: Velocity conference (2015), David Woods’ resilience engineering paper (impenetrable at first), reread every 6 months until it clicked.
Goal: a “rubber ring” for developers dropped into distributed systems — start with tractable primitives (observability, timeouts), build toward resilience engineering concepts (socio-technical systems, safety vs. resilience).
Resilience is not binary; it’s a degree. Trade-offs everywhere: Monzo runs a standby bank (different cloud, stripped features, 1/10th cost, complete rewrite) — but don’t start there.
Need observability first to know how close you are to the red line; then business conversation: how resilient do we need to be? (Zoom accepted US-East-1 outage impact — conscious trade-off).
Must communicate with stakeholders in their language, not “99th percentile latency.”
Three rules of distributed systems (the root causes)
Rule 1: You cannot send information instantaneously from A to B (physics; you control payload/serialization, not cables/BGP).
Rule 2: Sometimes the thing you want to talk to isn’t there (rabbits eating cables, SAN fires, load balancer failures — you can reduce probability but not eliminate).
Rule 3: Resource pools are not infinite (CPU, memory, I/O, connections); vast majority of outages = resource saturation somewhere (often triggered by retry storms from Rule 2).
All complex distributed systems concepts (CAP, PACELC, Byzantine consensus) trace back to these three.
Observability as foundational primitive
Single program: high-quality signals for free (debugger, core dump, thread dump, CPU 100% = signal).
Distributed system: signals harder to get, don’t correlate without work, individual signals insufficient.
OpenTelemetry (vendor-agnostic) provides structured events (logs, metrics, traces) as primitives.
Traces = spans stitched by trace ID; spans = activities with timing, nesting, attributes.
Without observability, you can’t set timeouts, detect degradation, or know if you’re meeting SLOs.
Idempotency: keys vs. fingerprints
Idempotent operation: safe to retry (e.g., elevator button); non-idempotent: toggle switch.
Payment example: client sends “pay Sam £100,” no response — did it succeed or not? Retry risks double charge.
Idempotency keys (preferred): client generates UUID, sends with request. Server checks “have I processed this key?” — safe retry. Requires client + server support; build in from start.
Fingerprints (retrofit): server hashes request fields (amount, payee) to detect duplicates. False negatives: legitimate same-amount transactions rejected (e.g., two 40-kroner purchases in 5 mins). Time-bounded storage helps but imperfect.
Combine both: key for identity, fingerprint to catch parameter changes between retries.
AWS APIs use keys; Stripe, PayPal same pattern.
Thundering herds and mismatched capacity
Generic definition: too many requests for too few resources (Rule 3 again).
Types: malicious (DDoS — use Fastly/Cloudflare), self-inflicted (cache collapse: restart → misses → origin overload — mitigate with cache snapshots to disk, delay service restart), retry storms (Square 2017: Multipass hammered Redis with 500 instant retries — fixed with backoff, externalized config), legitimate success waves (Healthcare.gov 36 states at once; Twitter — constrain signups, invite codes).
Scaling out works for success waves (transient), not DDoS (cost explosion, still can’t keep up).
Mitigations depend on context — know your business, know your load profile.
Business context drives resilience decisions (fail open vs. fail closed)
Same technical uncertainty, different business impact → different decision.
E-commerce: inventory service down → sell anyway (likely have stock, can back-order, worst case refund). Fail open.
Uber: early years fail open (growth > losses); later fail closed (profitability focus).
Decision is per-feature, even per-transaction (€10 vs €1,000 ride).
CAP/PACELC models formalize this, but resolution starts with: “Rule 2 — I can’t talk to that service. Product owner, what should we do?”
Resilience engineering: four concepts (David Woods)
Robustness: absorb known perturbations (Kubernetes restarts failed pod). Engineers over-focus here. Risk: robustness mechanisms add complexity → new failure surface area (Kubernetes itself fails).
Rebound: recover quickly when things go wrong (detection + recovery time). Requires humility: “what if we got it wrong?” Drills, chaos engineering (Netflix Chaos Monkey), Uptime Labs training, Google’s “Wheel of Misfortune.”
Graceful extensibility: deal with surprise (rabbits, data center fire). Requires human flexibility — command-and-control hierarchies fail here. Drills, cross-training, psychological safety to raise issues.
Sustained adaptability: continuous learning and adaptation over long timescales. Read others’ incident reports (The Void), share learnings, adapt system and organization. Psychological safety is prerequisite — without it, issues hidden, no learning.
AI and resilience: macro vs. micro
Macro: model providers (OpenAI, Anthropic) need $2.6T revenue by 2030 to justify capex; entire software market is $1.4T. Economics don’t stack — can’t fix this.
Micro (tactical): useful tools exist (correlation analysis, incident.io). But hedge: multi-vendor, multi-model (Google/AWS/Azure for availability), break agents into workflows where deterministic code can replace non-deterministic steps.
LLMs have no causality: “why did LLM delete my DB?” — no concept that action A causes outcome B. Guardrails are band-aids; world models (pre-LLM) handle causality better.
Industry naively overestimates LLM capabilities because they “seem smart.”
Cognitive debt and cognitive surrender
Cognitive debt: erosion of critical thinking/reasoning from AI overuse (studies show chatbot use reduces critical thinking). Shared mental model breaks when everyone goes different directions with AI.
Cognitive surrender: “LGTM” for AI output — approving without review. Oncologist analogy: AI highlighting “look here” (human in loop, augmented) vs. AI saying “here’s what I approved, sign off” (human out of loop, rubber stamp). Latter removes expertise, creates single point of failure.
AI was supposed to free us from drudgery for critical thinking; instead: more context switching, longer hours, less thinking time. Not AI’s fault — how we’re using it.
Modular architecture and AI “software factories”
Don’t make one big black box (forces all-or-nothing factory adoption). Modularize: humans design module boundaries and connections; AI runs wild inside well-bounded, easily changeable modules.
Adopt factory approach per-module: low-risk, non-critical-path, great test coverage, good observability/SLOs, steady state. Verify output on three axes: development requirements (maintainability), operational (latency, uptime), business (features work).
Start with spec-driven development (Kiro, Tessl, Spec Kit) — define “good” as input, verify as output. If you can’t define good, don’t do factory — but specs still valuable.
May be blind alley (like Model-Driven Development), but hope it works this time.
Resources for learning software architecture
Software Architecture Fundamentals (Neil Ford, Mark Richards) — good overview, though Sam disagrees on some points.
Balanced Coupling (Vlad Khononov) — fantastic on modularity; also his DDD book.
Parnas papers (1971-72) on information hiding — the 1970s idea Sam’s been “smuggling” for years.
George Fairbanks — “architecturally evident coding style”: code should communicate architecture; languages lack good module concepts, so go above and beyond.
Practice: modularize existing codebase now (IDE refactoring + tests); make architecture explicit in code (or future specs). Production is truth — network topology reveals real architecture.
