Maude Lemaire, a principal engineer and author of Refactoring at Scale, shares lessons from scaling Slack (2016–2024), GitHub (2024), and Cursor, covering database sharding, load testing, Rails monolith constraints, and where AI coding tools fall short for platform work.
Early career and path to Slack
Lemaire started in microbiology, switched to CS after an intro programming class sparked the “I type and the computer does things” feeling.
Interned at Rent the Runway, drawn by Camille Fournier’s leadership and the blend of fashion and tech; Fournier later helped her land at Slack.
Joined Slack in 2016, pre-enterprise GA, when the largest workspaces hit ~30k users and the architecture sharded everything by team.
Slack’s 2016–2017 architecture and the enterprise scaling crisis
Team-sharded model: each workspace lived on one MySQL shard; large customers got dedicated shards, smaller ones shared.
Websocket connections were also team-affinitized — all users in a workspace hit the same chat server.
Enterprise Grid introduced an overarching “enterprise” row linking multiple workspaces, but DMs and org-wide channels forced everyone to query a single enterprise shard.
Channel membership query became a massive join/union across four separate tables (public channels, private channels, group DMs, DMs), causing severe latency.
Migrating channel membership to a unified table
A teammate on the mentions path realized the join could not be optimized further; Lemaire took on the migration.
Blame framework context: Slack’s PHP codebase had raw SQL strings in 400+ call sites, no ORM, no data-access layer.
Strategy: created a minimal abstraction layer (one class per query pattern), copy-pasted every raw SQL call into it, feature-flagged every call site.
Double-writes + shadow reads + backfills for months; compared old vs new results in data-warehouse dashboards.
Incident: flipped the read flag to all of Slack during “gather hour,” saw discrepancies, forgot the flag was on, truncated the new table to re-backfill — blank sidebars for ~30 min for all internal users (who were at the party, so impact was minimal).
Lesson: even with exhaustive shadowing, a manual mistake during cutover can cause a visible outage; keep rollback instant.
Trigger: a prospective 500k-user customer (later Amazon) forced an 18-month plan into months; COVID accelerated it further.
Failure modes to simulate: thundering herds (CEO posts → 100k users react in seconds) and event floods (power users in 30+ workspaces receiving duplicate messages per workspace).
First attempt: headless Chrome instances in K8s pods — 37¢/instance/5-hr run → ~$150k per test, pods crashed constantly, half the window spent recovering.
Pivot: lightweight Go processes that open a raw websocket, replay recorded API/edge-cache request sequences, and exercise the same fan-out paths.
Recorder tool: slash command in Slack captured every outbound/inbound websocket message and API call for a flow; Lemaire curated the trace into a reusable script.
PM involvement: gave PMs the recorder so they could contribute real-world flows (e.g., canvas interactions) to the load suite.
Takeaway: understand the product’s actual usage patterns, simulate only the critical path, keep the harness cheap and editable.
GitHub: 16 shards, scatter queries, and the Rails monolith
Context: GitHub still ran heavily on-prem, moving to Azure; space/power running out.
Vitess adoption: wrapper around MySQL for dynamic resharding; Slack had thousands of shards keyed by channel, GitHub had 16 shards for all PRs and issues.
Why only 16? Scatter queries — queries without the sharding key (PR ID) that must fan out to every shard. More shards = linearly slower scatter queries.
ActiveRecord opacity: Rails’ ORM hides SQL; a 5-line function can generate a complex join with no visible query string. Lemaire paired with a Rails expert to extract stars queries and realized no one sees the generated SQL, making scatter-query elimination nearly impossible.
Files page example: 100+ MySQL queries across 3+ clusters to render the repo homepage; counts are live-calculated, not cached, because ActiveRecord only auto-caches exact-query matches.
Product-driven fix: stop counting past 99 (show “99+”) — a product decision, not an engineering one.
AI coding tools: where they help and where they don’t
Not helpful: performance investigation. LLMs cannot trace async, multi-hop flows (context building → inference proxy → LLM provider → streaming gRPC → client) well enough to find the real bottleneck.
Example: at Cursor, LLM (Opus) proposed parallelizing four independent tasks to improve time-to-first-token; longest task was always the bottleneck, and the other three ran for <1% of users — pure waste.
Real wins: trivial config changes (raise Redis TTL where safe), boilerplate implementation once the design is fixed.
Instrumentation: LLMs naively sample traces at the exporter (duplicating sampling points), creating useless million-span traces. Lemaire manually instrumented the critical path, printed a giant sequence diagram on blueprint paper, and annotated it by reading code.
Testing: LLMs write tests that assert the code they just wrote. Lemaire runs mutation testing to force edge-case coverage.
Workflow: constant LLM chat for exploration, but manual verification of every output; platform work demands a foundation that covers 100% of cases, not 80%.
Platform ↔ product engineer rotation
Best platform engineers start as product engineers who feel the pain repeatedly, then move to platform to solve it once.
Risk: platform folks drift from product reality; product folks ship “AI slop” without guardrails.
Fix: mandatory rotations — product engineers embed on platform teams, platform engineers embed on product teams.
Analogy: embedded SRE model; today the “spaghetti has a life of its own and wants off the plate” — platform must red-team its own assumptions.
Goal: platform provides guardrails that let product move fast safely; product teaches platform what “fast” actually looks like now.
