Efficient scaleups in 2024 vs 2021: Sourcegraph (with CEO & Co-founder Quinn Slack)

The Pragmatic Engineer 1h6 9 min #13
Efficient scaleups in 2024 vs 2021: Sourcegraph (with CEO & Co-founder Quinn Slack)
Watch on YouTube

Summary

  • Quinn Slack, CEO and co-founder of Sourcegraph, joins The Pragmatic Engineer to discuss how the realities of running a tech scale-up have shifted between 2021 and 2024, how AI is reshaping software development, and how Sourcegraph’s own engineering culture and practices have evolved over its 11-year history.
    • Sourcegraph is a code search and intelligence platform founded in 2013, now with about 180 employees, over $248 million in funding, and customers including Uber. Quinn still codes daily and sees that as essential to his effectiveness as CEO.
    • The conversation covers Sourcegraph’s origin story, the 2020–2021 growth frenzy and its aftermath, the company’s approach to AI (including its product Cody), changes to compensation and team structure, and Quinn’s personal journey from engineer to long-tenured CEO.

How Sourcegraph started and how it has evolved over 11 years

  • Quinn has been a lifelong developer who loves coding and was frustrated by how hard it was to navigate large codebases at banks and in open source projects like Chrome, OpenSSL, and Firefox.
    • Existing code search tools in 2009 were primitive, and even at large organizations there was no good way to find code ownership or understand why something was broken.
    • His co-founder Beyang had seen Google’s internal code search tool, which was beloved by developers there, and they wanted to bring that experience to every organization.
  • Sourcegraph was founded to sit at the intersection of something the founders would personally use, could build, and that every developer would benefit from.
    • Code search was a rare gap in the developer tools landscape because it aggregated code across many repos, languages, and code hosts — unlike most dev tools, which are fragmented.
    • The company expected the journey to take about a year; it has now been 11 years.
  • Uber was one of Sourcegraph’s first major customers, though that came about five years after founding.
    • By the time the pandemic-driven growth frenzy hit, Sourcegraph had years of history, staying power, and a clear identity, which helped it make better use of momentum and funding than younger, less-established companies.

How scale-ups have changed between 2021 and 2024

  • In 2020–2021, Sourcegraph experienced explosive growth — more than 30X revenue growth during the pandemic period.
    • Developers were in extreme demand, and many joined companies because they saw valuations skyrocket between their interview and their offer, creating unsustainable expectations.
    • Quinn regrets many decisions from that period: the company shipped features and started projects that shouldn’t have made the cut, and too much management time was spent managing growth with a “peacetime” mentality when the moment actually demanded wartime urgency.
  • The company hired many talented people during the boom, but not all of them had the “hacker” or founder mentality of building something from zero to one.
    • Early-career hires in particular were not told clearly that achieving their ambitious goals would require intense effort — nights, weekends, and grinding — and Quinn sees this as a disservice.
  • Steve Yegge (of Google code search fame) was brought in as head of engineering, injecting a focus on speed, iteration, and high standards.
    • Under his influence, Sourcegraph shifted from primarily self-hosted enterprise deployments with month-long feedback cycles to shipping new versions of Cody and code search many times per day.
  • Quinn’s biggest personal lesson from the 2021 period: stay close to the code and the product.
    • He codes nearly every day and believes this fluency is essential for making good decisions as CEO of a developer tools company.
    • He pushes back on the common advice that non-engineering executives should never write code, arguing that AI tools make it even more accessible and valuable for people in all roles to engage with code.

Sourcegraph’s approach to AI and large language models

  • Quinn is excited about AI’s potential to eliminate toil from software development, but favors a gradual, bottom-up approach to autonomy over ambitious “replace the developer” visions.
    • He compares the current moment to the early days of self-driving cars: the winning approach was not concept cars with no steering wheel, but putting a steering wheel in and incrementally shifting driving responsibility from human to AI (100% → 99% → 98%…).
    • Sourcegraph focuses on automating the rote, error-prone parts of every task — like updating a changelog from a diff — and stacking up small increments of automation rather than attempting one-shot, fully autonomous code generation.
  • Two things matter for AI in software development: context and ground truth.
    • Context means giving the AI access to the right code, data, and runtime information.
    • Ground truth means the AI must be able to check its work against objective signals — type checking, tests, deployment success, error rates, and ultimately business outcomes like revenue impact — rather than reviewing its own output.
  • Changelogs are both a customer-facing feature and an internal discipline signal.
    • Maintaining a changelog requires a simple, consistent process that developers actually follow, which is harder than it sounds.
    • A sparse changelog is a warning sign that the team isn’t shipping much.
  • AI is currently much better at writing new code than understanding and working within existing, complex codebases.
    • Most developers work in large, messy, existing codebases where understanding runtime behavior, callers, and dependencies is critical before making changes.
    • Until AI tools can do staged rollouts, work with feature flags, and notify on-call engineers, they won’t be able to safely make changes in production systems.
    • Quinn thinks it’s acceptable for companies to accumulate more tech debt now if they’re moving fast, on the bet that AI will help clean it up in a few years — but this is a cautious bet, not a blank check.
  • AI will reshape the developer tools landscape because AI will become a primary consumer of those tools.
    • Tools like logging, issue tracking, deployment, and security scanning were built with human-facing UIs and enterprise features. The AI versions need to be fast, lean, and programmatically accessible — piping performance and log data back to AI for rapid iteration.
    • Fast builds (under a second for targeted test runs) become enormously valuable when AI can try thousands of code changes and see which ones pass.
  • Current limitations of AI in development:
    • Users often treat AI chat like a command-line tool, asking free-form questions without understanding that the AI only sees what context is explicitly provided (similar to pasting files into ChatGPT).
    • Runtime data — what values a variable actually holds, what calls a function at runtime — is critical and underutilized. Most developers still rely on printf-style debugging rather than using debuggers and profilers, and AI tools don’t yet bridge this gap well.
    • There’s an opportunity to bring powerful runtime and perf tools directly into the editor so developers (and AI) can access them with one click, dramatically increasing adoption.

Why early adopters of AI coding tools have an advantage

  • New graduate engineers tend to be more fluent with AI coding tools than senior engineers, but they often don’t realize how low industry-wide adoption still is.
    • Their early adoption is a genuine competitive advantage, especially since the engineers interviewing them may not even understand how coding workflows are changing.
  • Quinn is optimistic about the future of software engineering despite fears of job displacement:
    • Software engineers have been taking jobs from other industries for 40 years; now the field itself is turning over, as it always has (editors, languages, deployment practices all change constantly).
    • Engineers who master AI tools may become so productive that they no longer need a traditional manager, can do the work of PMs and salespeople, and capture much more of the value they create — potentially earning significantly more.

Changes at Sourcegraph since the 2023 Pragmatic Engineer deep dive

  • A previous Pragmatic Engineer deep dive documented Sourcegraph’s culture: default transparency, globally equal salaries, full remote work, DRIs (directly responsible individuals), RFCs, job fairs, and well-defined career levels.
    • Most of these practices remain, with some tweaks.
  • The “job fair” experiment:
    • To rapidly shift company priorities from code search to building Cody (their AI product), Sourcegraph ran a “job fair” for about six months where engineers self-selected into the highest-priority projects every month or two, rather than being assigned to long-lived teams.
    • This successfully shocked the system and moved people quickly onto Cody, but it didn’t create long-term ownership or team camaraderie, and was hard to manage.
    • After about six months, enough people had shifted to Cody that the company returned to long-lived, stable teams.
    • Quinn sees value in the shock but acknowledges it was disruptive and difficult, and that communication around it could have been much better.
    • Ironically, outside observers praised the speed of Sourcegraph’s AI pivot without seeing the internal difficulty.
  • The broader lesson: engineers want to work on high-impact things tied to the business, and in today’s environment — where funding is tighter and priorities are clearer — this creates more security than the “bubbly money” era of 2021, even though the broader tech layoff environment makes people feel less secure.

Compensation transparency and the shift from location-independent to zone-based pay

  • Sourcegraph was an early adopter of compensation transparency, publishing its compensation approach years before US state regulations (California, New York) began requiring salary ranges in job postings.
    • Quinn supports transparency but notes that most job seekers focus on cash salary and have a poor understanding of equity compensation, which is a major component of total comp at startups and is not included in job postings.
  • Sourcegraph recently moved from location-independent pay (same salary for a given role regardless of location) to zone-based pay.
    • Location-independent pay was adopted early on for simplicity and to enable hiring globally (the first two hires were in South Africa and Phoenix), since dev tools companies couldn’t compete on salary with fintech or social/mobile companies in SF in 2013.
    • Over time, Quinn came to see location-independent pay as creating weird incentives: it makes hiring in high-cost tech hubs (SF, New York, London) very difficult or impossible, and incentivizes employees to move to low-cost regions to maximize their effective compensation.
    • From a shareholder perspective, a company that can’t hire efficiently in key markets is at a disadvantage. GitLab, often assumed to use location-independent pay, has always used zone-based pay.
    • Sourcegraph still pays well above median salary and remains competitive, but now adjusts for location.
    • Quinn believes any company maintaining location-independent pay past ~200 employees is likely struggling to be realistic with employees about business realities, and that thinking like a shareholder — asking what’s best for the company’s long-term survival and ability to pay salaries — should guide these decisions.

Quinn’s journey from engineer to CEO over 11 years

  • In the early days, Quinn coded ~95% of the time and closed the loop directly with customers — building features, emailing users, visiting them, and iterating based on feedback.
    • The first version of Sourcegraph saved him and Beyang 2–3 weeks by finding existing code they were about to rewrite, which felt like a “time machine” and validated the product vision.
  • As the company grew from ~25 people at the end of 2019 to ~180 today, Quinn’s role shifted:
    • He became less involved in some parts of the delivery loop but more focused on product vision, strategy, and customer engagement.
    • He still visits customer sites, sits with groups of ~20 users, hears complaints and feedback, and pairs with them on how they use Sourcegraph — this is as important to him as coding.
  • The hardest part of the CEO transition was learning to trust his own intuition and overrule experts to maintain a single direction.
    • He would hire smart people in sales, marketing, or other functions and be tempted to just do what they said, but realized the company needs one focused direction, which sometimes means overruling people who know more about their specific function.
    • This was a struggle because he initially lacked confidence and was affected by reading complaints about CEOs on Hacker News and Twitter.
    • Having a strong co-founder (Beyang, whom he talks to every day), a good board, and a strong exec team has helped him build that confidence.
  • A typical week now balances product/strategy work, customer visits, and the unavoidable “boring” work of alignment, OKRs, and one-on-ones.
    • Quinn deliberately prioritizes the things he finds fun (coding, customer interaction) because he does those things 100 times better than things he finds boring.
    • When communicating strategy, bringing direct customer feedback from recent visits makes the strategy more credible and easier for others to align around than polished slides without that visceral input.

Rapid fire round

  • First programming language: Perl (last used around 1999–2000).
  • Handy utility apps: screenshot/screen recording tools (Clean Shot, Kazam on Linux, Loom) — used many times per day.
  • Exciting early-stage startup: Ollama, which lets you run AI models locally on your laptop and has new open-source models available within minutes of release. Cody can use Ollama for autocomplete even offline.
  • Favorite books: The LBJ biographies by Robert Caro (~4,000 pages total) — the most in-depth biographies ever written. Also enjoys biographies of Gandhi and Admiral Nimitz.
    • Reading about historical leaders reinforces that great leaders make a few good decisions per year and don’t need to be “on” all the time — they golf, play tennis, and take weeks to travel back from the front. This gives Quinn perspective on what really matters in a leadership role.
  • How listeners can help: keep coding, try new AI tools, hold a high standard, and give honest, critical feedback — push through the hype and say when “the emperor has no clothes” to help make these tools genuinely useful.
Back to The Pragmatic Engineer