Greg Kroah-Hartman, a Linux kernel maintainer and Linux Foundation fellow, explains how the Linux kernel is built, maintained, and evolved by thousands of contributors worldwide — revealing a development model built on trust, email-based workflows, ruthless efficiency, and a strict time-based release cycle that has made Linux the most widely used operating system on Earth.
How widespread is Linux?
Linux powers an estimated 4 billion Android devices alone, plus servers, Chromebooks, smart TVs, washing machines, watches, 5G modems, electric car charging stations, satellites, air traffic control systems, and financial markets.
Samsung runs its own Linux-based distro across TVs, appliances, and wearables.
The kernel is everywhere — the top five best-selling laptops for the past decade are Chromebooks, all running Linux.
Even 5G modems inside iPhones and other devices run a copy of Linux.
Scale and complexity of the kernel
The full kernel source tree is just under 40 million lines of code.
The core kernel that everyone runs is about 5% of that; the rest is hardware drivers, architecture support, and device-specific code.
A server runs about 1.5 million lines of code; a laptop about 2.5 million; a phone about 4 million.
Phones are the most complex because they contain system-on-chips (SoCs) with power control, clocks, multiple buses, battery management, modem communication, USB, audio, and heterogeneous big/little cores — all requiring distinct drivers.
The kernel’s job is to abstract hardware so user-space programs can run unchanged across different machines.
Kernel vs. user space
The kernel runs in a chip’s protected mode with a flat address space, managing memory, devices, and multitasking.
User-space processes are isolated and each thinks it has the whole machine.
All drivers live inside the kernel (monolithic architecture), not in user space — meaning a bug in any driver can crash the whole kernel.
This monolithic design allows refactoring across driver boundaries, making Linux drivers on average one-third smaller than those in other operating systems.
The one inviolable rule: never break user space. The interface between kernel and user space is a hard guarantee. If a kernel upgrade breaks a user program, it’s a regression and gets fixed.
The development and release cycle
Linux uses a strict 9-week time-based release cycle:
Weeks 1–2 (merge window): Linus Torvalds accepts new features and changes from subsystem maintainers.
Weeks 3–9 (stabilization): Only bug fixes, regression fixes, and reverts — no new features. Release candidates are cut weekly.
This replaced the old 2–3 year development cycles, which created pressure to accept unfinished features.
The short cycle means maintainers can say “not ready, try again in 9 weeks” without guilt.
During the 7-week stabilization period, new features accumulate in the linux-next tree, which is merged and tested daily in preparation for the next merge window.
Linus does not auto-pull from subsystem trees — maintainers must explicitly request pulls, giving them control over quality.
How a patch flows through the system
A contributor (Chester) submitted a 4-line patch adding a new USB device ID for a Cat 6 modem to the USB serial “option” driver.
The patch was sent via email to the subsystem maintainer (Johan) and the USB mailing list, with an extensive description explaining the hardware context and a Signed-off-by line certifying GPL licensing.
Chester noticed a comment formatting issue and sent a version 2 patch with a changelog explaining what changed — standard practice so reviewers don’t have to dig through old emails.
Johan explained the patch arrived during the merge window (when no new features are accepted) and would be queued for the next cycle.
Johan applied the patch to his tree, fixed the comment formatting himself, and sent a pull request to Greg (USB subsystem maintainer) using git request-pull with a signed GPG tag.
Greg trusted Johan’s review, pulled the change without re-reviewing each patch individually, and forwarded it to Linus.
Linus merged it into his tree. The entire process from first email to mainline took about a week and a half.
The fix also flowed into the stable tree (maintained weekly by Greg and Sasha) and was backported to long-term stable (LTS) kernels.
The hierarchy of trust
Linux development is a pyramid of trust: ~4,000 developers → ~800 maintainers → ~200 subsystem trees → Linus.
When a maintainer accepts code, they put their name on it and become responsible for it — so they must trust the contributor will be around to fix problems, or that they understand the code well enough to maintain it themselves.
This trust model was learned the hard way: a major networking feature was merged, the author disappeared, and it took six months to unwind.
For simple driver changes (like adding device IDs), maintainers accept almost anything. For core kernel changes, scrutiny is much higher.
Greg receives ~1,000 emails per day and reviews them in batches — for example, processing 200 USB-related emails once a week.
Testing infrastructure
linux-next is a daily merge of all subsystem trees, built and boot-tested automatically by a developer in Australia.
Kernel CI runs automated tests on real hardware across multiple labs, with results reported back automatically.
The zero-day bot applies patches from the mailing list and performs build/boot testing.
Stable tree maintainers also use pattern-matching tools (sometimes called “AI” before it was trendy) to identify bug fixes that should be backported but weren’t tagged for stable.
Companies like Google and Collabora provide testing infrastructure for older LTS kernels, running them on real devices.
Stable and long-term stable (LTS) trees
When Linus releases a new kernel (e.g., 6.4), Greg forks it as a stable branch and releases weekly updates (6.4.1, 6.4.2, etc.).
Every fix in the stable tree must first be in Linus’s tree — the trees never diverge.
This acts as a forcing function: developers must get fixes upstream to Linus before they can appear in stable.
LTS kernels are picked once per year and maintained for 2–6 years. Android phones typically run 5-year-old kernels that still receive backports.
Greg maintains ~4 LTS kernels simultaneously. Older code is harder to maintain because it has diverged from current APIs and conventions — backporting a fix to 10-year-old code requires more skill than writing the original fix.
Some Spectre/Meltdown fixes were never backported to certain LTS kernels because the changes were too complex.
No product managers
Linux has zero project managers or product managers at the community level.
Companies that contribute (IBM, Google, Intel, Red Hat, etc.) handle project management internally — the community only sees the finished patches.
The development model is ruthlessly efficient: no RFCs, no design documents, no planning meetings. Code that works is the proposal.
Greg dislikes the “wouldn’t it be great if…” approach — he wants to see working code as proof a problem can be solved.
Once a year, core maintainers meet to discuss process (not technology) and refine how things work.
Scale of change
Linux runs at 9–10 changes per hour in the mainline tree.
Stable kernels see 30–40 changes per day.
About 10 CVEs (security vulnerabilities) are reported per day.
The project has never plateaued — it keeps growing because new hardware constantly requires new support.
Greg’s answer to “when will Linux be done?”: “When they stop making new hardware.”
Rust in the kernel
Linux already has 25,000 lines of Rust (mostly bindings, not functional code).
The QR code crash dump feature was written in Rust.
Writing a driver in Rust requires bindings to every kernel subsystem the driver touches — this is complex because C and Rust have very different memory and object models.
Rust fixes about half of all kernel bugs — the “stupid” ones: off-by-one errors, missing error path cleanup, forgotten unlocks. It does not eliminate logic bugs.
Rust is forcing the C code to be better documented and cleaner. Even if Rust disappeared, the C code is improved.
Resistance exists from some core developers who prefer single-language projects, but the project has reached a tipping point: Red Hat is writing new NVIDIA GPU drivers in Rust, and Apple’s GPU drivers for MacBooks are being written in Rust.
Government mandates against memory unsafe languages in products make Rust adoption necessary for Linux’s future.
Why Linux won
Linux succeeded because companies contribute selfishly to solve their own problems, but the solutions are forced to be generic — benefiting everyone.
Example: Embedded developers needed power efficiency; the resulting power management saved data centers billions of dollars.
Example: Multiprocessor support for servers now benefits phones with 16 cores.
It’s cheaper for companies to pay a few engineers to add features to Linux than to build and maintain their own OS.
Companies make money selling hardware, support, or services around Linux — not the OS itself.
The open-source model creates better engineering because your name is on every change, publicly — creating strong incentives for quality.
Linux has had one team working on its USB stack for 20 years, while Windows has had five different teams — depth of expertise matters.
Contributing to Linux
Contributing is easier than most people think: the workflow is email-based git, and even a one-line fix is welcome.
Professional benefits:
It’s a public resume showing you can work with others and existing codebases.
Engineers who contributed were preferentially hired at IBM and elsewhere.
You learn from the best developers in the world — exposure to corner cases and scale you won’t find at any single company.
Good first contributions: fixing spelling mistakes, coding style issues, and dead code in designated “beginner” drivers.
Resources: kernelnewbies.org has guides on writing your first kernel patch. Greg has given university workshops where students write and submit an accepted patch in two hours.
LLMs are used for pattern-matching bug fixes and finding backport candidates, but they don’t replace the human review process. Most driver code is too specialized for LLMs to generate meaningfully.
Greg’s day-to-day
Greg has worked for the Linux Foundation for 13 years as a fellow — one of only a handful of people paid to work on whatever they think improves Linux.
His daily work: reading and reviewing ~1,000 emails, applying patches to his subsystem trees, requesting pulls from Linus, maintaining stable and LTS releases, and doing security fixes.
He describes maintainers as “editors” — they used to be writers, but now mostly critique others’ work.
He still does some coding (~80 changes last year) but mostly focuses on review and process improvement.
Rapid fire
Most memorable patch: A high school student (17 years old) submitted excellent hardware patches that no one could figure out how he reverse-engineered. He was invited to the maintainer summit, showed up with his mom, and is now a professor at Stanford.
Another proud moment: Getting Braille display drivers merged into mainline after years of being out-of-tree, making them available to everyone.
Favorite language: C (30 years of daily use). Rust is growing on him because it lets him write “sloppy” code that still works.
Recommended books:Code Complete (coding style and patterns matter) and Programming Pearls (bit fiddling and algorithms that still matter today).
