David Deutsch: Knowledge Creation and The Human Race, Part 2

Naval 34min 6 min #14
David Deutsch: Knowledge Creation and The Human Race, Part 2
Watch on YouTube

Summary

  • This is the second part of a conversation with David Deutsch, physicist and author of The Fabric of Reality and The Beginning of Infinity, exploring his philosophy of knowledge, science, and progress rooted in Karl Popper’s epistemology. The discussion covers how knowledge is created through conjecture and criticism rather than induction, why there is no final theory or endpoint to discovery, the practical implications of taking theories seriously, and why the Enlightenment culture of open-ended problem-solving—particularly as it developed in Britain—has been uniquely productive.

Popper’s Core Insight: Problems All the Way Down

  • Popper’s entire philosophy rests on one idea: everything begins with problems, and there is no royal road to solving them.
    • This leads to fallibilism (all knowledge is conjectural), anti-authoritarianism (no source is an ultimate authority), and the method of conjecture and criticism.
    • Popper applied this across dozens of fields, but even most supporters only absorbed part of it—they tended to treat their own domain as special rather than seeing the universal method.
  • A common misconception is that science aims to collect a final bucket of truths after which intellectual work ends.
    • Deutsch argues this is “infinitely wrong”: there will never be a time when discovery is complete and humans can rest.
    • Knowledge growth is unbounded; there is no final theory, no endpoint, no sun chairs and cocktails.

Creative Guesses: The Universal Method

  • All knowledge creation—in science, technology, art, even child-rearing—is a process of creative guessing followed by criticism.
    • This is not copying from the environment, not induction, not Bayesian updating toward truth. It is theory-laden conjecture.
    • Deutsch teaches this to his six-year-old: when asked “why,” they start making guesses together.
  • Even the foundations of epistemology are themselves imperfect and require improvement.
    • Popper did not emphasize enough that the purpose of science is explanation, not just testability.
    • Testability matters because in physics it is how we test explanations—but the goal is explanatory knowledge, not merely falsifiable statements.

Experiments, Demonstrations, and Measurements

  • Deutsch draws a sharp distinction between three things commonly lumped together as “experiments”:
    • Demonstration: Shows that something happens (e.g., pouring acid into base and watching color change). No rival theory is being tested.
    • Experiment: Tests two rival explanations against each other when you cannot tell without the test which is correct. This is rare.
    • Measurement: Refines a theory by determining a quantity (e.g., Cavendish measuring Newton’s gravitational constant G). No rival theory is involved.
  • Most things called “experiments” are actually measurements or demonstrations.
    • The Cavendish “experiment” was not an experiment in this sense—nobody doubted Newton’s theory; they just needed the constant.
    • If someone had proposed that G varies by location, then Cavendish’s work would have become an experiment.
  • Good explanations make risky, narrow predictions that you would not have anticipated beforehand (e.g., Einstein’s prediction of starlight bending around the Sun).
  • Simplicity is not fundamental—it is always defined relative to a theory of physics.
    • Solomonoff induction assumes a particular measure (Turing program length), which itself depends on assuming a specific physical substrate.
    • There is no a priori scale of complexity; it is always posterior to physics. Quantum computation already changed what counts as simple vs. complex.

Taking Theories Seriously

  • A recurring Deutsch move is showing that a theory refutes itself when taken seriously—not as a trick, but as what it means to actually engage with an idea.
    • The precautionary principle: civilization has never followed it, so adopting it now violates its own standard—it refutes itself.
    • Solipsism: if everything is my dream, I must explain how novel content (a person in a yellow suit I’ve never imagined) arises in that dream. Solipsism cannot account for its own content and thus destroys itself.
  • Physicists often shut down questions about quantum theory by saying “that’s the wrong question” or “you’ll get used to it.”
    • This is the opposite of taking the theory seriously. The student’s question—what does it mean for something to be both a particle and a wave?—is legitimate and exposing a lack of explanation.
  • Good explanations are accounts of reality, but successive theories do not simply discard their predecessors.
    • Newton and Einstein share heliocentric cosmology, the Sun as the source of gravitational influence, and many structural features.
    • Newton’s theory contained the problems Einstein solved: instantaneous action at rest, the stability of an infinite universe (Olbers’ Paradox), and the ad hoc nature of Newtonian cosmology.
    • Kepler’s ellipses were rejected by Galileo because circles were philosophically “perfect.” Newton explained why ellipses arise from the inverse-square law—a deeper explanation than aesthetic preference.

Against Kuhn’s Paradigm Incommensurability

  • Thomas Kuhn’s picture—that paradigms are incommensurable and scientists literally cannot conceive of rival frameworks—is largely fiction.
    • The young iconoclasts vs. old guard story does not describe actual scientific history.
    • People do irrationally cling to ideas, but there is no algorithm for who is right based on stubbornness or generation.
    • Real cases like Lister and Semmelweis involved resistance to changing working practices and perceived loss of dignity, not generational paradigm blindness.
  • Scientific progress is more like technology: analog computing → vacuum tubes → transistors.
    • Each step builds on the prior one; earlier stages are not “wrong” but are necessary stepping stones closer to the truth.
  • Social feedback loops in academia can pull people away from reality.
    • In physics this shouldn’t happen, but it does—philosophy and academic culture sometimes prioritize peer interaction over engagement with the real world.
    • Progress is possible even in philosophy and morality through the same method: taking ideas seriously and subjecting them to criticism.

No Limit to Knowledge—and No Shortcuts

  • There may be a physical limit to how far we can go (asteroid impact, unknown catastrophe), but invoking this as an argument about what we can know is logically equivalent to believing in the supernatural.
    • It makes a sophisticated prediction without an explanation—exactly like saying the world will end on a specific Tuesday because of a Bible interpretation.
    • There is no reason to believe explanatory universality will run out, and no way to know if it will.
  • Deutsch works at the foundations of science while rejecting foundationalism.
    • Foundations in his sense means theories that explain why higher-level theories are as they are (e.g., Newtonian mechanics explains why buildings stand up).
    • But you cannot derive engineering from fundamental physics alone—Newton’s Principia contains no suspension bridge. Engineering is a separate domain of knowledge.
    • Foundational theories unify understanding and enable criticism of designs, but they do not replace domain-specific knowledge.

The Enlightenment and British Political Culture

  • The Enlightenment took hold in England faster than elsewhere because it was a non-utopian rebellion against authority.
    • Instead of replacing old authority with a new final truth, it said: here is a problem, let’s fix it using existing structures extended incrementally.
    • Reforms extended privileges from the aristocracy to broader classes—“an Englishman’s home is his castle” was originally an aristocratic privilege gradually universalized.
  • France and Germany pursued utopian reforms: abolish the tyrant, write down final truths (human rights), make them hard to change.
    • Britain’s approach produced rapid change without sudden revolutions or extremism.
    • In the 1930s, totalitarian movements swept Europe but British fascism never won a single parliamentary seat and dissolved on its own.
  • British political culture assumes the system exists to solve problems through rival theories confronting each other.
    • You petition for redress of grievances, not to line people up against a wall.
    • No one is assumed to have the final answer.
  • The current rage against “misinformation” is deeply troubling from a Popperian perspective.
    • Every successful new truth is defined as misinformation by those who hold the prior belief.
    • Eliminating misinformation as a priority is impossible because knowledge cannot be known a priori—it must be creatively conjectured and discovered.
    • Popper’s actual view requires open debate, rival opposing theories, and systems for removing bad rulers and reversing bad decisions.

Knowledge in the Multiverse

  • Deutsch proposes a striking synthesis: knowledge is information that causes itself to be replicated in the environment.
    • A gene that improves survival gets copied; junk DNA does not. An invention that works gets imitated; a failed philosophy does not.
    • In the multiverse, random or non-useful information diverges across branches, while knowledge—useful, adaptive, explanatory—becomes common across branches.
    • This creates a “crystal” of knowledge across the multiverse: what is shared is what is true; what differs is what is not.
  • However, even if we could observe the multiverse, it would not give us a shortcut to truth.
    • There is no limit to the size of error we can make. A false idea can generate a large, growing “crystal” of false knowledge that overwhelms the truth for a long time across many branches.
    • You cannot examine a large crystal with a magnifying glass and determine whether it is heading toward truth or is a successful falsehood (like a persuasive religion).
    • There is no bound like “no one can make more than 256 errors in a row”—no such guarantee exists.
  • The nature of knowledge is inherently creative, conjectural, and contextual.
    • This guarantees an infinity of improvement ahead and keeps life interesting—there are no shortcuts.
Back to Naval