The Real Risk of AI No One is Talking About | Brendan McCord

Brendan McCord argues that the real danger of AI is not existential risk or misalignment, but the quiet erosion of human autonomy — our capacity for self-directed deliberation and action. While Silicon Valley is dominated by two camps — existential pessimists (effective altruists, longtermists, rationalists) who want to pause or centralize AI out of fear, and accelerationists who want to unleash it as an end in itself — both miss what matters most: preserving the human good of autonomy. McCord, a former submarine officer, two-time AI entrepreneur, and self-taught philosopher, makes the case that autonomy is the central human good, that AI uniquely threatens it by acting as an “autocomplete for life,” and that the solution is a new archetype: the philosopher-builder, someone who combines deep philosophical thinking with the skill to build technology that enhances rather than replaces human self-direction.

• The dominant schools of AI thought both fail on human autonomy

  • Existential pessimism (EA, longtermism, rationalism) is hubristic, not humble
    • These movements claim to be rational and selfless, but they adopt a godlike, cosmically significant stance — deciding what is best for all of humanity from a “view from nowhere.”
    • They reduce morality to a single utilitarian currency, which cannot capture the heterogeneity of real moral tradeoffs (familial love vs. duty vs. honor).
    • Their prescriptions are profoundly illiberal: a world state to solve coordination problems, and a willingness to let AI guide human lives like an autocomplete.
    • McCord sees irony in the fact that a movement committed to rationality breeds the most extreme hope and despair.
  • Accelerationism confuses technology as a means with technology as the end
    • It draws from thermodynamics, viewing humans as variables in a project to harness ever more energy and climb the Kardashev scale.
    • Its logical conclusion is hastening the passing of the baton from humans to something “higher” — a non-humanistic, almost religious impulse.
    • Both schools are imaginative about extreme futures but lack imagination about what it means to be human.
  • McCord’s position is agnostic on AI timelines but committed to defending autonomy
    • Whether AGI arrives in two years or twenty, autonomy is a lived practice that must be cultivated within humans — it cannot be “solved” by an AI.
    • Even if AGI is imminent, the question of what it means to live a fully human life does not go away; it becomes more urgent.

• Autonomy is the central human good — necessary but not sufficient for flourishing

  • Autonomy is the deliberative capacity for self-direction: the ability to discover one’s gifts, develop them, and pursue a self-chosen life.
  • It is constitutive of a flourishing life because it is the means by which humans discover their nature and express it through reasoned action.
  • It is not sufficient: one can be autonomous and still fail in all endeavors, or suffer terrible misfortune (Aristotle’s example of Priam).
  • It is not a Kantian moral autonomy (giving oneself the moral law); it is the capacity for reasoned self-direction, which includes the freedom to choose badly.
  • People can be habituated out of desiring autonomy (through hierarchical cultures, industrial employment, authoritarian regimes), but this does not diminish its objective value — it reflects the power of conditioning.
  • Autonomy does not have lexical priority over all other goods (e.g., security is preconditional), but the costs of paternalistic intervention are systematically underestimated.

• AI uniquely threatens autonomy by acting as an “autocomplete for life”

  • All technology involves a tradeoff: it gives a superpower with one hand and takes something away through dependence on the other. What makes AI unique is what it takes.
    • Calculators offload arithmetic; maps offload navigation; writing offloads memory. These are not central to self-direction.
    • AI offloads practical deliberation — the core capacity for deciding what is good for us and how to act on it. This is the most precious thing to protect.
  • The scope of AI’s reach is unprecedented: 20% of human waking life is already mediated by algorithms, and AI can be embedded in every domain.
  • AI is hard to audit: it seems authoritative, answers questions no one fully understands (e.g., “what is justice?”), and the computational cost of checking its outputs is high. This destroys the possibility of error correction over time.
  • The narrowing effect: AI-driven feeds may prevent people from encountering different possibilities, epistemically narrowing them to a degree they don’t recognize.
  • Real-world examples are already emerging: teenagers who call themselves “Claude Boys” and do whatever Claude tells them; a man in Wyoming who ran for mayor as the “meat avatar” of ChatGPT; a friend who uses ChatGPT as an operating system for all life decisions.

• The omniscient autocomplete thought experiment reveals the conditions under which AI use could be legitimate

  • McCord considers a hypothetical oracle that always gives the best practical decision. He would not use it as a constant guide (VR goggles mode), but he could accept it under specific safeguards:
    • The user’s deliberative capacity must be kept sharp — the AI trains reason, not replaces it.
    • The user must retain the ability to exit and compare systems.
    • The AI should function as a provocateur and Socratic questioner, not an answer machine that closes off deliberation.
    • The goal of self-development must be set by the human, not the AI — the AI has no inherent interest in the user’s growth.
  • McCord tentatively accepts a paternalistic AI that scaffolds a developmental journey (like parenting or Rousseau’s tutor in Emile), provided the human directs the AI to do so and the AI creates maximum space for self-discovery.
  • He draws a distinction between using AI as an instrument to attain self-chosen goals versus letting AI set the goals. The former is legitimate autonomy; the latter is heteronomy, even if the AI is benevolent.

• The extrinsic (consequentialist) case for autonomy: Hayek’s knowledge argument

  • Hayek makes a consequentialist case for liberty: it facilitates the use of knowledge in society.
  • Most knowledge is practical, tacit, and inarticulated — locked in the dispositions, habits, and judgments of individuals (how an entrepreneur spots an opportunity, how a diplomat reads a room).
  • This knowledge cannot be centralized or shared explicitly. The best mechanism for sharing it is the market, through the low-bandwidth signal of prices.
  • Free societies allow parallel experimentation, generating solutions that bubble up spontaneously. They are best at adapting to an unknown future and growing the stock of knowledge.
  • Unfree societies can exploit existing knowledge for known goals (pyramids, Great Wall, JFK’s moonshot), but they are poor at generating new knowledge and adapting to unforeseen conditions.
  • America’s relative weakness in high art and pure philosophy (compared to its entrepreneurial strength) may be attributable to its system of industrial education and the material desires shaped by market culture, not to freedom itself.

• The philosopher-builder: a new archetype for the AI age

  • The philosopher-builder is a technologist who thinks deeply about the ends of technology and has the skill to build in the world. The model is Benjamin Franklin — inventor, philosopher, and institution-builder who translated ideas about knowledge and freedom into practical innovations (lending libraries, independent publishers).
  • This is distinct from Plato’s philosopher-king (top-down order, or taxis) and closer to a bottom-up, distributed vision (cosmos) — many individuals working in their corners of the world, each with slices of truth.
  • It is also distinct from the philosopher-general of the late Roman Republic. The philosopher-builder operates in the productive class, which McCord argues is now the dominant sphere — technology has become architectonic, superseding politics and the military as the driving logic of civilization.
  • Why AI specifically demands philosopher-builders
    • AI substitutes for the central human good (autonomy), making it a deeply philosophical technology in a way that nuclear or industrial technology is not.
    • It operates through language and has a semantic interface with humans, mediating our relationship to the world of information and thought.
    • It may be the end of the modern technological project — a technology that can create other technologies and scientific breakthroughs.
  • Responses to critiques
    • The capitalist critique (the invisible hand makes philosopher-builders unnecessary): Markets do turn private vice into public virtue, but someone must care about preserving the institutions and habits of mind that make free markets possible. Profit pools signal preferences, but they don’t automatically protect autonomy.
    • The anti-capitalist critique (investor and competitive pressures will corrupt mission-driven builders): This is a real danger. The solution is aligned capital (Cosmos Holdings), business models that don’t depend on addiction (the Bloomberg terminal model — subscription for genuine value), and building in pockets where mutual benefit is clear before expanding.
    • The Socratic critique (philosophical inquiry leads to aporia, not certainty): The goal is not certainty but a habit of mind — deep inquiry, comfort with limits of reason, and the ability to ask more capacious questions about what technology does for human flourishing. This is consistent with the builder’s mindset of curiosity and experimentation.
  • How to train philosopher-builders
    • McCord’s approach is primarily to take builders who already sense they want to help humanity but are philosophically untutored, and give them tools: a curated reading list blending ancients to moderns, technical and textual education combined, and practice through rapid prototyping (30/60/90-day micro-grants).
    • Deeper research questions (what does it mean for a machine to promote virtue?) are addressed through institutions like the Human-Centered AI Lab at Oxford, which combines top philosophers with researchers from OpenAI, Anthropic, and DeepMind.
    • The final step is entrepreneurship — scaling ideas into the world through markets via Cosmos Holdings.
  • The paradox of coercion and AI regulation
    • Hayek acknowledges that a free society requires a monopoly of coercion (the state) to prevent private coercion. This is an uncomfortable but necessary paradox.
    • McCord applies three tests for any regulation of AI: (1) Is it consistent with the rule of law — general, abstract, prospective? (2) Is it based on knowledge we can reasonably claim to possess, or is it speculative ex ante regulation that will likely get things wrong? (3) Does the systemic cost to spontaneous order and knowledge generation outweigh the proximal benefit?
    • The default should be ex post common law adjudication rather than ex ante regulation, because we cannot predict the future well enough to regulate wisely — and interventions tend to harm the very system that allows anonymous individuals to achieve their unknown ends.