In this essay, Venkatesh Rao examines the evolution of the “EA formation”—an umbrella encompassing Effective Altruism, rationalism, longtermism, and existential-risk research—from an idiosyncratic blogging subculture into a formidable institutional force shaping modern artificial intelligence. Rao argues that while the movement aims to solve the critical challenge of AI safety, its rapid institutional ascent, abstract moral reasoning, and totalizing certainty have generated a second-order crisis: “EA safety,” or the urgent need to keep governance of transformative technologies safe from any single group’s moral monopoly.
The Nature and Epistemic Fragility of the EA Formation
The EA formation originated from three intertwined strands: evidence-based philanthropy (GiveWell), consequentialist life-planning (Singer, MacAskill), and existential-risk futurism (LessWrong, Yudkowsky, Bostrom). Over time, the astronomical stakes introduced by longtermism and existential risk captured the movement’s intellectual commanding heights, transforming AI safety into its dominant priority.
Despite projecting epistemic modesty through Bayesian credences and statistical forecasts, the formation suffers from deep conceptual fragility. It excels at displaying calculated uncertainty within its analytical models, yet remains remarkably blind to doubts about the validity of the models themselves. By translating complex, irreducible human goods into abstract utility metrics, the framework readily dismisses illegible intuitions, non-quantifiable evidence, and competing moral philosophies.
A Secular Theology and Its Extremizing Tails
Rao observes that underneath its aggressively secular, mathematical exterior, the EA formation behaves structurally like a religion. AI serves as an eschatology featuring apocalyptic extinction or transcendent salvation, while cause prioritization acts as a cosmic vocation. This dynamic can foster an elite “vision of the anointed,” granting adherents an intercessory authority to dictate historical trajectories.
Because the formation relentlessly converts abstract optimization directly into consequential action with minimal empirical feedback, it repeatedly produces extreme outliers. These range from radical philosophical experiments to concerning social deviations, high-control dynamics, and high-profile institutional collapses (e.g., FTX). Rather than dismissing these incidents as disconnected aberrations, Rao views them as stress-test failures inherent to an optimizer that strips away ordinary moral constraints in pursuit of astronomical future value.
Institutional Power and the Broader Threat of Moral Monopoly
While the EA formation did not invent modern foundational AI architecture—nor does it control global development, as evidenced by independent technical progress in mainstream labs and the Chinese AI ecosystem—it has established formidable bridges into Western frontier labs, funding networks, and regulatory bodies. It has effectively set the vocabulary for how AI risks are conceptualized and governed.
The dilemma extends beyond Effective Altruism: rival totalizing philosophies, such as effective accelerationism (e/acc), progressive regulatory doctrines, and nationalist state agendas, similarly vie for sovereign control over AI’s moral framing.
Conclusion: Containing Doctrines Through Pluralism
Drawing an analogy to the institutional settlements that followed Europe’s wars of religion, Rao concludes that the solution is neither to eradicate the EA movement nor to replace it with a rival orthodoxy. The EA formation offers valuable utility as an “immune response” that isolates overlooked tail risks, but it cannot be permitted to dictate the entire agenda. Policymakers, frontier labs, and academic institutions must deliberately cultivate a resilient, pluralistic governance ecosystem where diverse traditions retain enough institutional power to challenge one another’s blind spots.
Mentoring question
When making high-stakes decisions within your own organization or field, how do you distinguish between genuine epistemic doubt about your foundational models versus merely calculating probabilities within an insulated framework?