On the Reasonableness of Low Probabilities in Scaled Systems

There's an idea that I see intuitively accepted on all sides that bothers me: that tail events with a precedent imply a minimum 5 or even 1 percent chance of orders of magnitude greater, unprecedented tail event. I think this is the crux of the disagreement I have with the “low percent x-risk” community, who tend to be suspicious of every individual story about AI x-risk such as those presented by Eliezer Yudkowsky, but have a generalized, indefinite worry about x-risk. My counterargument was the subject of a few debate questions and my follow-up talk at Manifest.

While small scale systems tend to be lower bounded by some constant, for example 1%, many more large scale systems, even very complex ones, often have supercritical behaviors where probability approaches 0 or 1. As a trivial example, the more coins you flip, the probability that all land on heads approaches 0 exponentially. The probability that 20 coins land on heads is less than one in a million, and the probability that 40 coins land on heads is less than one in a trillion. 

In my experience in probability theory and combinatorics, the same is true not just for simple systems at scale. Even very complex systems at scale tend to be supercritical outside of very small ranges of parameters. As a less trivial example, even famous unpredictable hamiltonian paths have straightforward supercritical thresholds in large-scale random graphs. [https://www.math.cmu.edu/~af1p/Texfiles/HamSurvey.pdf]

Real-World Precedents

Of course, there's only so much generality you can draw from patterns in mathematical or computational models. It's not possible to make a definitive formal proof about real-world events. I don't draw upon them for a final description of the real world, but to provide context for rejecting the common intuition which I believe is misapplied to larger-scale systems. 

I would argue the closest thing to "common sense" about tail events comes from economic and scientific history. During the industrial and scientific revolutions, you tended to have a rapid period of concentrated innovation, expansion of methods, and later applications to industry, followed by a gradual decline in progress and ultimately stagnation. There's an unbroken pattern of scientific and industrial revolutions which have demonstrated far more impact than AI has so far reaching diminishing returns far before anything that would qualify as "superintelligence". 

If there's an oversimplified historical lesson from the booms and busts of economic history, it's that exponential scaling requires exponential resources in the long-run. 

Rebutting the Metaphysical Argument

There's a counterargument that EAs often reply with, including Neel Nanda publicly[https://x.com/NeelNanda5/status/1808249096484147599] is that large-scale systems are actually small-scale systems in disguise, because however you think about the question, regardless of the scale of a system, the act of thinking about it is basically a small-scale system. 

I reject this for two reasons.

1. Both historical patterns and mathematical models must be confirmed with evidence. The degree of evidence presents a degree of scale, and while not infallible, requires contradicting evidence or a definite methodological flaw to overturn, neither of which is ever provided.

2. This argument is much more proper for metaphysical questions, like the original Pascal's wager[https://en.wikipedia.org/wiki/Pascal%27s_wager]. I assign much greater probability to the metaphysical claims made using this argument, which by definition are much harder to collect evidence about. I find it extremely suspicious that believers in the metaphysical counterargument are concerned with problems of machine learning rather than theology (I would suggest that the solution to this puzzle is that they are in fact theologically motivated).

Neel comes close to admitting a kind of theological thinking by analogy. “I'm a pretty firm atheist, but definitely not at 10^-20 that I would change my mind after extensive deliberation.”

I’m not sure if a billion-dollar political campaign about the original Pascal’s Wager would be more or less authoritarian, but it might be more honest.

The Metaphysical Argument as a Media Form

What’s interesting about the metaphysical argument is that it’s a necessary property of the kind of narrative storytelling popular in the x-risk and broader blog ecosystem. The blogosphere ecosystem that Scott Alexander, Eliezer Yudkowsky, and other popular x-risk commentators are incentivized to tell stories which can be responded to in blog comments or exceeding common reply articles, which in turn spark replies to the replies, and so on. 

I believe that this media form was a major cause of the eventual domination of the x-risk portion of EA over the malaria net portion of EA, which remains empirical. I’m reminded of the rule in improv drama: “never say no”. Saying ‘no’ stops the conversation in improv drama, while raising the demand for evidence significantly limits the conversation in the blogosphere. So for Effecitve Altruism to thrive as a media form, it had to sacrifice the fact.