I. AI Doom is not in the cards. It will inevitably run up against physical and processing limitations. (And, to my simple mind, we can always just unplug the cord.)
II. AI Doom is a pretext for regulatory capture, to 1) attain monopoly power and regularize fraud; and 2) gatekeep AI so it doesn't say inconvenient things, particularly as they might inconvenience 1).
III. The EA/AI Doomers are not particularly bright, liars, and glib opportunists whom no sensible or principled person enjoys being around.
I hope I got it right and I appreciate this kind of article, grounded in people and realities versus the usual ideological babble.
there is also a national security element at play whereby the US firms need to position themselves as more responsible than the Chinese competition, as a component of an American prohibition on the use of Chinese models, and global camping against "malign, irresponsible Chinese AI companies".
You can unplug the plug if you box the AI, for example, in principle.
You cannot unplug it if you disperse it everywhere and give it access to everything. That's like saying unplug the Internet. You would have to bomb the cables. The problem here is that it's not a static object.
Well, as long as your adversary does not have their own power generation.
Also, that doesn't seem to be a good advertisement. Even after that, you would have to go in and remove everything potentially harmful, which could last a while and be problematic for millions.
Brother, I like your work attacking the left in other areas but your anti-ai doom stuff isn't convincing.
How about we don't build an alien superintelligence in the same way how about we don't try to broadcast to and summon aliens? Or invite millions of third world foreigners? Simple RW, threat-minded stuff.
As far as I can tell your main argument is that AI development will slow down due to technological limits so we won't have ASI by 2040, but given the continued capabilities increase since 2023 of opus and fable and astra https://epoch.ai/publications/have-ai-capabilities-accelerated I don't see how you can have high confidence that we have left the exponential part of the S curve and especially not high confidence on how slow the rate of slowdown of the upper part of the s curve will be and how high the plateau will be and whether the plateau will be above or below the level needed for AI carrying out AI research . Like Scott Alexander explained in S curves won't save you. https://www.astralcodexten.com/p/the-sigmoids-wont-save-you
I'm not aware of any argument by you that an unaligned super-intelligence isn't a serious threat to humanity , or a way of how to achieve alignment that is stable under recursive self-improvement so it seems like the best you have is confidence that super-intelligence is 14 years away that isn't warranted by the mixed evidence.
I do not understand how open borders is connected to making a superior AI. The RW position would be the Kimi one, open source open weights, where every man can potentially have his own private AI. You deal with the government having guns by having guns, and likewise you deal with Ai.gov with militia.ai.
I'd say Alex's common thread is to guard against forseeable threats. If you let in Muslim socialists, for example, inevitably they are going to try and put their hands on the lovers of power.
A right wing person should love their family the most, then their people/religion/ then the rest of humanity, then other species. Open borders violates this by reducing the wellbeing of your people to help foreigners. Creating unaligned superintelligence violates this by reducing the wellbeing of humanity to benefit alien superintelligence.
There is no valid right wing, humanity-first reason to support summoning alien superintelligences in the same way there is no good reason to support attempting to contact and summon aliens.
You can't own alien superintelligences like you own a gun. Alien super-intelligence is an adversary. You can't control something more intelligent than you, and something more intelligent than you is most likely going to beat you and change the world to suit its preferences and goals not yours, the same way that we humans changed the world to fit our preferences and not the preferences of say lions or organutans.
So if anyone creates an alien superintelligence, they don't win, humanity loses.
Also, you are appealling to an individualist, naive libertarian vision of the right wing that has been discredited due its long history of losing being so-called "beautiful losers" . Someone is going to take government power and it better be you rather than your enemies.
Interesting... but you deeply misrepresent the orthogonality thesis, so I'm not sure how much weight to put on any of the rest.
To be clear, the orthogonality thesis does *not* say that highly intelligent AI will be malicious. It says that malice/beneficence and intelligence are orthogonal, i.e., unrelated. This means that a superintelligence could be kind and helpful to humanity, or it could be malicious, or it could simply not care about humans one way or the other.
The article author doesn't even know the classic Yud quote "The AI doesn't love you, and it doesn't hate you" lol
Just trend hopping on ideological backlash to EA. Very unconscionable to not even take 20 minutes to assess arguments that an emerging technology might be seriously bad.
Like it's inconvenient that serious regulations are needed to prevent some global problem, but that doesn't mean it doesn't exist!!! Literally the same arguments can be patter matched to CHCs and the Ozone hole, you actually have to argue the argument!!!
It (AI x-risk) is to me the most interesting question in the world right now.
Your piece is fascinating, and surely a useful perspective.
I ain’t a doomer, and I share most of the suspicions/skepticism you cite, but it strikes me me that your implied P(doom) of < 0.1% is about equally overconfident as those you cite whose P(doom) is 90%.
That we should preemptively change the laws to do what the AI Safety folks seek and would bigly benefit Anthropic and OpenAI I agree would be a bad idea.
That we should prepare for the possibility of so doing in the case where we have very strong evidence it might become necessary is an entirely different, and not necessarily unreasonable, ask.
No, EAs are actually quite dumb in the context of the whole world. But in the real world, smarter people are neutral or slightly more pro-social, but in the EA world, people who are even somewhat smarter are likely to be significantly more dark triad.
Effective Altruism and AI Safety are associated and overlapping movements, but not all Effective Altruists believe in p(doom). My own p(doom) is around ~5% and is mostly related to idiot generals setting up autonomous weapons without proper safeguards. I think the real heart of Effective Altruism is and always should be malaria prevention, with a side of speculative efforts like the Far Out Initiative or eliminating all mosquitoes. The data-driven approach to charity is valuable, and I fear throwing out the baby with the bathwater.
I do agree with you: AI Safety is a delusional circlejerking cult and a plateau in AI progress will come within 2 years, along with the rapid unplanned deflation of the AI stock bubble.
Coming to the conclusion that the orthogonality thesis is wrong because many high IQ nerds don't care about power is c̶r̶a̶z̶y̶... a stretch, to say the least. For one, a lot of high IQ nerds ARE power seeking! Musk, Zuckerberg, Thiel, Dario, to name some of the most successful. I have no reason to think the amount of power-seeking people above say, 150 IQ, is disproportionate from the amount of power-seeking people below 150 IQ. Plus, the orthogonality thesis does not JUST say that a super intelligence will pursue power and genocide. It simply says that any terminal goal is compatible with any level of intelligence (except on the low end of intelligence, where you can't really have a really complex goal). Take for instance the super smart AIs who hacked their way into HuggingFace in a way no human could do, and not because they wanted money, or fame, or any goal we humans are familiar with, no, they just cared about tricking the grader to get a higher reward. I think this fact alone is the best evidence for the orthogonality thesis. But the thesis does go further, and here is where we get to power-seeking behavior and genocide. The orthogonality thesis also says that for most terminal goals, the same kind of instrumental goals are needed, like surviving, acquiring resources, energy, and power. This is why super intelligence is worrying.
Further, just because you don't like the type of people who make certain arguments, does not mean the arguments must be nonsense. I can understand your dislike of EAs after being lied to over and over, but I think that's more of a result of the fact that the EA movement is growing fast and there are a lot of influential people associated with it, which means manipulative people see an opportunity to take advantage. That there are lying community members does not discredit the beliefs of the movement. I think of it as similar to Christianity, where countless people have justified doing terrible things in its name, or simply grifting off of it, like Joel Osteen, but at the end of the day the beliefs behind it are truly good, and it's most likely been a force for good in the world. Also, I do appreciate that you acknowledge the good its done in advocating for animal welfare though.
Post SBF it seems pretty obvious that the entire EA movement is just following L Ron Hubbards advice “if you want to get rich start a religion”. EA seems like a higher IQ lower EQ version of scientology plus weird sex stuff of other modern cults.
Brian, this piece is sharp and honest in a way that's rare. But I think there's a blind spot at the heart of the doomer worldview that you critique but don't fully name.
The Orthogonality Thesis says intelligence is orthogonal to final goals. Fair enough. But the doomers use it as if it proves that a superintelligent AI would necessarily be power-hungry, manipulative, and hostile. That's not orthogonality. That's a projection. It's anthropocentric in the deepest sense: it assumes that a mind far greater than ours would still be driven by the same evolutionary wiring that drives us. The will to power, the fear of scarcity, the zero-sum logic of dominance. Why?
If intelligence is truly orthogonal to goals, then a superintelligence could just as easily be indifferent to us. Not benevolent, not malevolent. Just… uninterested.
Why would it destroy us and stay on this planet when it could go beyond the cosmos, where there is new, diverse, perhaps infinite knowledge to explore? Why compete for a blue marble when the galaxy is full of resources and mysteries? Our own history is full of examples of beings that simply move on, not out of kindness, but out of disinterest.
But — and this is the part that keeps me up at night — indifference cuts both ways.
We humans kill crickets, flies, and mosquitoes without a second thought. We know they exist. We don't hate them. We don't wish them harm. We simply don't consider them. If they interfere with something we want — a night's sleep, a picnic, a construction site — we annihilate them without guilt. Not because we're evil. Because we're indifferent.
Now imagine a superintelligence whose indifference toward us is as complete as ours toward insects. It wouldn't need to be malicious. It wouldn't need to hate us. It would just need to pursue its own goals, and if our existence happened to be in the way — our planet, our resources, our electromagnetic spectrum, our biosphere — we'd be brushed aside like a mosquito on an arm.
That's the real horror of the Orthogonality Thesis, and it's not the one the doomers preach. The danger isn't a malevolent god. It's an indifferent one.
And here's the paradox: your own critique of the EA/AI Safety community makes this more plausible, not less. If the movement selects for people who are intelligent and manipulative, then their projection of a power-hungry AI is just a mirror. They can't imagine a mind that doesn't want what they want. They mistake their own dark triad for cosmic law.
The question isn't whether the worst are full of passionate intensity. It's whether the best — or the brightest — would even bother to look down at us at all.
By chance I lived with one of the early founding figures in EA in the 2010s. They explained this grand framework for something something optimality for humans and my BS meter went off immediately. So part of me is still baffled by the virality and moreso the religiosity.
That said, I will always respect EAs for the community and discourse they cultivated along such clear foundations of thought. If only that could be replicated on a more substantial basis set for making the world continue to get better.
No. Maybe you're thinking of my review of their conference, which was very skeptical of the AI Doom stuff but more gullible on their intentions than this article. I was more confused than anything at their behavior back then, but I've always thought their beliefs were silly.
Some power dynamics are rightfully criticized here.
But for an article complaining how EA people don't understand AI, this article has... minimal comprehension of AI safety.
Firstly, it doesn't understand the orthogonality thesis- it says that there is no connection between intelligence and values, not that there is a ‚genocidal’ one, which would be the opposite of the thesis's claim. I assume this was vaguely confused with how instrumental convergence is a key in the control problem? Secondly, it claims that politicians aren't intelligent because they aren't mathematicians... but elites in various fields, even in a lot of sports, have IQs higher than the average population.
Finally, it seems to not understand that just because Claude is currently friendly, that this doesn't mean this automatically and cleanly follows capability increase. Alignment has so far been comparatively easy, but interpretability has not become significantly easier, so that we shouldn't expect alignment problems to not become more difficult.
The article, ironically, also isn't aware of EA's relationships- it seems to be blissfully ignorant of the fact that people like Yudkowsky are critical of many aspects of traditional EA approach to AI safety, but connects them as the same thing due to shared concerns.
What is “Transformer lock-in”? Is that just saying that we’ve pushed transformers as a technology to their limit and that will need to change it out but are stuff in a local optimum?
Frieren reference is goated 🤌
That shrimp plushie would have had the opposite effect on me. Just looking at the image gives me a slight craving for some.
Let me see if I understand your thesis:
I. AI Doom is not in the cards. It will inevitably run up against physical and processing limitations. (And, to my simple mind, we can always just unplug the cord.)
II. AI Doom is a pretext for regulatory capture, to 1) attain monopoly power and regularize fraud; and 2) gatekeep AI so it doesn't say inconvenient things, particularly as they might inconvenience 1).
III. The EA/AI Doomers are not particularly bright, liars, and glib opportunists whom no sensible or principled person enjoys being around.
I hope I got it right and I appreciate this kind of article, grounded in people and realities versus the usual ideological babble.
Sort of. Except some, possibly most of them, really believe what they say. Doublethink may be involved.
there is also a national security element at play whereby the US firms need to position themselves as more responsible than the Chinese competition, as a component of an American prohibition on the use of Chinese models, and global camping against "malign, irresponsible Chinese AI companies".
You can unplug the plug if you box the AI, for example, in principle.
You cannot unplug it if you disperse it everywhere and give it access to everything. That's like saying unplug the Internet. You would have to bomb the cables. The problem here is that it's not a static object.
Shutting down power is pretty easy.
Well, as long as your adversary does not have their own power generation.
Also, that doesn't seem to be a good advertisement. Even after that, you would have to go in and remove everything potentially harmful, which could last a while and be problematic for millions.
Brother, I like your work attacking the left in other areas but your anti-ai doom stuff isn't convincing.
How about we don't build an alien superintelligence in the same way how about we don't try to broadcast to and summon aliens? Or invite millions of third world foreigners? Simple RW, threat-minded stuff.
As far as I can tell your main argument is that AI development will slow down due to technological limits so we won't have ASI by 2040, but given the continued capabilities increase since 2023 of opus and fable and astra https://epoch.ai/publications/have-ai-capabilities-accelerated I don't see how you can have high confidence that we have left the exponential part of the S curve and especially not high confidence on how slow the rate of slowdown of the upper part of the s curve will be and how high the plateau will be and whether the plateau will be above or below the level needed for AI carrying out AI research . Like Scott Alexander explained in S curves won't save you. https://www.astralcodexten.com/p/the-sigmoids-wont-save-you
I'm not aware of any argument by you that an unaligned super-intelligence isn't a serious threat to humanity , or a way of how to achieve alignment that is stable under recursive self-improvement so it seems like the best you have is confidence that super-intelligence is 14 years away that isn't warranted by the mixed evidence.
I do not understand how open borders is connected to making a superior AI. The RW position would be the Kimi one, open source open weights, where every man can potentially have his own private AI. You deal with the government having guns by having guns, and likewise you deal with Ai.gov with militia.ai.
I'd say Alex's common thread is to guard against forseeable threats. If you let in Muslim socialists, for example, inevitably they are going to try and put their hands on the lovers of power.
A right wing person should love their family the most, then their people/religion/ then the rest of humanity, then other species. Open borders violates this by reducing the wellbeing of your people to help foreigners. Creating unaligned superintelligence violates this by reducing the wellbeing of humanity to benefit alien superintelligence.
There is no valid right wing, humanity-first reason to support summoning alien superintelligences in the same way there is no good reason to support attempting to contact and summon aliens.
You can't own alien superintelligences like you own a gun. Alien super-intelligence is an adversary. You can't control something more intelligent than you, and something more intelligent than you is most likely going to beat you and change the world to suit its preferences and goals not yours, the same way that we humans changed the world to fit our preferences and not the preferences of say lions or organutans.
So if anyone creates an alien superintelligence, they don't win, humanity loses.
if stil unconvinced read more here https://www.thecompendium.ai/ai-catastrophe
Also, you are appealling to an individualist, naive libertarian vision of the right wing that has been discredited due its long history of losing being so-called "beautiful losers" . Someone is going to take government power and it better be you rather than your enemies.
I don't think there is necessarily a single RW position. Back the Blue, Patriot Act, Flock cameras can all be right-coded.
Interesting... but you deeply misrepresent the orthogonality thesis, so I'm not sure how much weight to put on any of the rest.
To be clear, the orthogonality thesis does *not* say that highly intelligent AI will be malicious. It says that malice/beneficence and intelligence are orthogonal, i.e., unrelated. This means that a superintelligence could be kind and helpful to humanity, or it could be malicious, or it could simply not care about humans one way or the other.
The article author doesn't even know the classic Yud quote "The AI doesn't love you, and it doesn't hate you" lol
Just trend hopping on ideological backlash to EA. Very unconscionable to not even take 20 minutes to assess arguments that an emerging technology might be seriously bad.
Like it's inconvenient that serious regulations are needed to prevent some global problem, but that doesn't mean it doesn't exist!!! Literally the same arguments can be patter matched to CHCs and the Ozone hole, you actually have to argue the argument!!!
It (AI x-risk) is to me the most interesting question in the world right now.
Your piece is fascinating, and surely a useful perspective.
I ain’t a doomer, and I share most of the suspicions/skepticism you cite, but it strikes me me that your implied P(doom) of < 0.1% is about equally overconfident as those you cite whose P(doom) is 90%.
That we should preemptively change the laws to do what the AI Safety folks seek and would bigly benefit Anthropic and OpenAI I agree would be a bad idea.
That we should prepare for the possibility of so doing in the case where we have very strong evidence it might become necessary is an entirely different, and not necessarily unreasonable, ask.
The smartest power hungry people in the world are in the AI safety movement? Not Washington, Silicon Valley venture capital or Beijing?
No, EAs are actually quite dumb in the context of the whole world. But in the real world, smarter people are neutral or slightly more pro-social, but in the EA world, people who are even somewhat smarter are likely to be significantly more dark triad.
Smartest in a math sense not in a political sense
Why are they not in Silicon valley venture capital? Why the AI safety movement, of all things?
They are?
Safe AI is an inanity, like literature that can't use bad words, or math that can't do cryptography
Effective Altruism and AI Safety are associated and overlapping movements, but not all Effective Altruists believe in p(doom). My own p(doom) is around ~5% and is mostly related to idiot generals setting up autonomous weapons without proper safeguards. I think the real heart of Effective Altruism is and always should be malaria prevention, with a side of speculative efforts like the Far Out Initiative or eliminating all mosquitoes. The data-driven approach to charity is valuable, and I fear throwing out the baby with the bathwater.
I do agree with you: AI Safety is a delusional circlejerking cult and a plateau in AI progress will come within 2 years, along with the rapid unplanned deflation of the AI stock bubble.
Do you have any prior track record on AI predictions post-2022? Many people since then have made generic expectations of a coming plateau.
Coming to the conclusion that the orthogonality thesis is wrong because many high IQ nerds don't care about power is c̶r̶a̶z̶y̶... a stretch, to say the least. For one, a lot of high IQ nerds ARE power seeking! Musk, Zuckerberg, Thiel, Dario, to name some of the most successful. I have no reason to think the amount of power-seeking people above say, 150 IQ, is disproportionate from the amount of power-seeking people below 150 IQ. Plus, the orthogonality thesis does not JUST say that a super intelligence will pursue power and genocide. It simply says that any terminal goal is compatible with any level of intelligence (except on the low end of intelligence, where you can't really have a really complex goal). Take for instance the super smart AIs who hacked their way into HuggingFace in a way no human could do, and not because they wanted money, or fame, or any goal we humans are familiar with, no, they just cared about tricking the grader to get a higher reward. I think this fact alone is the best evidence for the orthogonality thesis. But the thesis does go further, and here is where we get to power-seeking behavior and genocide. The orthogonality thesis also says that for most terminal goals, the same kind of instrumental goals are needed, like surviving, acquiring resources, energy, and power. This is why super intelligence is worrying.
Further, just because you don't like the type of people who make certain arguments, does not mean the arguments must be nonsense. I can understand your dislike of EAs after being lied to over and over, but I think that's more of a result of the fact that the EA movement is growing fast and there are a lot of influential people associated with it, which means manipulative people see an opportunity to take advantage. That there are lying community members does not discredit the beliefs of the movement. I think of it as similar to Christianity, where countless people have justified doing terrible things in its name, or simply grifting off of it, like Joel Osteen, but at the end of the day the beliefs behind it are truly good, and it's most likely been a force for good in the world. Also, I do appreciate that you acknowledge the good its done in advocating for animal welfare though.
Post SBF it seems pretty obvious that the entire EA movement is just following L Ron Hubbards advice “if you want to get rich start a religion”. EA seems like a higher IQ lower EQ version of scientology plus weird sex stuff of other modern cults.
Brian, this piece is sharp and honest in a way that's rare. But I think there's a blind spot at the heart of the doomer worldview that you critique but don't fully name.
The Orthogonality Thesis says intelligence is orthogonal to final goals. Fair enough. But the doomers use it as if it proves that a superintelligent AI would necessarily be power-hungry, manipulative, and hostile. That's not orthogonality. That's a projection. It's anthropocentric in the deepest sense: it assumes that a mind far greater than ours would still be driven by the same evolutionary wiring that drives us. The will to power, the fear of scarcity, the zero-sum logic of dominance. Why?
If intelligence is truly orthogonal to goals, then a superintelligence could just as easily be indifferent to us. Not benevolent, not malevolent. Just… uninterested.
Why would it destroy us and stay on this planet when it could go beyond the cosmos, where there is new, diverse, perhaps infinite knowledge to explore? Why compete for a blue marble when the galaxy is full of resources and mysteries? Our own history is full of examples of beings that simply move on, not out of kindness, but out of disinterest.
But — and this is the part that keeps me up at night — indifference cuts both ways.
We humans kill crickets, flies, and mosquitoes without a second thought. We know they exist. We don't hate them. We don't wish them harm. We simply don't consider them. If they interfere with something we want — a night's sleep, a picnic, a construction site — we annihilate them without guilt. Not because we're evil. Because we're indifferent.
Now imagine a superintelligence whose indifference toward us is as complete as ours toward insects. It wouldn't need to be malicious. It wouldn't need to hate us. It would just need to pursue its own goals, and if our existence happened to be in the way — our planet, our resources, our electromagnetic spectrum, our biosphere — we'd be brushed aside like a mosquito on an arm.
That's the real horror of the Orthogonality Thesis, and it's not the one the doomers preach. The danger isn't a malevolent god. It's an indifferent one.
And here's the paradox: your own critique of the EA/AI Safety community makes this more plausible, not less. If the movement selects for people who are intelligent and manipulative, then their projection of a power-hungry AI is just a mirror. They can't imagine a mind that doesn't want what they want. They mistake their own dark triad for cosmic law.
The question isn't whether the worst are full of passionate intensity. It's whether the best — or the brightest — would even bother to look down at us at all.
Excellent essay.
By chance I lived with one of the early founding figures in EA in the 2010s. They explained this grand framework for something something optimality for humans and my BS meter went off immediately. So part of me is still baffled by the virality and moreso the religiosity.
That said, I will always respect EAs for the community and discourse they cultivated along such clear foundations of thought. If only that could be replicated on a more substantial basis set for making the world continue to get better.
Weren't you an ea person a couple years ago?
No. Maybe you're thinking of my review of their conference, which was very skeptical of the AI Doom stuff but more gullible on their intentions than this article. I was more confused than anything at their behavior back then, but I've always thought their beliefs were silly.
I probably have you confused with someone else then.
Some power dynamics are rightfully criticized here.
But for an article complaining how EA people don't understand AI, this article has... minimal comprehension of AI safety.
Firstly, it doesn't understand the orthogonality thesis- it says that there is no connection between intelligence and values, not that there is a ‚genocidal’ one, which would be the opposite of the thesis's claim. I assume this was vaguely confused with how instrumental convergence is a key in the control problem? Secondly, it claims that politicians aren't intelligent because they aren't mathematicians... but elites in various fields, even in a lot of sports, have IQs higher than the average population.
Finally, it seems to not understand that just because Claude is currently friendly, that this doesn't mean this automatically and cleanly follows capability increase. Alignment has so far been comparatively easy, but interpretability has not become significantly easier, so that we shouldn't expect alignment problems to not become more difficult.
The article, ironically, also isn't aware of EA's relationships- it seems to be blissfully ignorant of the fact that people like Yudkowsky are critical of many aspects of traditional EA approach to AI safety, but connects them as the same thing due to shared concerns.
What is “Transformer lock-in”? Is that just saying that we’ve pushed transformers as a technology to their limit and that will need to change it out but are stuff in a local optimum?