AI Monopolies in Arena Magazine

Contra Contrarianism

Yesterday Max Meyer joined the From the New World podcast to discuss Arena Magazine and storytelling for a techno-optimist future. As part of that project, I contributed a long essay on the coming market structure of AI — “AI Monopolies? Not So Fast.” 

In my view, there is a tendency to undervalue traditional economics in forecasting tech markets. A theme of this newsletter is that there is too much sci-fi speculation, not enough empirically grounded measures of machine learning research and industry.

Reading the entire piece is ideal if you want to learn more about “How LLMs are Made” (my original title pitch). I have two main highlights:

An important takeaway from the current AI ecosystem is that there is rapid diversification throughout the AI pipeline. What was once a single, vertically integrated process done by one company is separated into specialized improvements. As funding becomes more available and demand grows for niche applications of AI, the machine learning process is being sliced into ever narrower, more precise subprocesses. This is vertical disassembly. 

Firstly is the tendency of immediate vendors to appear in specialized processes. Historically, larger companies decay. They struggle to oversee and incentivize teams working on subprocesses which may individually comprise a small fraction of that company’s product, but accumulate into large inefficiencies. Startup vendors appear to specialize in that niche, creating a better product in that small niche and often selling their product back to those companies. 

To refine this point, I draw upon the microeconomic foundation of competition and specialization in a section aptly titled “A Return to Traditional Microeconomics”. 

Traditional microeconomics predicts that competition enables optimization through specialization. As Adam Smith puts it, “The wealth of nations is built on the division of labor.” This point is sharpened by Hayek’s knowledge problem:

“Today it is almost heresy to suggest that scientific knowledge is not the sum of all knowledge. But a little reflection will show that there is beyond question a body of very important but unorganized knowledge which cannot possibly be called scientific in the sense of knowledge of general rules: the knowledge of the particular circumstances of time and place. It is with respect to this that practically every individual has some advantage over all others because he possesses unique information of which beneficial use might be made, but of which use can be made only if the decisions depending on it are left to him or are made with his active cooperation.”

Thiel overturned this with his thesis that the incumbent network effects led to natural monopolies if executed upon correctly. “Brand, scale, network effects, and technology in some combination define a monopoly; but to get them to work, you need to choose your market carefully and expand deliberately.” The value of the accumulated participants in Facebook, Google, or Microsoft was inherently worth more than the fruits of a more specialized network.

Thiel’s analysis is based on the comparison of two variables: network effects and returns to specialization. Social media saw significant improvement in the strength of the social network. The second, often overlooked variable is the rate of improvement of smaller, specialized networks, or lack thereof. With social media, specialized innovation never reached the pace required to compete with network effects. The same is not true for AI.

Contrarian storytelling is overrated, at least when compared to microeconomics. That’s my takeaway. “Technology”, which itself is an oddly defined category, is now assumed to be the land of free monopolies, as if by the blessing of a magical San Francisco spirit. It’s poetic that the closest thing to a machine learning monopoly is actually in hardware, an industry that has typically not been considered a natural monopoly.

Of course, government policy can break up a monopolistic market or make an artificial monopoly out of a diverse market.

https://arenamag.com/2024/06/03/ai-monopolies-not-so-fast/