No recent issue has framed the difference between reality and hype more than AI. For those in the know we are in for a very long slog before AI can possibly act in a way autonomous enough to eliminate any but the most repetitive and uncreative jobs, and even in those the failures will be over very simple things and will dash many a company’s stock price and ability to raise further financing.
I often hear the classic example of the doorman from actual software engineers who understand the actual scaling issues involved - eliminating the doorman led to the lobby being destroyed and people sleeping in the building literally overnight, yet that’s one of the simplest, most “automatable” jobs in the entire economy!
AI in the form of LLMs cannot react to information outside its training set so by definition can only regurgitate prior solutions to existing problems. Yes it looks novel on its face but when you actually get to the limit of its training set it breaks down immediately. My company is inundated by multimillion dollar requests to license the totality of our private data to give them an advantage in their next model. This is a zero sum game. A fundamentally different type of model will need to be created to create any reasonably capable GAI, and that’s not where the money is being invested right now. It costs an order of magnitude more compute to train a model only incrementally more capable.
There is an absolute ceiling to the current technology but to your point, lay people only see a trajectory and assume it’s going to continue. They see geometric growth where it’s actually logarithmic. They see us at the bottom of the curve when it’s actually very far along.
As a technical question, is the Petering-out of scaling for text independent of potential gains from scaling when training on different or multiple kinds of input, likely to involve more computational intensity, for example, many good and varied kinds of electronic sensors observing humans doing complex work together and gleaming subtle patterns and so forth.
No recent issue has framed the difference between reality and hype more than AI. For those in the know we are in for a very long slog before AI can possibly act in a way autonomous enough to eliminate any but the most repetitive and uncreative jobs, and even in those the failures will be over very simple things and will dash many a company’s stock price and ability to raise further financing.
I often hear the classic example of the doorman from actual software engineers who understand the actual scaling issues involved - eliminating the doorman led to the lobby being destroyed and people sleeping in the building literally overnight, yet that’s one of the simplest, most “automatable” jobs in the entire economy!
AI in the form of LLMs cannot react to information outside its training set so by definition can only regurgitate prior solutions to existing problems. Yes it looks novel on its face but when you actually get to the limit of its training set it breaks down immediately. My company is inundated by multimillion dollar requests to license the totality of our private data to give them an advantage in their next model. This is a zero sum game. A fundamentally different type of model will need to be created to create any reasonably capable GAI, and that’s not where the money is being invested right now. It costs an order of magnitude more compute to train a model only incrementally more capable.
There is an absolute ceiling to the current technology but to your point, lay people only see a trajectory and assume it’s going to continue. They see geometric growth where it’s actually logarithmic. They see us at the bottom of the curve when it’s actually very far along.
As a technical question, is the Petering-out of scaling for text independent of potential gains from scaling when training on different or multiple kinds of input, likely to involve more computational intensity, for example, many good and varied kinds of electronic sensors observing humans doing complex work together and gleaming subtle patterns and so forth.
Never bet against the law of diminishing returns.
One of my favorite games