The Age of Feature Engineering is Here
Specialization in Technology is Inevitable
Feature engineering is a technique for making AI that specializes at a specific task. In some circles, it’s become a general insult, used to villainize those making specialized AI. But that perception couldn’t be further from the truth.
Feature engineering is good for the field of machine learning, good for the economy, and good for society.
Imagine you could only have one electric appliance. Any device that uses electricity had to be a washing machine, microwave, induction stove, and vacuum cleaner. This would be ridiculous, impractical, and absurdly costly. But why?
We can work backwards from reality. It’s practical to have different appliances for different things. We expect washing machines to wash, microwaves to microwave, and vacuum cleaners to vacuum. This works well in our everyday life. It works well because designing one electric appliance is easier than designing a huge omni-appliance. The engineers can do a better job. Engineers can specialize, learning more about one appliance because they have little need for knowledge about the others. In the omni-appliance, doing each task well creates additional constraints on the other tasks. The burden exceeds the costs saved by sharing resources.
It also works well because we understand how to use our appliances. We know which button to press for vacuuming, which button for microwaving, and which button for washing. The complexity of using an appliance is not entirely different than the complexity of designing it. For each individual appliance, each limitation to the design space is also a limitation to the use space. This network of constraints add complexity, inefficiency, and ambiguity.
Division enables simplification, for both design and use.
Despite the possibility of saving money by combining appliances into an omni-appliance that shares the same batteries, wires, cases, and screens, it is just not worth it.
All of this is to say that a design paradigm that dogmatically refuses to sacrifice general capability for specific application is both leaving free money on the table and leaving the public worse off. Of course, this is about AI.
Feature engineering originated as a technical term for designing an AI model to follow inputs designed for one pattern, or ‘feature’. But as AGI hype accelerated, it became an insult. The implication was that people making AI models that worked best for a narrow task were unserious, or even bad for the field. This is not only wrong, but the exact opposite of the truth. Feature engineering is the heart and soul of commercial success! It is synonymous with specialization. As Adam Smith writes in the Wealth of Nations:
It is the great multiplication of the productions of all the different arts, in consequence of the division of labour, which occasions, in a well-governed society, that universal opulence which extends itself to the lowest ranks of the people.
Moreover, techniques discovered for specific use cases often contribute to broader developments. This arguably includes many of the early predecessors to modern transformer models like GPT-4. For example, RNNs, a direct predecessor of the transformer, only had narrow use cases at the time of its invention and development.
As I wrote extensively in Arena Magazine, the belief that a single all-purpose AI model will be optimal or even inevitable comes not from reality, but from fantasies of monopoly rents.
The curse of every contrarian is for his views to be imitated by the crowd, twisted to the point of cliché. Peter Thiel believed he was stoking controversy when he wrote “Creative monopoly means new products that benefit everybody and sustainable profits for the creator. Competition means no profits for anybody, no meaningful differentiation, and a struggle for survival.” Now, regulators and columnists are assuming that all emerging technologies naturally become monopolistic. This simplistic view fails to account for the distinct qualities and dynamics of each sector. As a result, regulators confident that monopolies are inevitable are setting policies with those assumptions — and with complete disregard for the consequences for startups.
…
[T]here is rapid diversification throughout the AI pipeline. What was once a single, vertically integrated process done by one company is separated into specialized improvements. As funding becomes more available and demand grows for niche applications of AI, the machine learning process is being sliced into ever narrower, more precise subprocesses.
Don’t be deterred. Build the specialized AI models and applications you want in the world.
