The Unreasonable Effectiveness of Language
It has been amazing to witness the power of language in AI models in the past year. Whereas I like most others once thought that language models can never be enough to get to AGI, it is now starting to seem like the current paradigm of LLMs may in fact get us there.
Or if they don’t get us to AGI, they may at least get us to TAI, i.e. transformative AI rather than artificial general intelligence, which is probably the more interesting milestone. AGI is seeming more and more like an unnecessary milestone. TAI is often defined as AI that has a massive impact on the economy. For example leading to an annual increase of GDP by more than 15%. This would be, as far as we can estimate, on par with the agrarian and industrial revolutions.
The stupefying growth in capabilities between GPT 2, 3, 3.5 and 4, suggests that just adding more language, so to say, and more ability to process that language, really will get us to truly powerful AI. What’s even more interesting is all the non-language abilities that the models now possess. Whereas we might once have thought that artistic and visual abilities were a separate kind of intelligence, DALL-E and the other models have now shown that language is enough to achieve visual intelligence. Similarly with math, the math skills shown by the GPT models arguably show that math should perhaps be thought of as just another language. Same with coding. The fact that GPT-4 has skills in many non-English languages despite being mostly trained on English is another indicator for the generality of language skills. Most impressive are perhaps the “common sense” skills displayed by GPT-4. In the Microsoft paper, GPT-4 knows how to stack a book, 9 eggs, a laptop, a bottle and a nail in a stable manner.
Many commentators, even ones I otherwise respect, like Noah Smith, seem interminably stuck in the “it’s just autocomplete” bucket. But watching the emergent skills in GPT-4 might rather suggest that language seems to be much more the basis for intelligence than we had thought. So maybe the opposite is true, it’s intelligence that is just autocomplete. Maybe human thought itself and our much-vaunted intelligence is really the glorified autocomplete. This would certainly be in line with earlier blows to humanity’s self-importance inflicted by Galileo and Darwin. Like Olle Häggström said on the John Danaher podcast, he also sometimes “hallucinates” in the sense of saying things that are incorrect.
Language is certainly the basis of human civilization, as Harari and Harris argue in the NYT: “In the beginning was the word. Language is the operating system of human culture.” And perhaps language is not just the tool of thought, but the basis of thought. This doesn’t just teach us about machines and how they think, but also about ourselves. There has long been a debate about whether language is innate to thinking or whether one can think without language. Not something we can know with certainty, of course, but GPT-4 definitely makes me feel more inclined to think that we cannot. Manipulating language might be instrumental to thinking, or even synonymous with thinking.
It is a bit reminiscent of the debate around consciousness. For centuries, we have debated what its roots are, why we developed it and how it arises. Now there are more theories that suggest that consciousness is perhaps just fundamental to the universe. Instead of trying to find complicated theories for how consciousness can arise, perhaps we just plug it into the beginning. It is alluring to think it might be the same with language. Instead of questioning why language arose and whether there was thought beforehand, we just plug it in at the core and say that language is thinking. No thinking can happen without a system of language. This doesn’t need to be an externally spoken language, but at least an internal system of naming concepts that is essential. Perhaps our greater use of language per se is then what separates us from other animals. After all, size of brain doesn’t seem to be the differentiator.
This could also settle the debate about whether there is a language instinct or whether we are born as a blank slate and acquire language like other skills. In this view, language is innate, that’s how the brain evolved to be able to think at all.
What does this tell us about the path to AGI or TAI? It would seem to suggest that timelines should be brought in closer. If we can get to transformative AI in the current LLM paradigm, a lot of the barriers that would slow down process fall away. Of course, there might still be barriers to come. Progress might plateau. We might find that, at some point, no matter the level of compute, intelligence ceases to increase. GPT-4 has already ingested much of the text that humanity has ever produced. Going forward, ingesting the often flawed text produced by itself and its brethren might be less valuable than the “original” human text. But barriers to progress are looking less likely, and near-term TAI is therefore looking more likely.