Or.. the era of turbocharged goalpost moving
The ARC benchmark was designed by François Chollet to serve one goal : be sufficiently difficult, demanding so that it cannot be “hacked” by some “cheating” AI techniques, LLM or whatever. But it must do so in a rigorous, systematic and simple to define way, it cannot be vague or ambiguous. It must be (relatively) easy for a human, but hard for a program. And when I first looked at it, I admired its simplicity and purity: the problems are simple but deep, and it’s clear that for many of them, you need to grasp something that goes beyond mere pattern recognition or superficial pattern matching. They seem to require some seriously deeper thinking. And if, like me, you thought for…
Or.. the era of turbocharged goalpost moving
The ARC benchmark was designed by François Chollet to serve one goal : be sufficiently difficult, demanding so that it cannot be “hacked” by some “cheating” AI techniques, LLM or whatever. But it must do so in a rigorous, systematic and simple to define way, it cannot be vague or ambiguous. It must be (relatively) easy for a human, but hard for a program. And when I first looked at it, I admired its simplicity and purity: the problems are simple but deep, and it’s clear that for many of them, you need to grasp something that goes beyond mere pattern recognition or superficial pattern matching. They seem to require some seriously deeper thinking. And if, like me, you thought for a minute about how you’d try to tackle them, in a programmatic way (ML or otherwise), it was quite easy to become convinced, that this is quite a good benchmark. And at first, what happened, on Kaggle, for instance, was exactly that: nobody could get even remotely decent results, the problem set really felt like a tough nut to crack. From there, the temptation was great, to suggest the idea that whenever ARC would be cracked, AGI would have arrived!
ARC was in 2019, before the LLM-fueled latest Cambrian explosion of AI. Now that we’ve got ChatGPT and that it’s so impressively intelligent, can we expect better? At first sight it’s not entirely clear, because, ChatGPT is adept at manipulating words, and these problems are of another nature, they are meant to operate more in other realms of reasoning, which are not necessarily connected to language in a clear way. But ChatGPT is also a very good programmer so.. maybe? Whatever might be the conceptual or technical difficulties, people have been trying even harder to solve ARC, recently, and now we’ve got the ARC Prize, to up the ante. And lo and behold, progress was made, and steadily! And it’s impressive!
But now this (relative) success begs the question: what did we achieve exactly? Are we closer to AGI? It’s not clear, or at least it seems that the debate is not ready to be settled soon.
So I’m asking the question: what could we logically conclude from this, the observation that the speed at which the goalpost is moved, in AI, seems to increase? And I would like to propose a radical answer, which I’d like to call the “Insurmountable Hans” principle, in honor of Clever Hans, the horse that learned to talk, and that we could not ever take seriously. I submit that whatever AI becomes or does, we will NEVER accept, in an anthropological way, that a machine can be really intelligent or behaving (or even just thinking) in a way that is equivalent to humans. This will never be a question of definition or criteria, it will just be a flat out anthropological refusal, to ever grant this ineffable quality, to what we consider ultimately a thing. In this sense, the problem of AGI is insurmountable, and the quest will never end.