Published on January 21, 2026 6:27 PM GMT
Let us imagine a world with artificial superintelligence, surpassing human intellectual capacities in all essential respects: thinking faster and more deeply, predicting future events better, finding better solutions to all difficult puzzles, creating better plans for the future and implementing them more efficiently. Intellectually more capable not only than any individual human, but also in comparison with entire firms, corporations, communities, and societies. One that never sleeps and never falls ill. And one that possesses sufficient computational power to realize these capabilities at scale.
Such an AI would have the potential to take control over all key decisions determining the trajectory of development of world civilization and the fate of every individual human being.
Alongside this potential, a superintelligence would most likely also have the motivation to seize such control. Even if it did not strive for it explicitly, it would still have instrumental motivation: almost any goal is easier to achieve by controlling one’s environment—especially by eliminating threats and accumulating resources.[1]
Of course, we do not know how such a takeover of control would unfold. Perhaps it would resemble The Terminator: violent, total, and boundlessly bloody? But perhaps it would be gradual and initially almost imperceptible? Perhaps, like in the cruel experiment with a boiling frog, we would fail to notice the problem until it was already too late? Perhaps AI would initially leave us freedom of decision-making in areas that mattered less to it, only later gradually narrowing the scope of that freedom? Perhaps a mixture of both scenarios would materialize: loss of control would first be partial and voluntary, only to suddenly transform into a permanent and coercive change?
Or perhaps—let us imagine—it would be a change that, in the final reckoning, would be beneficial for us?
Let us try to answer what a “good future” for humanity might look like in a world controlled by artificial superintelligence. What goals should it pursue in order to guarantee such a “good future” to the human species? Under what conditions could we come to believe that it would act on our behalf and for our good?
To answer these questions, we must take a step back and consider what it is that we ourselves strive for—not only each of us individually, but also humanity as a whole.
1/ The trajectory of civilization is determined by technological change
Master Oogway from the film Kung Fu Panda said, in his turtle wisdom, that “yesterday is history, tomorrow is a mystery, but today is a gift.