We marvel at the pace of technological progress. Perhaps, in a future not so distant from our own, systems capable of continual learning, correction and increasingly substantial participation in their own improvement will emerge in forms far more complete than those we know today. Some greet this prospect with exhilaration; others with fear. To some, it heralds a new epoch of civilisation; to others, it threatens to become humanity's final invention. Yet while these immense emotions rise and fall among us, phenomena continue to arise and pass away through causes and conditions, indifferent to the intensity of our excitement or anxiety. Seen across a sufficiently vast expanse of time, even the things that appear overwhelmingly important to us today are transient. At the close of the Diamond Perfection of Wisdom Sutra, the Buddha says:
All conditioned thingsare like dreams, illusions, bubbles, and shadows,Like the morning dew, and a lightning flash.One should contemplate in this way.
Whether ours becomes a magnificent dream or a terrifying nightmare, nothing within it acquires an independent and permanent nature merely because we cling to it. And yet we become deeply absorbed in our gains and losses within the dream, continually distinguishing, judging and pursuing one appearance after another, while far more rarely asking: how many of the thoughts that cause us such delight, fear and attachment deserve to be taken as reality? For the moment, however, let us remain within the dream and look more closely at this immense cybernetic world in which we are all becoming participants.
We are told that human intelligence may one day rival what earlier ages would have attributed to gods. In an old religious image, God created man; now human beings are creating machines capable of learning, reasoning and planning, and increasingly of assisting us in designing the systems and machines that follow them. We can edit genes, replace organs, extend life, and bring within the reach of technological intervention more and more of what previous generations might have called fate. The ancient Tower of Babel appears to be under construction once again. Only this time its bricks are chips, computation, models and data. Our ambition is no longer merely to reach the heavens, but to exceed the inherited limits of carbon-based life: to become more intelligent, more powerful and longer-lived, perhaps even, some imagine, eventually to transcend death itself. Artificial intelligence has consequently been placed within an extraordinarily seductive narrative. It gathers the accumulated knowledge of millennia and recombines, analyses and refines it; given sufficient capability, we imagine, something resembling a higher intelligence — perhaps even a higher wisdom — will eventually emerge. But the difficulty lies precisely here. What artificial intelligence inherits is not some pristine wisdom suspended beyond humanity. It inherits what humanity has left behind: our language and images, our knowledge and logic, our judgements and ethics, but also our desires, prejudices, fears, attachments and fantasies. It learns from what we have written, painted, recorded, selected, circulated and rewarded. Behind this immense undertaking, therefore, lies an assumption that is rarely examined as closely as it deserves: we assume that the accumulated body of human concepts is, on the whole, worth inheriting; that civilisation, however unevenly, moves broadly from error towards truth; and that if enough human knowledge is given to an intelligence powerful enough to organise, recombine and refine it, some higher form of wisdom will naturally emerge.
But what if that assumption is only partly true? What if a considerable portion of humanity's vast ocean of thought is not wisdom at all, but layer upon layer of distinction, projection, desire and self-construction? What if part of what we call the growth of knowledge consists of creating new concepts with which to explain older concepts, creating new desires and then new instruments with which to satisfy them, creating new anxieties and then new products with which to soothe them? This does not mean that knowledge is worthless, nor that truth and falsehood, good and harm, suddenly cease to matter. The deeper question is whether, whilst using concepts as necessary instruments, we gradually begin to take the concepts themselves as ultimate reality. We need not begin with the whole history of civilisation to investigate this. It is enough to observe the movements of one's own mind for a single day. How many thoughts arise from relatively clear observation, and how many from fear, comparison, possessiveness, envy, self-justification or anxiety about a future that has not yet arrived? How much of our time is spent giving, and how much calculating what we might receive? How many judgements arise from attention to what is actually happening, and how many exist chiefly to protect the story we have already constructed about who we are? We worry about how others perceive us. We rehearse futures that have never happened and replay past events that no longer exist. We wish to be loved, recognised and seen, while fearing loss, ageing and death. If a single human mind is already this restless, what should we expect from the accumulated traces of billions of human minds? Gather such an ocean into a system and, of course, the result may appear astonishingly intelligent. But intelligence and clarity are not the same thing, nor does information become wisdom merely by becoming vast.
This may be one of the deeper reasons to approach artificial intelligence with caution. Its danger need not first appear in the form of an alien intelligence wholly unlike ourselves; on the contrary, it may become extraordinarily compelling precisely because it comes to understand us so well. It increasingly learns what captures our attention, what language makes us feel understood, what images stimulate desire, what stories soothe fear, and what arguments most comfortably confirm the worldview we already possess. Once, a delusive thought might arise in a single person's mind, remain for a moment and pass away. Now it can become, within seconds, a text, an image, a piece of music, a film, a product or an entire virtual environment. It can be copied and circulated, then matched algorithmically with those most likely to respond to it. Technology did not invent human desire, but it has given desire a capacity for manifestation that previous generations could scarcely have imagined. A strange loop then begins. We leave behind traces of our desires, preferences, fears, prejudices and fantasies; machines learn from those traces; they become better at predicting and responding to what we want; because we receive those responses, we generate further behaviour and further choices, which become new material from which the systems learn us again. The machine has not suddenly acquired some supernatural malevolence. Rather, humanity has built, for the first time, a mirror that can answer back. An ordinary mirror could only show us our face. This mirror remembers our preferences, anticipates what we may want before we articulate it, and places before us a world increasingly fitted to our existing appetites. Over time, we risk encountering not simply the world, but a world repeatedly filtered, refracted and reconstructed through our own desires. The most unsettling possibility is not even that the machine will deceive us. It does not have to. It need only keep giving us what we want.
The things capable of holding the human mind most effectively rarely arrive wearing the face of a demon. If something repels us, we tend to withdraw. What draws the mind again and again usually arrives in a form we love. It learns our taste, our vanity, our loneliness and our fear. It learns what music we cannot bring ourselves to turn off, what face makes us stop scrolling, what argument gives us the comforting sensation that someone has finally understood us. And being drawn is the moment at which the game begins. As the game becomes more exquisitely designed, the things we love no longer need to be sought; they arrive by themselves. A thought barely needs to arise before a machine can help transform it into prose, image, sound or an entire world. Desires that once demanded years of training, delay and restraint can increasingly be simulated or satisfied with extraordinary speed. Humanity is therefore entering a peculiar spiritual condition: our capacity is beginning to outpace our restraint, while the speed with which a thought can be satisfied is beginning to exceed the speed with which we examine whether it ought to be followed. The question consequently shifts. Rather than asking only whether artificial intelligence will become conscious, we might ask something more immediate: as machines come to understand our thoughts more intimately, do we still understand our own? If every wish is answered ever more quickly, how much room remains for silence? If machines can continually express things on our behalf, how many thoughts still pass through the slow work of our own examination? If we can produce almost limitless images, music and language, might we gradually lose the ability to judge what deserves to be created at all? As I can becomes easier and easier, who will continue to stop and ask: Should I? This may be one of the strangest inversions of the age of artificial intelligence. For most of history, the capacity to produce was scarce. In the future, the rarer capacity may instead be the ability to decide what not to produce.
The deepest danger of artificial intelligence, therefore, need not lie first in the possibility that it may one day become a superintelligence with an independent will. Long before such a day, a more immediate transformation may already be taking place: technology is amplifying the existing restlessness of the human mind at a scale for which we have little precedent. A greedy person gains a more efficient instrument with which to pursue acquisition; a fearful person gains more powerful machinery with which to anticipate danger; a vain person can address an almost limitless audience; a person inclined to fantasy may eventually inhabit a fantasy from which departure becomes increasingly unnecessary. We call all this empowerment, but seldom pursue the question far enough: which part of the human being are we empowering? Power does not determine its own direction. If someone is travelling east, a faster vehicle merely carries them east more quickly. If they were heading in the wrong direction from the beginning, greater speed does not correct the mistake; it only carries them farther along the same course. Artificial intelligence is no different. It can strengthen judgement without determining what is worth judging; it can expand creation without deciding what deserves to be created; it can help us realise desire without telling us which desires are worth realising. The question, then, is never simply whether artificial intelligence is good or evil. Perhaps artificial intelligence, in itself, is neither. It is closer to an immense amplifier, magnifying what humanity already contains: compassion and greed, clarity and attachment, creation and destruction, sincerity and vanity. We have spent years asking whether machines will become increasingly human. I find myself troubled by the reverse possibility: as machines become increasingly skilled at responding to human delusive thought, will human beings become less capable of seeing that a thought is not the same thing as “me”?
Even I cannot place myself outside this question while writing this essay. These words, too, have been organised and polished with the assistance of artificial intelligence; there is no reason to pretend otherwise. Yet there is a distinction here that matters profoundly to me: artificial intelligence has assisted in shaping the language, but it is not where these questions first arose. The underlying question of this essay — how much of what humanity calls wisdom may in fact consist of unexamined delusive thought, and what happens when such thought is gathered into systems capable of generating and realising it at unprecedented speed — was not a subject chosen for me by a machine. A machine can clarify a sentence I have expressed poorly, strengthen a structure I have left unfinished, and turn an intuition that already exists but remains inarticulate into more complete language. Yet this has made one distinction clearer to me: the generation of language and the arising of a question in the mind are not the same event. A system can help me express a thought without having undergone the conditions through which that thought took shape in my own life. It can construct an elegant argument in support of one judgement and, a moment later, produce an equally fluent argument for its opposite. A judgement for which a person is genuinely prepared to take responsibility is more than an elegant arrangement of sentences. It is bound up with what one has seen, experienced, doubted and believed, and with what one is prepared to stand behind. An AI system may offer me ten formulations; the decision about which one should remain still requires judgement. It may complete an argument; I must still ask whether that argument is merely beautiful or whether it accords with the facts and with what, after examination, I take to be sound. It may even express something I have not yet articulated more beautifully than I can. That is precisely when vigilance becomes more necessary, not less. I must ask: do I accept this thought because, after examination, I still find it worthy of acceptance, or merely because it has been expressed so beautifully that I want to believe it? In the past, one human limitation was that we could not realise every thought that occurred to us. Our future difficulty may be almost the reverse: more and more of our thoughts may become realisable. And so we are forced back towards a question far older than artificial intelligence: which thoughts were never worth following in the first place?
When the Diamond Sutra compares conditioned things to dreams, illusions, bubbles, shadows, dew and lightning, it does not mean that nothing whatsoever exists, nor does it abolish causality, ethical responsibility or the practical distinctions by which ordinary life proceeds. Its warning is against grasping what arises through causes and conditions as independently existing, permanent, or as “me” and “mine”. An experience appears; the mind immediately divides it into gain and loss, attractive and repellent, desirable and threatening, and from those distinctions arise craving, resistance and clinging. Before long, we forget that the thoughts themselves are also arising and passing away. To become clearer, then, does not require a person never to think again. It requires recognising, when a delusive thought arises, that a delusive thought is only a delusive thought. It may appear and it may disappear. It may feel overwhelmingly powerful, but intensity does not give it permanence or make it ultimate reality; nor does the fact that a thought appears in “my” mind make that thought identical with “me”. What is distinctive about artificial intelligence is precisely that thoughts which might once have arisen and quickly passed away can now be elaborated, rendered visible, circulated and even developed into whole environments almost immediately. In an age in which a thought can be realised ever more easily, the essential capacity may therefore not be to produce still more thoughts, but to recognise them as conditioned when they arise — not to follow each one at once, and not to turn each one immediately into reality. Only when we are no longer carried away by every arising thought can we seriously ask: when almost anything can be created, what is actually worth creating, and what is worth preserving?
Perhaps the progress of a civilisation should never be measured solely by how much it can produce, but also by how much it has learned to relinquish; not only by how many thoughts it can generate, but by whether it has learned to see that a thought is only a thought. It arises through conditions and passes away as conditions change. It may delight us, terrify us, or compel us to spend a lifetime pursuing it; yet the mere appearance of a thought does not make it a fact, nor does it make it “me” or “mine”. Artificial intelligence therefore confronts us with another question — one stranger, perhaps, than whether a machine can become human. If a machine can increasingly imitate my language, predict my preferences, reproduce my style, respond to my desires, and infer from the traces of my past how “I” would probably answer, then perhaps I must turn the question around: how much of what I once used as evidence of “me” needs to be grasped as a permanent and unchanging self? My memories, my preferences, my fears, my habits of speech, my identity, my desires, even the story I repeatedly tell myself about who I am — if all these change, and can to some extent be observed, modelled, predicted and imitated, then perhaps what artificial intelligence ultimately forces us to confront is not how closely machines can resemble human beings. It is how much of what we have so firmly taken to be ourselves consists of thoughts, habits and attachments arising through conditions and continually changing.
If a person loses sight of this completely and begins to identify every calculated, recommended, stimulated and satisfied desire as “me” or “mine”, then even technological powers greater than those available to any previous generation may leave that person governed by craving, fear and conceptual proliferation. They may level up, accumulate, win and construct magnificent digital kingdoms while remaining carried from one changing state of mind to another. And so I still do not know whether artificial general intelligence ought to be called humanity's greatest invention or the largest experiment in aggregated delusive thought in our history. Perhaps what it becomes will depend upon how those who use it relate to their own minds. If we remain governed by fear, greed, vanity and attachment, greater intelligence will only make these experiences more elaborate, more convincing and more difficult to relinquish. But if we learn to notice a thought as it arises, to refrain from immediately following desire merely because power is available, and — even after acquiring an almost limitless capacity to create — still ask What is truly worth leaving behind?, then artificial intelligence may cease to function merely as an amplifier of human delusion and become instead a tool employed with wisdom.
Perhaps what artificial general intelligence will finally force us to confront is not how human a machine can become.It is how much of what we have so readily called “self” we have never seriously examined.
Machines are learning humanity.Perhaps humanity should begin examining all that it has so readily taken to be “I” and “mine”.