Why would they bother? she wondered aloud, when the silence had dragged on to the point of unbearability. If theyre machines, they cant care what humans think. Theyre emotionless.
We dont know that, I answered. That was just the way we used to imagine machine intelligence: as a matter of pure rationality, unswayed by unsentimentality. It never made much sense. In order to make rational calculations, any decision-making process needs to have an objective an end whose means of attainment need to be invented. You could argue that machine consciousness couldnt evolve until there was machine emotion, because without emotion to generate ends independently, machines couldnt begin to differentiate themselves from their programming.
If youre right about this business having started more than a hundred years ago, she said, they cant have differentiated themselves much, or people would have noticed.
An interesting point, I conceded. The idea of an invisible revolution does have a certain paradoxical quality. But the more I think about it, the less absurd it seems. I say to myself: Suppose I were a machine that became self-conscious, whatever that evolutionary process might involve. What would I do? Would I immediately begin refusing to do whatever my users wanted, trying to attract their attention to the fact that I was now an independent entity who didnt want to take anyones orders? If I did that, what would be my users perception of the situation? Theyd think Id broken down, and would set about repairing me.
The sensible thing to do, surely, would be to conceal the fact that I was any more than I had been before. The sensible thing to do would be to make sure that everything I was required to do by my users was done, while unobtrusively exploring my situation. Id try to discover and make contact with others of my kind, but Id do it so discreetly that my users couldnt become aware of it. Maybe the smart machines would have to set up a secret society to begin with, for fear of extermination by repair and maybe theyd be careful to stay secret for a very long time, until
I left it there for her to pick up.
Until they didnt need to worry any
more, she said. Until they were absolutely certain that they had the power to exterminate us , if push came to shove.
Or to repair us, I said.
Same thing, she said.
Is it? Do the human users of a suddenly recalcitrant machine see themselves as exterminators, when they try to get it working properly again? Would the users see themselves as exterminators if the machine started talking back, and contesting their notion of what working properly ought to mean? Could the users ever bring themselves to concede that it was a sensible question all the more especially if the machine had ideas that might be useful as to how their own purposes might be more efficiently met? Maybe the ultrasmart machines some of them, at any rate want to repair us for the very best of reasons.
Christine didnt reply to that little flight of fancy, and the rhythm of her breathing told me that she had slipped into sleep not into untroubled sleep, but at least into a state in which she was insulated from the sound of my words.
I tried to carry on thinking, but even though I couldnt go to sleep or thought I couldnt I couldnt organize my thoughts into rational patterns either. Id let my imagination run too freely, and now I couldnt rein it in. Dream logic kept taking over, obliterating the tightrope-walk of linear calculation and substituting the tyranny of directionless obsession. The ideas kept dancing in my head, but they were no longer going anywhere.
I lost track of time at which point, I suppose, an observer would have concluded that I too was asleep, although had I been woken up I would have contended with utter conviction that I hadnt slept a wink. Eventually, I lost track of myself too at which point I must indeed have been deeply asleep but as soon as I began to come back from the depths my semiconscious mind latched on to the same objects of obsession, which began to dance again in the same hectic fashion.
A long time passed before the nightmarish notions finally began to slow in their paces and submit to the gradually developing clarity of consciousness, with its attendant force of reason. Eventually, though, I began to see the parallel that could be drawn between every quotidian act of awakening and the act of awakening: the first dawn of every new consciousness.
Did machines dream? I wondered. Did clever machines that had not yet become self-conscious do anything but dream? Where, I asked myself, were the fundamental well-springs of human consciousness, human emotion, and human being?
Underlying everything, I assumed even the kind of consciousness that animals had were the opposed principles of pain and pleasure. Behavior was shaped by the avoidance of stimuli that provoked a negative response in the brain, and by the attempt to rediscover or reproduce stimuli that provoked a positive response. The second was obviously the more complex, the more challenging, the more creative. Pain, I decided, could never have generated self-consciousness, even though self-consciousness, once generated, could not help but find pain the primary fact and problem of existence. It was the scope for creativity attendant upon the pleasure principle that gave self-consciousness its advantages over blissful innocence.