The Two Faces of Tomorrow
I set the story forty years into the next century, by which time an integrated
global system is managing much of the world's affairs. However, proposals
for a major upgrade involving new software that learns are causing serious questions
to be asked about the degree of decision-making that it can be entrusted with.
The trouble is that while the solutions that it comes up with are logically
flawless, they are unconstrained by the kind of common sense that humans acquire
through a lifetime of real-world experience--which on several occasions has
almost resulted in catastrophe. One school of opinion argues that the
only way to go is forward, accepting the risks and allowing the system to learn
from experience in the same way that people did. "Besides, if anything
really bad starts happening, we can always downgrade again or pull the plug."
"How can you guarantee that it will always let you?" the opponents
The answer eventually agreed is to run a test on a world-in-miniature.
One of the new space habitats is taken over for the experiment and equipped
with a supersystem containing all the advanced capabilities proposed for incorporation
into the global net. The system is programmed for self-preservation as
its highest goal, introducing deliberately the faculty of a "survival instinct"
that the critics have speculated could arise spontaneously. The scientists
then begin "attacking" it in a series of escalating tests to find
out what it's capable of. It's far from Earth, so anything unexpected
will be isolated and contained. A strong military presence is included
in the mini-world's population--just in case things should take a nasty turn.
And if it gets out of hand, we can always evacuate the whole place and nuke
it. But the System, of course, doesn't quite see things that way.