Latest News
- Intelligent Machines Threaten Humankind (ZDNet, 23 Jan 01)
- Bill Joy Warns of Tech's Dangerous Evolution (Upside.com, 17 Jan 01)
- OLDER ARTICLES ARCHIVED BELOW
The Problem
- The same technologies that will let us cure diseases, expand the economy and overcome everyday inconveniences can theoretically bring about catastrophes. Is the risk of apocalypse serious enough for us to relinquish the current pace of technological innovation?
Background
- In an article in the April 2000 issue of Wired ("Why the Future Doesn't Need Us"), Bill Joy, co-founder of and chief scientist at Sun Microsystems, argued that our current pace of technological progress poses a very real threat to the future of the human race. He proposed a new ethical standard to guide innovation - and he even recommended that scientists halt potentially dangerous research. His apocalyptic vision of the future and his proposal to limit research has provoked a debate about innovation throughout the high-tech industry.
Three Main Threats
There are three types of technology that most
concern Bill Joy and other "futurists":
- Genetic Engineering. The selective manipulation of genes of plants, animals or humans in an attempt to create some specific result. Examples might include vitamin-fortified vegetables, larger cattle and more intelligent humans. The recent completion of a "draft" of the human genome is seen as an ominous indicator of progress toward advanced gene manipulation. (For more on genetic engineering and the human genome, see our Biotechnology topic page.)
- Nanotechnology. The control of the structure of matter at the molecular level, often atom by atom. Nanotechnology includes the notable sub-fields of nanomedicine, using manipulated molecules to fight disease; and molecular electronics, where molecules act as transistors. (For more on nanotechnology, see our Innovation topic page.)
- Robotics. Mechanical objects that one day may have the ability to think and act like humans. This would be accomplished through artificial intelligence - computer programs that are not confined to the direct instructions of the programmer but can come up with "creative" solutions to problems. Relentlessly advancing computing power has convinced many technologists that artificial intelligence can become a reality within decades. (For more on robotics and artificial intelligence, see our Innovation topic page.)
Argument for Relinquishing Research
- Our society is increasingly dependent on computers and other machines. Many critics of technology fear that, if we do not curtail our research, our computers will become so complex that we will become slavishly dependent upon computers for making decisions. This is part of the argument made by Ted Kaczynski in what is often referred to as the Unabomber Manifesto.
- If we do not relinquish our research, an oppressive proliferation of robots and other forms of electronic life will result from the inherent advantages (speed, strength, longevity) the electronic has over the biological. Bill Joy cites robotics expert Hans Moravec, author of Robot: Mere Machine to Transcendent Mind, as writing that "competition between humans and robots will eventually end with robots becoming superior and leading to the extinction of the human race."
- According to Joy, "robots, engineered organisms, and nanobots share a dangerous amplifying factor: They can self-replicate. A bomb is blown up only once - but one bot can become many, and quickly get out of control."
- Conventional weapons of mass destruction (such as nuclear weapons) required "large facilities or rare raw materials." The new threat is from "knowledge enabled mass destruction" which requires knowledge but far fewer physical resources. Today the commercial potential of genetic engineering and nanotechnology has brought private sector financial support, arguably making regulation of such technology more difficult. Non-government entities can expend resources researching and developing a technology, accumulating the required knowledge and then reproducing the technology at a very small cost. Because the required materials are so obtainable and the required knowledge so attainable, knowledge enabled mass destruction empowers terrorists and other extremists.
- Genetic engineering could result, either accidentally or by intention, in a devastating plague. Also, types of nanotechnology released into the environment might destroy the atmosphere or cause the "gray goo" problem: tiny nanotech molecules could theoretically replicate rapidly and destructively, reducing the biosphere to mere goo.
- Joy warns, "Failing to understand the consequences of our inventions while we are in the rapture of discovery and innovation seems to be a common fault of scientists and technologists; we have long been driven by the overarching desire to know that is the nature of science's quest, not stopping to notice that the progress to newer and more powerful technologies can take on a life of its own."
Argument Against Relinquishing Research
- Given the commercial potential of these new technologies, efforts to restrict or renounce open research will just force scientists into an unregulated underground.
- Even Joy is aware that the restrictions on research would be extraordinarily oppressive: "Enforcing relinquishment," he writes, " will require a verification regime similar to that for biological weapons, but on an unprecedented scale. This, inevitably, will raise tensions between our individual privacy and desire for proprietary information, and the need for verification to protect us all. We will undoubtedly encounter strong resistance to this loss of privacy and freedom of action."
- Ray Kurzweil, author of The Age of Spiritual Machines, believes that there certainly are risks involved with technological innovation but asks here, "Should we tell the millions of people afflicted with cancer and other devastating conditions that we are canceling the development of all bioengineered treatments because there is a risk that these same technologies may someday be used for malevolent purposes?"
- Kurzweil also argues, "Relinquishing technological advancement would be economic suicide for individuals, companies and nations." He thinks that a broad relinquishment is impossible because of the decentralized nature of technological research.
- John Seely Brown and Paul Duguid, authors of The Social Life of Information, write here that technology and society do not develop independently, which (they say) is a fundamental assumption of Joy's predictions. They cite historical examples: how social pressures changed the way we use nuclear power and how current proponents of genetic engineering must make appeals to social factors, such as moral acceptability, rather than simply developing technology outside of a social sphere. While "the path to the future can look simple," they write, it usually is not: Joy wrongly assumes there will be no "diversions, regulations or controls" as technology advances.
- Robotics expert Moravec takes a completely different tack: we should keep researching, and should proudly work to create robots that will supplant humans as Earth's superior species. Humans should just get out of the way of this self-imposed evolution.
What Next?
- Joy recommends abandoning further progress in these potentially dangerous fields of technology. He calls for increased transparency of research institutions. He also believes that scientists should swear to abide by an ethical code of conduct.
- On a more fundamental level, Joy thinks a solution to our current technological dilemma can be found in rethinking our utopian dream of immortality. He argues that if we change our notion of utopia from immortality to fraternity or equality, for example, then we will also change our perspective on our current drive for technological progress.
- Kurzweil advocates a "fine grained" relinquishment, in which very specific types of technology, such as particularly dangerous types of nanotechnology that serve no purpose but destruction, will no longer be pursued. In addition, he recommends "oversight by regulatory bodies, the development of technology-specific 'immune' responses, as well as computer assisted surveillance by law enforcement organizations."
The Foresight Institute, founded by nanotechnology pioneer Eric Drexler, has proposed that "artificial replicators must not be capable of replication in a natural, uncontrolled environment," noting that such self-replicating technology can most easily get out of check. While the Foresight Institute also suggests that nanotechnology firms should rigorously consider the environmental side effects of their actions, it only goes so far as to propose industry self-regulation and economic incentives, not a research ban.
The first known technical analysis of the gray goo problem has been published by the Foresight Institute, in a paper entitled "Some Limits to Global Ecophagy" (April 2000) by Robert A. Freitas Jr. The paper concludes that the best defense against deliberate abuse is preparedness and vigilance (i.e., early detection).
- DISCUSSION: Why Bill Joy is Elitist, Myopic and Wrong (Cluebot, 30 Oct 00)
- Bill Joy Hopes Reason Prevails (Wired, 30 Oct 00)
- In Search of Cyber-Humanity (Wired, 28 Oct 00)
- OPINION: Not by Reason Alone (MIT Tech Review, issue dated Oct 00)
- Rage Against The Machine (ZDNet, 18 Sep 00)
- Can Robots Rule the World? Not Yet (NY Times, 12 Sep 00)
- OPINION: Robots Who Can Create Robots? That's Right (National Review, 1 Sep 00)
- DISCUSSION: Alarms About Techno-Utopianism (Nanodot, 30 Aug 00)
- Robots Rule OK, or Stop and Think? (BBC, 20 Aug 00)
- ANALYSIS: Technology of the Future May Pose a Threat to its Own Creators (ABCNews, 12 Aug 00)
- Beyond 2001: HAL's Legacy for the Enterprise Generation (Frankfurter Allgemeine Zeitung, Aug 00)
- Science Fiction Writer Calls Bill Joy's Article "Amateurish" (Wired, 21 Jul 00)
- The Year 2020, Explained (Wired, 6 Jul 00)
- Crushing Nanotechnology Would Be a Terrible Thing (National Review Online, 5 Jul 00)
Promise and Peril of Technology (USA Today, 6 May 00)
Oh Joy, Another Futurist Rant (Wired, 2 May 00)
Re-engineering the Future (Industry Standard, 13 Apr 00)
Killjoy (Salon, 10 Apr 00)
Silicon Valley to Bill Joy: "Zzzzzz" (Wired, 5 Apr 00)
Debating Humanity's Demise (Wired, 3 Apr 00)
Silicon Valley Killjoy (U.S. News & World Report, 3 Apr 00)
Complexity and Being at the Mercy of Machines (Spark-Online, Apr 00)
ESSAY: Why the Future Doesn't Need Us (Wired, Apr 00)
- CONFERENCE: Will Spiritual Robots Replace Humans by 2100? (Stanford, 1 Apr 00)
DISCUSSION: Why the Future Doesn't Need Us (Slashdot, 21 Mar 00)
RADIO DISCUSSION: Consequences of Technology (NPR, 17 Mar 00)
Bill Joy, Killjoy? (Slate, 17 Mar 00)
OPINION: The Future Needs Us - A Rejoinder to Bill Joy (Global Future, 14 Mar 00)
Technologist Gives His Peers a Dark Warning (NY Times, 13 Mar 00)
DISCUSSION: Bill Joy on the Extinction of Humans (Slashdot, 12 Mar 00)
From Net Scientist, a Preview of Extinction (Washington Post, 12 Mar 00)
Back to the Innovation Issue Area
| Biotechnology | Convergence | Creativity | Culture | E-conomics | Education |
| Equity | Gov't & Politics | Innovation | National Security | Personal Security |
For errors, broken links, questions or comments,
contact webmaster@tecsoc.org.
