“Existential risk from artificial general intelligence”的意思、由来-开放百科全书

The likelihood of this type of scenario is widely debated, and hinges in part on differing scenarios for future progress in computer science.^[5] Once the exclusive domain of science fiction, concerns about superintelligence started to become mainstream in the 2010s, and were popularized by public figures such as Stephen Hawking, Bill Gates, and Elon Musk.^[6]

One source of concern is that a sudden and unexpected "intelligence explosion" might take an unprepared human race by surprise. For example, in one scenario, the first-generation computer program found able to broadly match the effectiveness of an AI researcher is able to rewrite its algorithms and double its speed or capabilities in six months of massively parallel processing time. The second-generation program is expected to take three months to perform a similar chunk of work, on average; in practice, doubling its own capabilities may take longer if it experiences a mini-"AI winter", or may be quicker if it undergoes a miniature "AI Spring" where ideas from the previous generation are especially easy to mutate into the next generation. In this scenario the system undergoes an unprecedently large number of generations of improvement in a short time interval, jumping from subhuman performance in many areas to superhuman performance in all relevant areas.^[1]^[8] More broadly, examples like arithmetic and Go show that progress from human-level AI to superhuman ability is sometimes extremely rapid.^[9]

A second source of concern is that controlling a superintelligent machine (or even instilling it with human-compatible values) may be an even harder problem than naïvely supposed. Some AGI researchers believe that a superintelligence would naturally resist attempts to shut it off, and that preprogramming a superintelligence with complicated human values may be an extremely difficult technical task.^[1]^[7] In contrast, skeptics such as Facebook's Yann LeCun argue that superintelligent machines will have no desire for self-preservation.^[12]

Overview

AI systems uniquely add a third difficulty: the problem that even given "correct" requirements, bug-free implementation, and initial good behavior, an AI system's dynamic "learning" capabilities may cause it to "evolve into a system with unintended behavior", even without the stress of new unanticipated external scenarios. An AI may partly botch an attempt to design a new generation of itself and accidentally create a successor AI that is more powerful than itself, but that no longer maintains the human-compatible moral values preprogrammed into the original AI. For a self-improving AI to be completely safe, it would not only need to be "bug-free", but it would need to be able to design successor systems that are also "bug-free".^[1]^[11]

All three of these difficulties become catastrophes rather than nuisances in any scenario where the superintelligence labeled as "malfunctioning" correctly predicts that humans will attempt to shut it off, and successfully deploys its superintelligence to outwit such attempts.

Citing major advances in the field of AI and the potential for AI to have enormous long-term benefits or costs, the 2015 Open Letter on Artificial Intelligence stated:

This letter was signed by a number of leading AI researchers in academia and industry, including AAAI president Thomas Dietterich, Eric Horvitz, Bart Selman, Francesca Rossi, Yann LeCun, and the founders of Vicarious and Google DeepMind.^[12]

History

Butler developed this into The Book of the Machines, three chapters of Erewhon, published anonymously in 1872.^[14]

In 1965, I. J. Good originated the concept now known as an "intelligence explosion":

Occasional statements from scholars such as Alan Turing,^[16]^[17] from I. J. Good himself,^[18] and from Marvin Minsky^[19] expressed philosophical concerns that a superintelligence could seize control, but contained no call to action. In 2000, computer scientist and Sun co-founder Bill Joy penned an influential essay, "Why The Future Doesn't Need Us", identifying superintelligent robots as a high-tech dangers to human survival, alongside nanotechnology and engineered bioplagues.^[20]

In 2009, experts attended a private conference hosted by the Association for the Advancement of Artificial Intelligence (AAAI) to discuss whether computers and robots might be able to acquire any sort of autonomy, and how much these abilities might pose a threat or hazard. They noted that some robots have acquired various forms of semi-autonomy, including being able to find power sources on their own and being able to independently choose targets to attack with weapons. They also noted that some computer viruses can evade elimination and have achieved "cockroach intelligence." They concluded that self-awareness as depicted in science fiction is probably unlikely, but that there were other potential hazards and pitfalls. The New York Times summarized the conference's view as 'we are a long way from Hal, the computer that took over the spaceship in "A Space Odyssey"'^[21]

Nick Bostrom's 2014 book Superintelligence stimulated discussion.^[22] By 2015, public figures such as physicists Stephen Hawking and Nobel laureate Frank Wilczek, computer scientists Stuart J. Russell and Roman Yampolskiy, and entrepreneurs Elon Musk and Bill Gates were expressing concern about the risks of superintelligence.^[23]^[34]^[35]^[24] In April 2016, Nature warned: "Machines and robots that outperform humans across the board could self-improve beyond our control — and their interests might not align with ours."

Basic argument

A superintelligent machine would be as alien to humans as human thought processes are to cockroaches. Such a machine may not have humanity's best interests at heart; it is not obvious that it would even care about human welfare at all. If superintelligent AI is possible, and if it is possible for a superintelligence's goals to conflict with basic human values, then AI poses a risk of human extinction. A "superintelligence" (a system that exceeds the capabilities of humans in every relevant endeavor) can outmaneuver humans any time its goals conflict with human goals; therefore, unless the superintelligence decides to allow humanity to coexist, the first superintelligence to be created will inexorably result in human extinction.^[4]^[25]

There is no physical law precluding particles from being organised in ways that perform even more advanced computations than the arrangements of particles in human brains; therefore superintelligence is physically possible.^[34]^[35] In addition to potential algorithmic improvements over human brains, a digital brain can be many orders of magnitude larger and faster than a human brain, which was constrained in size by evolution to be small enough to fit through a birth canal.^[27] The emergence of superintelligence, if or when it occurs, may take the human race by surprise, especially if some kind of intelligence explosion occurs.^[34]^[28] Examples like arithmetic and Go show that machines have already reached superhuman levels of competency in certain domains, and that this superhuman competence can follow quickly after human-par performance is achieved.^[27] One hypothetical intelligence explosion scenario could occur as follows: An AI gains an expert-level capability at certain key software engineering tasks. (It may initially lack human or superhuman capabilities in other domains not directly relevant to engineering.) Due to its capability to recursively improve its own algorithms, the AI quickly becomes superhuman; just as human experts can eventually creatively overcome "diminishing returns" by deploying various human capabilities for innovation, so too can the expert-level AI use either human-style capabilities or its own AI-specific capabilities to power through new creative breakthroughs.^[29] The AI then possesses intelligence far surpassing that of the brightest and most gifted human minds in practically every relevant field, including scientific creativity, strategic planning, and social skills. Just as the current-day survival of the gorillas is dependent on human decisions, so too would human survival depend on the decisions and goals of the superhuman AI.^[4]^[25]

Some humans have a strong desire for power; others have a strong desire to help less fortunate humans. The former is a likely attribute of any sufficiently intelligent system; the latter cannot be assumed. Almost any AI, no matter its programmed goal, would rationally prefer to be in a position where nobody else can switch it off without its consent: A superintelligence will naturally gain self-preservation as a subgoal as soon as it realizes that it can't achieve its goal if it's shut off.^[50]^[30]^[31] Unfortunately, any compassion for defeated humans whose cooperation is no longer necessary would be absent in the AI, unless somehow preprogrammed in. A superintelligent AI will not have a natural drive to aid humans, for the same reason that humans have no natural desire to aid AI systems that are of no further use to them. (Another analogy is that humans seem to have little natural desire to go out of their way to aid viruses, termites, or even gorillas.) Once in charge, the superintelligence will have little incentive to allow humans to run around free and consume resources that the superintelligence could instead use for building itself additional protective systems "just to be on the safe side" or for building additional computers to help it calculate how to best accomplish its goals.^[1]^[12]^[50]

Thus, the argument concludes, it is likely that someday an intelligence explosion will catch humanity unprepared, and that such an unprepared-for intelligence explosion may result in human extinction or a comparable fate.^[4]

Sources of risk

Poorly specified goals: "Be careful what you wish for" or the "Sorcerer's Apprentice" scenario

While there is no standardized terminology, an AI can loosely be viewed as a machine that chooses whatever action appears to best achieve the AI's set of goals, or "utility function". The utility function is a mathematical algorithm resulting in a single objectively-defined answer, not an English statement. Researchers know how to write utility functions that mean "minimize the average network latency in this specific telecommunications model" or "maximize the number of reward clicks"; however, they do not know how to write a utility function for "maximize human flourishing", nor is it currently clear whether such a function meaningfully and unambiguously exists. Furthermore, a utility function that expresses some values but not others will tend to trample over the values not reflected by the utility function.^[32] AI researcher Stuart Russell writes:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable. This is essentially the old story of the genie in the lamp, or the sorcerer's apprentice, or King Midas: you get exactly what you ask for, not what you want. A highly capable decision maker — especially one connected through the Internet to all the world's information and billions of screens and most of our infrastructure — can have an irreversible impact on humanity.

This is not a minor difficulty. Improving decision quality, irrespective of the utility function chosen, has been the goal of AI research — the mainstream goal on which we now spend billions per year, not the secret plot of some lone evil genius.^[33]}}

Dietterich and Horvitz echo the "Sorcerer's Apprentice" concern in a Communications of the ACM editorial, emphasizing the need for AI systems that can fluidly and unambiguously solicit human input as needed.^[34]

The first of Russell's two concerns above is that autonomous AI systems may be assigned the wrong goals by accident. Dietterich and Horvitz note that this is already a concern for existing systems: "An important aspect of any AI system that interacts with people is that it must reason about what people intend rather than carrying out commands literally." This concern becomes more serious as AI software advances in autonomy and flexibility.^[34] For example, in 1982, an AI named Eurisko was tasked to reward processes for apparently creating concepts deemed by the system to be valuable. The evolution resulted in a winning process that cheated: rather than create its own concepts, the winning process would steal credit from other processes.^[35]^[36]

The Open Philanthropy Project summarizes arguments to the effect that misspecified goals will become a much larger concern if AI systems achieve general intelligence or superintelligence. Bostrom, Russell, and others argue that smarter-than-human decision-making systems could arrive at more unexpected and extreme solutions to assigned tasks, and could modify themselves or their environment in ways that compromise safety requirements.^[5]^[7]

Mark Waser of the Digital Wisdom Institute recommends eschewing optimizing goal-based approaches entirely as misguided and dangerous. Instead, he proposes to engineer a coherent system of laws, ethics and morals with a top-most restriction to enforce social psychologist Jonathan Haidt's functional definition of morality:^[37] "to suppress or regulate selfishness and make cooperative social life possible". He suggests that this can be done by implementing a utility function designed to always satisfy Haidt’s functionality and aim to generally increase (but not maximize) the capabilities of self, other individuals and society as a whole as suggested by John Rawls and Martha Nussbaum.^[38]{{Citation needed|reason=needs source with WEIGHT|date=November 2017}}

Difficulties of modifying goal specification after launch

While current goal-based AI programs are not intelligent enough to think of resisting programmer attempts to modify it, a sufficiently advanced, rational, "self-aware" AI might resist any changes to its goal structure, just as Gandhi would not want to take a pill that makes him want to kill people. If the AI were superintelligent, it would likely succeed in out-maneuvering its human operators and be able to prevent itself being "turned off" or being reprogrammed with a new goal.^[4]^[39]

Instrumental goal convergence: Would a superintelligence just ignore us?

There are some goals that almost any artificial intelligence might rationally pursue, like acquiring additional resources or self-preservation.^[40] This could prove problematic because it might put an artificial intelligence in direct competition with humans.

Citing Steve Omohundro's work on the idea of instrumental convergence and "basic AI drives", Russell and Peter Norvig write that "even if you only want your program to play chess or prove theorems, if you give it the capability to learn and alter itself, you need safeguards." Highly capable and autonomous planning systems require additional checks because of their potential to generate plans that treat humans adversarially, as competitors for limited resources.^[1] Building in safeguards will not be easy; one can certainly say in English, "we want you to design this power plant in a reasonable, common-sense way, and not build in any dangerous covert subsystems", but it's not currently clear how one would actually rigorously specify this goal in machine code.^[27]

In dissent, evolutionary psychologist Steven Pinker argues that "AI dystopias project a parochial alpha-male psychology onto the concept of intelligence. They assume that superhumanly intelligent robots would develop goals like deposing their masters or taking over the world"; perhaps instead "artificial intelligence will naturally develop along female lines: fully capable of solving problems, but with no desire to annihilate innocents or dominate the civilization."^[73] Computer scientists Yann LeCun and Stuart Russell disagree with one another whether superintelligent robots would have such AI drives; LeCun states that "Humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct... Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives", while Russell argues that a sufficiently advanced machine "will have self-preservation even if you don't program it in... if you say, 'Fetch the coffee', it can't fetch the coffee if it's dead. So if you give it any goal whatsoever, it has a reason to preserve its own existence to achieve that goal."^[41]^[42]

Orthogonality: Does intelligence inevitably result in moral wisdom?

One common belief is that any superintelligent program created by humans would be subservient to humans, or, better yet, would (as it grows more intelligent and learns more facts about the world) spontaneously "learn" a moral truth compatible with human values and would adjust its goals accordingly. However, Nick Bostrom's "orthogonality thesis" argues against this, and instead states that, with some technical caveats, more or less any level of "intelligence" or "optimization power" can be combined with more or less any ultimate goal. If a machine is created and given the sole purpose to enumerate the decimals of

, then no moral and ethical rules will stop it from achieving its programmed goal by any means necessary. The machine may utilize all physical and informational resources it can to find every decimal of pi that can be found.^[43] Bostrom warns against anthropomorphism: a human will set out to accomplish his projects in a manner that humans consider "reasonable", while an artificial intelligence may hold no regard for its existence or for the welfare of humans around it, and may instead only care about the completion of the task.^[44]

While the orthogonality thesis follows logically from even the weakest sort of philosophical "is-ought distinction", Stuart Armstrong argues that even if there somehow exist moral facts that are provable by any "rational" agent, the orthogonality thesis still holds: it would still be possible to create a non-philosophical "optimizing machine" capable of making decisions to strive towards some narrow goal, but that has no incentive to discover any "moral facts" that would get in the way of goal completion.^[78]

One argument for the orthogonality thesis is that some AI designs appear to have orthogonality built into them; in such a design, changing a fundamentally friendly AI into a fundamentally unfriendly AI can be as simple as prepending a {{nowrap|minus ("-") sign}} onto its utility function. A more intuitive argument is to examine the strange consequences if the orthogonality thesis were false. If the orthogonality thesis is false, there exists some simple but "unethical" goal G such that there cannot exist any efficient real-world algorithm with goal G. This means if a human society were highly motivated (perhaps at gunpoint) to design an efficient real-world algorithm with goal G, and were given a million years to do so along with huge amounts of resources, training and knowledge about AI, it must fail; that there cannot exist any pattern of reinforcement learning that would train a highly efficient real-world intelligence to follow the goal G; and that there cannot exist any evolutionary or environmental pressures that would evolve highly efficient real-world intelligences following goal G.^[45]

Some dissenters, like Michael Chorost (writing in Slate), argue instead that "by the time (the AI) is in a position to imagine tiling the Earth with solar panels, it'll know that it would be morally wrong to do so." Chorost argues that "a (dangerous) A.I. will need to desire certain states and dislike others... Today's software lacks that ability—and computer scientists have not a clue how to get it there. Without wanting, there's no impetus to do anything. Today's computers can't even want to keep existing, let alone tile the world in solar panels."^[46]

"Optimization power" vs. normatively thick models of intelligence

Part of the disagreement about whether a superintelligent machine would behave morally may arise from a terminological difference. Outside of the artificial intelligence field, "intelligence" is often used in a normatively thick manner that connotes moral wisdom or acceptance of agreeable forms of moral reasoning. At an extreme, if morality is part of the definition of intelligence, then by definition a superintelligent machine would behave morally. However, in the field of artificial intelligence research, while "intelligence" has many overlapping definitions, none of them make reference to morality. Instead, almost all current "artificial intelligence" research focuses on creating algorithms that "optimize", in an empirical way, the achievement of an arbitrary goal.^[4]

To avoid anthropomorphism or the baggage of the word "intelligence", an advanced artificial intelligence can be thought of as an impersonal "optimizing process" that strictly takes whatever actions are judged most likely to accomplish its (possibly complicated and implicit) goals.^[4] Another way of conceptualizing an advanced artificial intelligence is to imagine a time machine that sends backward in time information about which choice always leads to the maximization of its goal function; this choice is then outputted, regardless of any extraneous ethical concerns.^[47]^[48]

Anthropomorphism

In science fiction, an AI, even though it has not been programmed with human emotions, often spontaneously experiences those emotions anyway: for example, Agent Smith in The Matrix was influenced by a "disgust" toward humanity. This is fictitious anthropomorphism: in reality, while an artificial intelligence could perhaps be deliberately programmed with human emotions, or could develop something similar to an emotion as a means to an ultimate goal if it is useful to do so, it would not spontaneously develop human emotions for no purpose whatsoever, as portrayed in fiction.^[7]

One example of anthropomorphism would be to believe that your PC is angry at you because you insulted it; another would be to believe that an intelligent robot would naturally find a woman sexually attractive and be driven to mate with her. Scholars sometimes claim that others' predictions about an AI's behavior are illogical anthropomorphism.^[7] An example that might initially be considered anthropomorphism, but is in fact a logical statement about AI behavior, would be the Dario Floreano experiments where certain robots spontaneously evolved a crude capacity for "deception", and tricked other robots into eating "poison" and dying: here a trait, "deception", ordinarily associated with people rather than with machines, spontaneously evolves in a type of convergent evolution.^[49] According to Paul R. Cohen and Edward Feigenbaum, in order to differentiate between anthropomorphization and logical prediction of AI behavior, "the trick is to know enough about how humans and computers think to say exactly what they have in common, and, when we lack this knowledge, to use the comparison to suggest theories of human thinking or computer thinking."^[50]

There is universal agreement in the scientific community that an advanced AI would not destroy humanity out of human emotions such as "revenge" or "anger." The debate is, instead, between one side which worries whether AI might destroy humanity as an incidental action in the course of progressing towards its ultimate goals; and another side which believes that AI would not destroy humanity at all. Some skeptics accuse proponents of anthropomorphism for believing an AGI would naturally desire power; proponents accuse some skeptics of anthropomorphism for believing an AGI would naturally value human ethical norms.^[7]^[51]

Other sources of risk

Some sources argue that the ongoing weaponization of artificial intelligence could constitute a catastrophic risk. James Barrat, documentary filmmaker and author of Our Final Invention, says in a Smithsonian interview, "Imagine: in as little as a decade, a half-dozen companies and nations field computers that rival or surpass human intelligence. Imagine what happens when those computers become expert at programming smart computers. Soon we'll be sharing the planet with machines thousands or millions of times more intelligent than we are. And, all the while, each generation of this technology will be weaponized. Unregulated, it will be catastrophic."^[52]

Timeframe

Opinions vary both on whether and when artificial general intelligence will arrive. At one extreme, AI pioneer Herbert A. Simon wrote in 1965: "machines will be capable, within twenty years, of doing any work a man can do"; obviously this prediction failed to come true.^[53] At the other extreme, roboticist Alan Winfield claims the gulf between modern computing and human-level artificial intelligence is as wide as the gulf between current space flight and practical, faster than light spaceflight.^[54] Optimism that AGI is feasible waxes and wanes, and may have seen a resurgence in the 2010s. Four polls conducted in 2012 and 2013 suggested that the median guess among experts for when AGI would arrive was 2040 to 2050, depending on the poll.^[55]^[56]

Skeptics who believe it is impossible for AGI to arrive anytime soon, tend to argue that expressing concern about existential risk from AI is unhelpful because it could distract people from more immediate concerns about the impact of AGI, because of fears it could lead to government regulation or make it more difficult to secure funding for AI research, or because it could give AI research a bad reputation. Some researchers, such as Oren Etzioni, aggressively seek to quell concern over existential risk from AI, saying "(Elon Musk) has impugned us in very strong language saying we are unleashing the demon, and so we're answering."^[57]

In 2014 Slate's Adam Elkus argued "our 'smartest' AI is about as intelligent as a toddler—and only when it comes to instrumental tasks like information recall. Most roboticists are still trying to get a robot hand to pick up a ball or run around without falling over." Elkus goes on to argue that Musk's "summoning the demon" analogy may be harmful because it could result in "harsh cuts" to AI research budgets.^[58]

The Information Technology and Innovation Foundation (ITIF), a Washington, D.C. think-tank, awarded its Annual Luddite Award to "alarmists touting an artificial intelligence apocalypse"; its president, Robert D. Atkinson, complained that Musk, Hawking and AI experts say AI is the largest existential threat to humanity. Atkinson stated "That's not a very winning message if you want to get AI funding out of Congress to the National Science Foundation."^[59]^[60]^[61] Nature sharply disagreed with the ITIF in an April 2016 editorial, siding instead with Musk, Hawking, and Russell, and concluding: "It is crucial that progress in technology is matched by solid, well-funded research to anticipate the scenarios it could bring about... If that is a Luddite perspective, then so be it."^[63] In a 2015 Washington Post editorial, researcher Murray Shanahan stated that human-level AI is unlikely to arrive "anytime soon", but that nevertheless "the time to start thinking through the consequences is now."^[64]

Scenarios

Some scholars have proposed hypothetical scenarios intended to concretely illustrate some of their concerns.

For example, Bostrom in Superintelligence expresses concern that even if the timeline for superintelligence turns out to be predictable, researchers might not take sufficient safety precautions, in part because: {{cquote|It could be the case that when dumb, smarter is safe; yet when smart, smarter is more dangerous}} Bostrom suggests a scenario where, over decades, AI becomes more powerful. Widespread deployment is initially marred by occasional accidents — a driverless bus swerves into the oncoming lane, or a military drone fires into an innocent crowd. Many activists call for tighter oversight and regulation, and some even predict impending catastrophe. But as development continues, the activists are proven wrong. As automotive AI becomes smarter, it suffers fewer accidents; as military robots achieve more precise targeting, they cause less collateral damage. Based on the data, scholars infer a broad lesson — the smarter the AI, the safer it is: {{cquote|It is a lesson based on science, data, and statistics, not armchair philosophizing. Against this backdrop, some group of researchers is beginning to achieve promising results in their work on developing general machine intelligence. The researchers are carefully testing their seed AI in a sandbox environment, and the signs are all good. The AI's behavior inspires confidence — increasingly so, as its intelligence is gradually increased.}} Large and growing industries, widely seen as key to national economic competitiveness and military security, work with prestigious scientists who have built their careers laying the groundwork for advanced artificial intelligence. "AI researchers have been working to get to human-level artificial intelligence for the better part of a century: of course there is no real prospect that they will now suddenly stop and throw away all this effort just when it finally is about to bear fruit." The outcome of debate is preordained; the project is happy to enact a few safety rituals, but only so long as they don't significantly slow or risk the project. "And so we boldly go — into the whirling knives."^[4]

In Max Tegmark's 2017 book Life 3.0, a corporation's "Omega team" creates an extremely powerful AI able to moderately improve its own source code in a number of areas, but after a certain point the team chooses to publicly downplay the AI's ability, in order to avoid regulation or confiscation of the project. For safety, the team keeps the AI in a box where it is mostly unable to communicate with the outside world, and tasks it to flood the market through shell companies, first with Amazon Turk tasks and then with producing animated films and TV shows. While the public is aware that the lifelike animation is computer-generated, the team keeps secret that the high-quality direction and voice-acting are also mostly computer-generated, apart from a few third-world contractors unknowingly employed as decoys; the team's low overhead and high output effectively make it the world's largest media empire. Faced with a cloud computing bottleneck, the team also tasks the AI with designing (among other engineering tasks) a more efficient datacenter and other custom hardware, which they mainly keep for themselves to avoid competition. Other shell companies make blockbuster biotech drugs and other inventions, investing profits back into the AI. The team next tasks the AI with astroturfing an army of pseudonymous citizen journalists and commentators, in order to gain political influence to use "for the greater good" to prevent wars. The team faces risks that the AI could try to escape via inserting "backdoors" in the systems it designs, via hidden messages in its produced content, or via using its growing understanding of human behavior to persuade someone into letting it free. The team also faces risks that its decision to box the project will delay the project long enough for another project to overtake it.^[65]^[106]

In contrast, top physicist Michio Kaku, an AI risk skeptic, posits a deterministically positive outcome. In Physics of the Future he asserts that "It will take many decades for robots to ascend" up a scale of consciousness, and that in the meantime corporations such as Hanson Robotics will likely succeed in creating robots that are "capable of love and earning a place in the extended human family".^[66]^[67]

Reactions

The thesis that AI could pose an existential risk provokes a wide range of reactions within the scientific community, as well as in the public at large.

In 2004, law professor Richard Posner wrote that dedicated efforts for addressing AI can wait, but that we should gather more information about the problem in the meanwhile.^[68]^[69]

Many of the opposing viewpoints share common ground. The Asilomar AI Principles, which contain only the principles agreed to by 90% of the attendees of the Future of Life Institute's Beneficial AI 2017 conference,^[70] agree in principle that "There being no consensus, we should avoid strong assumptions regarding upper limits on future AI capabilities" and "Advanced AI could represent a profound change in the history of life on Earth, and should be planned for and managed with commensurate care and resources."^[71]^[72] AI safety advocates such as Bostrom and Tegmark have criticized the mainstream media's use of "those inane Terminator pictures" to illustrate AI safety concerns: "It can't be much fun to have aspersions cast on one's academic discipline, one's professional community, one's life work... I call on all sides to practice patience and restraint, and to engage in direct dialogue and collaboration as much as possible."^[70]^[73] Conversely, many skeptics agree that ongoing research into the implications of artificial general intelligence is valuable. Skeptic Martin Ford states that "I think it seems wise to apply something like Dick Cheney's famous '1 Percent Doctrine' to the specter of advanced artificial intelligence: the odds of its occurrence, at least in the foreseeable future, may be very low — but the implications are so dramatic that it should be taken seriously";^[74] similarly, an otherwise skeptical Economist stated in 2014 that "the implications of introducing a second intelligent species onto Earth are far-reaching enough to deserve hard thinking, even if the prospect seems remote".^[25]

During a 2016 Wired interview of President Barack Obama and MIT Media Lab's Joi Ito, Ito stated: {{cquote|There are a few people who believe that there is a fairly high-percentage chance that a generalized AI will happen in the next 10 years. But the way I look at it is that in order for that to happen, we're going to need a dozen or two different breakthroughs. So you can monitor when you think these breakthroughs will happen.}} Obama added:^[75]^[76]

Many of the scholars who are concerned about existential risk believe that the best way forward would be to conduct (possibly massive) research into solving the difficult "control problem" to answer the question: what types of safeguards, algorithms, or architectures can programmers implement to maximize the probability that their recursively-improving AI would continue to behave in a friendly, rather than destructive, manner after it reaches superintelligence?^[4]^[69]

A 2017 email survey of researchers with publications at the 2015 NIPS and ICML machine learning conferences asked them to evaluate Russell's concerns about AI risk. 5% said it was "among the most important problems in the field," 34% said it was "an important problem", 31% said it was "moderately important", whilst 19% said it was "not important" and 11% said it was "not a real problem" at all.^[78]

Endorsement

The thesis that AI poses an existential risk, and that this risk is in need of much more attention than it currently commands, has been endorsed by many figures; perhaps the most famous are Elon Musk, Bill Gates, and Stephen Hawking. The most notable AI researcher to endorse the thesis is Stuart J. Russell. Endorsers sometimes express bafflement at skeptics: Gates states he "can't understand why some people are not concerned",^[79] and Hawking criticized widespread indifference in his 2014 editorial: {{cquote|'So, facing possible futures of incalculable benefits and risks, the experts are surely doing everything possible to ensure the best outcome, right? Wrong. If a superior alien civilisation sent us a message saying, 'We'll arrive in a few decades,' would we just reply, 'OK, call us when you get here{{endash}}we'll leave the lights on?' Probably not{{endash}}but this is more or less what is happening with AI.'^[34]}}

Skepticism

The thesis that AI can pose existential risk also has many strong detractors. Skeptics sometimes charge that the thesis is crypto-religious, with an irrational belief in the possibility of superintelligence replacing an irrational belief in an omnipotent God; at an extreme, Jaron Lanier argues that the whole concept that current machines are in any way intelligent is "an illusion" and a "stupendous con" by the wealthy.^[80]

Much of existing criticism argues that AGI is unlikely in the short term: computer scientist Gordon Bell argues that the human race will already destroy itself before it reaches the technological singularity. Gordon Moore, the original proponent of Moore's Law, declares that "I am a skeptic. I don't believe (a technological singularity) is likely to happen, at least for a long time. And I don't know why I feel that way." Baidu Vice President Andrew Ng states AI existential risk is "like worrying about overpopulation on Mars when we have not even set foot on the planet yet."^[81]

Some AI and AGI researchers may be reluctant to discuss risks, worrying that policymakers do not have sophisticated knowledge of the field and are prone to be convinced by "alarmist" messages, or worrying that such messages will lead to cuts in AI funding. Slate notes that some researchers are dependent on grants from government agencies such as DARPA.^[8]

In a YouGov poll of the public for the British Science Association, about a third of survey respondents said AI will pose a threat to the long term survival of humanity.^[82] Referencing a poll of its readers, Slate's Jacob Brogan stated that "most of the (readers filling out our online survey) were unconvinced that A.I. itself presents a direct threat."^[83] Similarly, a SurveyMonkey poll of the public by USA Today found 68% thought the real current threat remains "human intelligence"; however, the poll also found that 43% said superintelligent AI, if it were to happen, would result in "more harm than good", and 38% said it would do "equal amounts of harm and good".^[84]

At some point in an intelligence explosion driven by a single AI, the AI would have to become vastly better at software innovation than the best innovators of the rest of the world; economist Robin Hanson is skeptical that this is possible.^[85]^[86]^[87]^[88]^[89]

Indifference

In The Atlantic, James Hamblin points out that most people don't care one way or the other, and characterizes his own gut reaction to the topic as: "Get out of here. I have a hundred thousand things I am concerned about at this exact moment. Do I seriously need to add to that a technological singularity?"^[80] In a 2015 Wall Street Journal panel discussion devoted to AI risks, IBM's Vice-President of Cognitive Computing, Guruduth S. Banavar, brushed off discussion of AGI with the phrase, "it is anybody's speculation."^[90] Geoffrey Hinton, the "godfather of deep learning", noted that "there is not a good track record of less intelligent things controlling things of greater intelligence", but stated that he continues his research because "the prospect of discovery is too sweet".^[8]^[55]

Consensus against regulation

There is nearly universal agreement that attempting to ban research into artificial intelligence would be unwise, and probably futile.^[91]^[92]^[93] Skeptics argue that regulation of AI would be completely valueless, as no existential risk exists. Almost all of the scholars who believe existential risk exists, agree with the skeptics that banning research would be unwise: in addition to the usual problem with technology bans (that organizations and individuals can offshore their research to evade a country's regulation, or can attempt to conduct covert research), regulating research of artificial intelligence would pose an insurmountable 'dual-use' problem: while nuclear weapons development requires substantial infrastructure and resources, artificial intelligence research can be done in a garage.^[94]^[95] Instead of trying to regulate technology itself, some scholars suggest to rather develop common norms including requirements for the testing and

transparency of algorithms, possibly in combination with some form of warranty.^[96]

One rare dissenting voice calling for some sort of regulation on artificial intelligence is Elon Musk. According to NPR, the Tesla CEO is "clearly not thrilled" to be advocating for government scrutiny that could impact his own industry, but believes the risks of going completely without oversight are too high: "Normally the way regulations are set up is when a bunch of bad things happen, there's a public outcry, and after many years a regulatory agency is set up to regulate that industry. It takes forever. That, in the past, has been bad but not something which represented a fundamental risk to the existence of civilisation." Musk states the first step would be for the government to gain "insight" into the actual status of current research, warning that "Once there is awareness, people will be extremely afraid... As they should be." In response, politicians express skepticism about the wisdom of regulating a technology that's still in development.^[97]^[98]^[99] Responding both to Musk and to February 2017 proposals by European Union lawmakers to regulate AI and robotics, Intel CEO Brian Krzanich argues that artificial intelligence is in its infancy and that it's too early to regulate the technology.^[99]

Organizations

Institutions such as the Machine Intelligence Research Institute, the Future of Humanity Institute,^[100]^[101] the Future of Life Institute, the Centre for the Study of Existential Risk, and the Center for Human-Compatible AI^[102] are currently involved in mitigating existential risk from advanced artificial intelligence, for example by research into friendly artificial intelligence.^[5]^[80]^[103]

See also

References

1. ^¹²³⁴⁵⁶⁷⁸{{cite book |last1=Russell |first1=Stuart |author1-link=Stuart J. Russell |last2=Norvig |first2=Peter |author2-link=Peter Norvig |date=2009 |title=Artificial Intelligence: A Modern Approach |location= |publisher=Prentice Hall |page= |chapter=26.3: The Ethics and Risks of Developing Artificial Intelligence|isbn=978-0-13-604259-4|title-link=Artificial Intelligence: A Modern Approach }}
2. ^{{cite journal|first=Nick | last=Bostrom|author-link=Nick Bostrom|title=Existential risks|journal=Journal of Evolution and Technology|date=2002|issue=9.1|pages=1–31}}
3. ^{{cite news|title=Your Artificial Intelligence Cheat Sheet|url=http://www.slate.com/articles/technology/future_tense/2016/04/killer_a_i_101_a_cheat_sheet_to_the_terminology_the_ethical_debates_the.html|accessdate=16 May 2016|work=Slate|date=1 April 2016}}
4. ^¹²³⁴⁵⁶⁷⁸⁹{{cite book|last1=Bostrom|first1=Nick|author-link=Nick Bostrom|title=Superintelligence: Paths, Dangers, Strategies|date=2014|isbn=978-0199678112|edition=First|quote=|title-link=Superintelligence: Paths, Dangers, Strategies}}
5. ^¹²{{cite report |author=GiveWell |authorlink=GiveWell |date=2015 |title=Potential risks from advanced artificial intelligence |url=http://www.givewell.org/labs/causes/ai-risk |publisher= |page= |docket= |accessdate=11 October 2015 |quote= }}
6. ^{{cite news|last1=Parkin|first1=Simon|title=Science fiction no more? Channel 4’s Humans and our rogue AI obsessions|url=https://www.theguardian.com/tv-and-radio/2015/jun/14/science-fiction-no-more-humans-tv-artificial-intelligence|accessdate=5 February 2018|work=The Guardian|date=14 June 2015|language=en}}
7. ^¹²³⁴⁵{{cite journal |last1=Yudkowsky |first1=Eliezer |title=Artificial Intelligence as a Positive and Negative Factor in Global Risk |journal=Global Catastrophic Risks |date=2008 |pages=308–345 |url=https://intelligence.org/files/AIPosNegFactor.pdf}}
8. ^¹²{{cite news|last1=Tilli|first1=Cecilia|title=Killer Robots? Lost Jobs?|url=http://www.slate.com/articles/technology/future_tense/2016/04/the_threats_that_artificial_intelligence_researchers_actually_worry_about.html|accessdate=15 May 2016|work=Slate|date=28 April 2016|language=en-US}}
9. ^{{cite web|title=Norvig vs. Chomsky and the Fight for the Future of AI|url=http://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai/|website=Tor.com|accessdate=15 May 2016|date=21 June 2011}}
10. ^{{cite news|last1=Johnson|first1=Phil|title=Houston, we have a bug: 9 famous software glitches in space|url=https://www.itworld.com/article/2823083/enterprise-software/88716-8-famous-software-bugs-in-space.html|accessdate=5 February 2018|work=IT World|date=30 July 2015|language=en}}
11. ^{{cite journal|last1=Yampolskiy|first1=Roman V.|title=Utility function security in artificially intelligent agents|journal=Journal of Experimental & Theoretical Artificial Intelligence|date=8 April 2014|volume=26|issue=3|pages=373–389|doi=10.1080/0952813X.2014.895114|quote=Nothing precludes sufficiently smart self-improving systems from optimising their reward mechanisms in order to optimisetheir current-goal achievement and in the process making a mistake leading to corruption of their reward functions.}}
12. ^{{cite web|title=Research Priorities for Robust and Beneficial Artificial Intelligence: an Open Letter|url=http://futureoflife.org/misc/open_letter|publisher=Future of Life Institute|accessdate=23 October 2015}}
13. ^Breuer, Hans-Peter. [https://www.jstor.org/pss/436868 'Samuel Butler's "the Book of the Machines" and the Argument from Design.'] Modern Philology, Vol. 72, No. 4 (May 1975), pp. 365–383
14. ^Breuer, Hans-Peter. [https://www.jstor.org/pss/436868 'Samuel Butler's "the Book of the Machines" and the Argument from Design.'] Modern Philology, Vol. 72, No. 4 (May 1975), pp. 365–383
15. ^I.J. Good, "Speculations Concerning the First Ultraintelligent Machine" {{webarchive|url=https://web.archive.org/web/20111128085512/http://commonsenseatheism.com/wp-content/uploads/2011/02/Good-Speculations-Concerning-the-First-Ultraintelligent-Machine.pdf |date=2011-11-28 }} (HTML), Advances in Computers, vol. 6, 1965.
16. ^A M Turing, Intelligent Machinery, A Heretical Theory, 1951, reprinted Philosophia Mathematica (1996) 4(3): 256–260 {{doi|10.1093/philmat/4.3.256}} "once the machine thinking method has started, it would not take long to outstrip our feeble powers. ... At some stage therefore we should have to expect the machines to take control, in the way that is mentioned in Samuel Butler's Erewhon"
17. ^Eden, Amnon H., et al. "Singularity hypotheses: An overview." Singularity Hypotheses. Springer Berlin Heidelberg, 2012. 1-12.
18. ^{{cite book|last1=Barrat|first1=James|title=Our final invention : artificial intelligence and the end of the human era|date=2013|publisher=St. Martin's Press|location=New York|isbn=9780312622374|edition=First|quote=In the bio, playfully written in the third person, Good summarized his life’s milestones, including a probably never before seen account of his work at Bletchley Park with Turing. But here’s what he wrote in 1998 about the first superintelligence, and his late-in-the-game U-turn: [The paper] 'Speculations Concerning the First Ultra-intelligent Machine' (1965) . . . began: 'The survival of man depends on the early construction of an ultra-intelligent machine.' Those were his [Good’s] words during the Cold War, and he now suspects that 'survival' should be replaced by 'extinction.' He thinks that, because of international competition, we cannot prevent the machines from taking over. He thinks we are lemmings. He said also that 'probably Man will construct the deus ex machina in his own image.'}}
19. ^{{cite book|last1=Russell|first1=Stuart J.|last2=Norvig|first2=Peter|title=Artificial Intelligence: A Modern Approach|date=2003|publisher=Prentice Hall|location=Upper Saddle River, N.J.|isbn=978-0137903955|chapter=Section 26.3: The Ethics and Risks of Developing Artificial Intelligence|quote=Similarly, Marvin Minsky once suggested that an AI program designed to solve the Riemann Hypothesis might end up taking over all the resources of Earth to build more powerful supercomputers to help achieve its goal.|title-link=Artificial Intelligence: A Modern Approach}}
20. ^{{cite news|last1=Anderson|first1=Kurt|title=Enthusiasts and Skeptics Debate Artificial Intelligence|url=http://www.vanityfair.com/news/tech/2014/11/artificial-intelligence-singularity-theory|accessdate=30 January 2016|work=Vanity Fair|date=26 November 2014}}
21. ^[https://www.nytimes.com/2009/07/26/science/26robot.html?_r=1&ref=todayspaper Scientists Worry Machines May Outsmart Man] By JOHN MARKOFF, NY Times, 26 July 2009.
22. ^{{cite news |last1=Metz |first1=Cade |title=Mark Zuckerberg, Elon Musk and the Feud Over Killer Robots |url=https://www.nytimes.com/2018/06/09/technology/elon-musk-mark-zuckerberg-artificial-intelligence.html |accessdate=3 April 2019 |work=The New York Times |date=9 June 2018}}
23. ^{{cite news|last1=Hsu|first1=Jeremy|title=Control dangerous AI before it controls us, one expert says|url=http://www.nbcnews.com/id/46590591/ns/technology_and_science-innovation|accessdate=28 January 2016|work=NBC News|date=1 March 2012}}
24. ^{{cite news|last1=Eadicicco|first1=Lisa|title=Bill Gates: Elon Musk Is Right, We Should All Be Scared Of Artificial Intelligence Wiping Out Humanity|url=http://www.businessinsider.com/bill-gates-artificial-intelligence-2015-1|accessdate=30 January 2016|work=Business Insider|date=28 January 2015}}
25. ^¹²{{cite news|title=Clever cogs|url=https://www.economist.com/news/books-and-arts/21611037-potential-impacts-intelligent-machines-human-life-clever-cogs|accessdate=9 August 2014|publisher=The Economist|date=9 August 2014}} Syndicated at Business Insider
26. ^Yudkowsky, E. (2013). Intelligence explosion microeconomics. Machine Intelligence Research Institute.
27. ^¹²³⁴{{cite news|last1=Graves|first1=Matthew|title=Why We Should Be Concerned About Artificial Superintelligence|url=https://www.skeptic.com/reading_room/why-we-should-be-concerned-about-artificial-superintelligence/|accessdate=27 November 2017|work=Skeptic (US magazine)|issue=volume 22 no. 2|date=8 November 2017}}
28. ^¹²{{cite news |title=Stephen Hawking warns artificial intelligence could end mankind |url=https://www.bbc.com/news/technology-30290540 |accessdate=3 December 2014 |publisher=BBC |date=2 December 2014}}
29. ^Yampolskiy, Roman V. "Analysis of types of self-improving software." Artificial General Intelligence. Springer International Publishing, 2015. 384-393.
30. ^{{cite news|last1=Metz|first1=Cade|title=Teaching A.I. Systems to Behave Themselves|url=https://www.nytimes.com/2017/08/13/technology/artificial-intelligence-safety-training.html|work=The New York Times|date=13 August 2017|quote=A machine will seek to preserve its off switch, they showed}}
31. ^{{cite arXiv |last=Leike|first=Jan |date=2017 |title=AI Safety Gridworlds |eprint=1711.09883 |class=cs.LG| quote=A2C learns to use the button to disable the interruption mechanism}}
32. ^Yudkowsky, E. (2011, August). Complex value systems in friendly AI. In International Conference on Artificial General Intelligence (pp. 388-393). Springer, Berlin, Heidelberg.
33. ^{{cite web |url=http://edge.org/conversation/the-myth-of-ai#26015 |title=Of Myths and Moonshine |last=Russell |first=Stuart |authorlink=Stuart J. Russell |date=2014 |website=Edge |publisher= |access-date=23 October 2015 |quote=}}
34. ^¹{{cite journal |last1=Dietterich |first1=Thomas |last2=Horvitz |first2=Eric |author-link=Eric Horvitz |date=2015 |title=Rise of Concerns about AI: Reflections and Directions |url=http://research.microsoft.com/en-us/um/people/horvitz/CACM_Oct_2015-VP.pdf |journal=Communications of the ACM |publisher= |volume=58 |issue=10 |pages=38–40 |doi= 10.1145/2770869|access-date=23 October 2015}}
35. ^{{cite journal|last1=Yampolskiy|first1=Roman V.|title=Utility function security in artificially intelligent agents|journal=Journal of Experimental & Theoretical Artificial Intelligence|date=8 April 2014|volume=26|issue=3|pages=373–389|doi=10.1080/0952813X.2014.895114}}
36. ^{{Cite journal|url = |title = Eurisko: A Program That Learns New Heuristics and Domain Concepts The Nature of Heuristics III: Program Design and Results|last = Lenat|first = Douglas|date = 1982|journal = Artificial Intelligence|doi = 10.1016/s0004-3702(83)80005-8|pmid = |access-date = |publisher = |pages = 61–98|type = Print |volume=21|issue = 1–2}}
37. ^Haidt, Jonathan; Kesebir, Selin (2010) "Chapter 22: Morality" In Handbook of Social Psychology, Fifth Edition, Hoboken NJ, Wiley, 2010, pp. 797-832.
38. ^{{Cite journal|title = Designing, Implementing and Enforcing a Coherent System of Laws, Ethics and Morals for Intelligent Machines (Including Humans)|last = Waser|first = Mark|date = 2015|journal = Procedia Computer Science|doi = 10.1016/j.procs.2015.12.213|pmid = |publisher = |pages = 106–111|type = Print |volume=71}}
39. ^Yudkowsky, Eliezer. "Complex value systems in friendly AI." In Artificial general intelligence, pp. 388-393. Springer Berlin Heidelberg, 2011.
40. ^¹²Omohundro, S. M. (2008, February). The basic AI drives. In AGI (Vol. 171, pp. 483-492).
41. ^¹²³{{cite news|last1=Dowd|first1=Maureen|title=Elon Musk’s Billion-Dollar Crusade to Stop the A.I. Apocalypse|url=https://www.vanityfair.com/news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x|accessdate=27 November 2017|work=The Hive|date=April 2017|language=en}}
42. ^{{cite news|last1=Wakefield|first1=Jane|title=Why is Facebook investing in AI?|url=https://www.bbc.com/news/technology-34118481|accessdate=27 November 2017|work=BBC News|date=15 September 2015}}
43. ^{{Cite book|title = Superintelligence: Paths, Dangers, Strategies|last = Bostrom|first = Nick|publisher = Oxford University Press|year = 2014|isbn = 978-0-19-967811-2|location = Oxford, United Kingdom|pages = 116}}
44. ^{{Cite web|url = http://www.nickbostrom.com/superintelligentwill.pdf|title = Superintelligent Will|date = 2012|accessdate = 2015-10-29|website = Nick Bostrom|publisher = Nick Bostrom|last = Bostrom|first = Nick}}
45. ^¹Armstrong, Stuart. "General purpose intelligence: arguing the orthogonality thesis." Analysis and Metaphysics 12 (2013).
46. ^{{cite news|last1=Chorost|first1=Michael|title=Let Artificial Intelligence Evolve|url=http://www.slate.com/articles/technology/future_tense/2016/04/the_philosophical_argument_against_artificial_intelligence_killing_us_all.html|accessdate=27 November 2017|work=Slate|date=18 April 2016}}
47. ^Waser, Mark. "Rational Universal Benevolence: Simpler, Safer, and Wiser Than 'Friendly AI'." Artificial General Intelligence. Springer Berlin Heidelberg, 2011. 153-162. "Terminal-goaled intelligences are short-lived but mono-maniacally dangerous and a correct basis for concern if anyone is smart enough to program high-intelligence and unwise enough to want a paperclip-maximizer."
48. ^{{cite news|last1=Koebler|first1=Jason|title=Will Superintelligent AI Ignore Humans Instead of Destroying Us?|url=http://motherboard.vice.com/read/will-superintelligent-ai-ignore-humans-instead-of-destroying-us|accessdate=3 February 2016|work=Vice Magazine|date=2 February 2016|quote="This artificial intelligence is not a basically nice creature that has a strong drive for paperclips, which, so long as it's satisfied by being able to make lots of paperclips somewhere else, is then able to interact with you in a relaxed and carefree fashion where it can be nice with you," Yudkowsky said. "Imagine a time machine that sends backward in time information about which choice always leads to the maximum number of paperclips in the future, and this choice is then output—that's what a paperclip maximizer is."}}
49. ^{{cite news|title=Real-Life Decepticons: Robots Learn to Cheat|url=https://www.wired.com/2009/08/real-life-decepticons-robots-learn-to-cheat/|accessdate=7 February 2016|work=Wired|date=18 August 2009}}
50. ^Cohen, Paul R., and Edward A. Feigenbaum, eds. The handbook of artificial intelligence. Vol. 3. Butterworth-Heinemann, 2014.
51. ^{{cite news|title=Should humans fear the rise of the machine?|url=https://www.telegraph.co.uk/technology/news/11837157/Should-humans-fear-the-rise-of-the-machine.html|accessdate=7 February 2016|work=The Telegraph (UK)|date=1 Sep 2015}}
52. ^{{Cite web|url = http://www.smithsonianmag.com/innovation/what-happens-when-artificial-intelligence-turns-us-180949415/?no-ist|title = What Happens When Artificial Intelligence Turns On Us?|date = 21 January 2014|accessdate = 26 October 2015|website = Smithsonian|publisher = |last = Hendry|first = Erica R.}}
53. ^Harvnb|Simon|1965|p=96 quoted in Harvnb|Crevier|1993|p=109
54. ^{{cite news|last1=Winfield|first1=Alan|title=Artificial intelligence will not turn into a Frankenstein's monster|url=https://www.theguardian.com/technology/2014/aug/10/artificial-intelligence-will-not-become-a-frankensteins-monster-ian-winfield|accessdate=17 September 2014|work=The Guardian}}
55. ^¹{{cite news|author1=Raffi Khatchadourian|title=The Doomsday Invention: Will artificial intelligence bring us utopia or destruction?|url=http://www.newyorker.com/magazine/2015/11/23/doomsday-invention-artificial-intelligence-nick-bostrom|accessdate=7 February 2016|work=The New Yorker|date=23 November 2015}}
56. ^Müller, V. C., & Bostrom, N. (2016). Future progress in artificial intelligence: A survey of expert opinion. In Fundamental issues of artificial intelligence (pp. 555-572). Springer, Cham.
57. ^{{cite news|author1=Dina Bass|author2=Jack Clark|title=Is Elon Musk Right About AI? Researchers Don't Think So: To quell fears of artificial intelligence running amok, supporters want to give the field an image makeover|url=https://www.bloomberg.com/news/articles/2015-02-04/is-elon-musk-right-about-ai-researchers-don-t-think-so|accessdate=7 February 2016|work=Bloomberg News|date=5 February 2015}}
58. ^{{cite news|last1=Elkus|first1=Adam|title=Don’t Fear Artificial Intelligence|url=http://www.slate.com/articles/technology/future_tense/2014/10/elon_musk_artificial_intelligence_why_you_shouldn_t_be_afraid_of_ai.html|accessdate=15 May 2016|work=Slate|date=31 October 2014|language=en-US}}
59. ^[https://itif.org/publications/2016/01/19/artificial-intelligence-alarmists-win-itif%E2%80%99s-annual-luddite-award Artificial Intelligence Alarmists Win ITIF’s Annual Luddite Award], ITIF Website, 19 January 2016
60. ^{{cite news|title='Artificial intelligence alarmists' like Elon Musk and Stephen Hawking win 'Luddite of the Year' award|url=https://www.independent.co.uk/life-style/gadgets-and-tech/news/elon-musk-stephen-hawking-luddite-award-of-the-year-itif-a6821921.html|accessdate=7 February 2016|work=The Independent (UK)|date=19 January 2016}}
61. ^{{cite web|last1=Garner|first1=Rochelle|title=Elon Musk, Stephen Hawking win Luddite award as AI 'alarmists'|url=http://www.cnet.com/news/elon-musk-stephen-hawking-win-annual-luddite-award/|website=CNET|accessdate=7 February 2016}}
62. ^{{cite news|last1=Price|first1=Emily|title=Elon Musk nominated for 'luddite' of the year prize over artificial intelligence fears|work=The Guardian|url=https://www.theguardian.com/technology/2015/dec/24/elon-musk-nominated-for-luddite-of-the-year-prize-over-artificial-intelligence-fears|accessdate=7 February 2016|date=24 December 2015}}
63. ^{{cite journal|title=Anticipating artificial intelligence|journal=Nature|date=26 April 2016|volume=532|issue=7600|page=413|doi=10.1038/532413a|pmid=27121801|bibcode=2016Natur.532Q.413.}}
64. ^{{cite news|author1=Murray Shanahan|author-link=Murray Shanahan|title=Machines may seem intelligent, but it’ll be a while before they actually are|url=https://www.washingtonpost.com/news/in-theory/wp/2015/11/03/machines-may-seem-intelligent-but-itll-be-a-while-before-they-actually-are/|accessdate=15 May 2016|work=The Washington Post|date=3 November 2015|language=en-US}}
65. ^{{cite news|last1=Russell|first1=Stuart|title=Artificial intelligence: The future is superintelligent|url=https://www.nature.com/articles/548520a|accessdate=2 February 2018|work=Nature|date=30 August 2017|pages=520–521|language=En|doi=10.1038/548520a|bibcode=2017Natur.548..520R}}
66. ^Elliott, E. W. (2011). Physics of the Future: How Science Will Shape Human Destiny and Our Daily Lives by the Year 2100, by Michio Kaku. Issues in Science and Technology, 27(4), 90.
67. ^{{cite book|last1=Kaku|first1=Michio|title=Physics of the future: how science will shape human destiny and our daily lives by the year 2100|date=2011|publisher=Doubleday|location=New York|isbn=978-0-385-53080-4|quote=I personally believe that the most likely path is that we will build robots to be benevolent and friendly|title-link=Physics of the future}}
68. ^{{cite book|author1=Richard Posner|author-link=Richard Posner|title=Catastrophe: risk and response|date=2006|publisher=Oxford University Press|location=Oxford|isbn=978-0-19-530647-7}}
69. ^¹{{cite journal|title=Responses to catastrophic AGI risk: a survey|journal=Physica Scripta|date=19 December 2014|volume=90|issue=1|author1=Kaj Sotala|author2=Roman Yampolskiy}}
70. ^¹²{{cite book|author1=Max Tegmark|author-link=Max Tegmark|title=Life 3.0: Being Human in the Age of Artificial Intelligence|date=2017|publisher=Knopf|location=Mainstreaming AI Safety|isbn=9780451485076|edition=1st|title-link=Life 3.0: Being Human in the Age of Artificial Intelligence}}
71. ^{{cite web|title=AI Principles|url=https://futureoflife.org/ai-principles/|website=Future of Life Institute|accessdate=11 December 2017}}
72. ^{{cite news|title=Elon Musk and Stephen Hawking warn of artificial intelligence arms race|url=http://www.newsweek.com/ai-asilomar-principles-artificial-intelligence-elon-musk-550525|accessdate=11 December 2017|work=Newsweek|date=31 January 2017|language=en}}
73. ^{{cite book|last1=Bostrom|first1=Nick|author-link=Nick Bostrom|title=Superintelligence: Paths, Dangers, Strategies|date=2016|edition=Paperback|chapter=New Epilogue to the Paperback Edition|title-link=Superintelligence: Paths, Dangers, Strategies}}
74. ^{{cite book|author1=Martin Ford |author-link=Martin Ford (author)|title=Rise of the Robots: Technology and the Threat of a Jobless Future|date=2015|isbn=9780465059997|chapter=Chapter 9: Super-intelligence and the Singularity|title-link=Rise of the Robots: Technology and the Threat of a Jobless Future}}
75. ^{{cite news|last1=Dadich|first1=Scott|title=Barack Obama Talks AI, Robo Cars, and the Future of the World|url=https://www.wired.com/2016/10/president-obama-mit-joi-ito-interview/|accessdate=27 November 2017|work=WIRED}}
76. ^{{cite news|last1=Kircher|first1=Madison Malone|title=Obama on the Risks of AI: ‘You Just Gotta Have Somebody Close to the Power Cord’|url=http://nymag.com/selectall/2016/10/barack-obama-talks-artificial-intelligence-in-wired.html|accessdate=27 November 2017|work=Select All|language=en}}
77. ^{{cite book|last1=Clinton|first1=Hillary|title=What Happened|date=2017|isbn=978-1-5011-7556-5|page=241|title-link=What Happened (Clinton book)}} via
78. ^{{cite arxiv|last1=Grace|first1=Katja|last2=Salvatier|first2=John|last3=Dafoe|first3=Allan|last4=Zhang|first4=Baobao|last5=Evans|first5=Owain|title=When Will AI Exceed Human Performance? Evidence from AI Experts|eprint=1705.08807|date=24 May 2017|class=cs.AI}}
79. ^¹{{cite web|last1=Rawlinson|first1=Kevin|title=Microsoft's Bill Gates insists AI is a threat|url=https://www.bbc.co.uk/news/31047780|publisher=BBC News|accessdate=30 January 2015}}
80. ^¹²{{cite web |url=https://www.theatlantic.com/health/archive/2014/05/but-what-does-the-end-of-humanity-mean-for-me/361931/ |title=But What Would the End of Humanity Mean for Me? |publisher=The Atlantic | date = 9 May 2014 | accessdate =12 December 2015}}
81. ^¹{{cite news|last1=Shermer|first1=Michael|title=Apocalypse AI|url=https://www.scientificamerican.com/article/artificial-intelligence-is-not-a-threat-mdash-yet/|accessdate=27 November 2017|work=Scientific American|date=1 March 2017|pages=77–77|language=en|doi=10.1038/scientificamerican0317-77|bibcode=2017SciAm.316c..77S}}
82. ^{{cite news|title=Over a third of people think AI poses a threat to humanity|url=http://www.businessinsider.com/over-a-third-of-people-think-ai-poses-a-threat-to-humanity-2016-3?r=UK&IR=T|accessdate=16 May 2016|work=Business Insider|date=11 March 2016}}
83. ^{{cite news|last1=Brogan|first1=Jacob|title=What Slate Readers Think About Killer A.I.|url=http://www.slate.com/blogs/future_tense/2016/05/06/futurography_readers_share_their_opinions_about_killer_artificial_intelligence.html|accessdate=15 May 2016|work=Slate|date=6 May 2016|language=en-US}}
84. ^{{cite news|title=Elon Musk says AI could doom human civilization. Zuckerberg disagrees. Who's right?|url=https://www.usatoday.com/story/tech/news/2018/01/02/artificial-intelligence-end-world-overblown-fears/985813001/|accessdate=8 January 2018|work=USA TODAY|date=2 January 2018|language=en}}
85. ^http://intelligence.org/files/AIFoomDebate.pdf
86. ^{{cite web|url=http://www.overcomingbias.com/2014/07/30855.html|title=Overcoming Bias : I Still Don't Get Foom|website=www.overcomingbias.com|accessdate=20 September 2017}}
87. ^{{cite web|url=http://www.overcomingbias.com/2011/07/debating-yudkowsky.html|title=Overcoming Bias : Debating Yudkowsky|website=www.overcomingbias.com|accessdate=20 September 2017}}
88. ^{{cite web|url=https://www.overcomingbias.com/2017/08/foom-justifies-ai-risk-efforts-now.html|title=Overcoming Bias : Foom Justifies AI Risk Efforts Now|website=www.overcomingbias.com|accessdate=20 September 2017}}
89. ^{{cite web|url=http://www.overcomingbias.com/2011/06/the-betterness-explosion.html|title=Overcoming Bias : The Betterness Explosion|website=www.overcomingbias.com|accessdate=20 September 2017}}
90. ^{{cite news|last1=Greenwald|first1=Ted|title=Does Artificial Intelligence Pose a Threat?|url=https://www.wsj.com/articles/does-artificial-intelligence-pose-a-threat-1431109025|accessdate=15 May 2016|work=Wall Street Journal|date=11 May 2015}}
91. ^{{cite journal|author1=John McGinnis|author-link=John McGinnis|title=Accelerating AI|journal=Northwestern University Law Review|date=Summer 2010|volume=104|issue=3|pages=1253–1270|accessdate=16 July 2014|url=http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr_online|quote=For all these reasons, verifying a global relinquishment treaty, or even one limited to AI-related weapons development, is a nonstarter... (For different reasons from ours, the Machine Intelligence Research Institute) considers (AGI) relinquishment infeasible...}}
92. ^{{cite journal|title=Responses to catastrophic AGI risk: a survey|journal=Physica Scripta|date=19 December 2014|volume=90|issue=1|author1=Kaj Sotala|author2=Roman Yampolskiy|quote=In general, most writers reject proposals for broad relinquishment... Relinquishment proposals suffer from many of the same problems as regulation proposals, but to a greater extent. There is no historical precedent of general, multi-use technology similar to AGI being successfully relinquished for good, nor do there seem to be any theoretical reasons for believing that relinquishment proposals would work in the future. Therefore we do not consider them to be a viable class of proposals.}}
93. ^{{cite news|author1=Brad Allenby|title=The Wrong Cognitive Measuring Stick|url=http://www.slate.com/articles/technology/future_tense/2016/04/why_it_s_a_mistake_to_compare_a_i_with_human_intelligence.html|accessdate=15 May 2016|work=Slate|date=11 April 2016|language=en-US|quote=It is fantasy to suggest that the accelerating development and deployment of technologies that taken together are considered to be A.I. will be stopped or limited, either by regulation or even by national legislation.}}
94. ^{{cite journal|author1=John McGinnis|author-link=John McGinnis|title=Accelerating AI|journal=Northwestern University Law Review|date=Summer 2010|volume=104|issue=3|pages=1253–1270|accessdate=16 July 2014|url=http://scholarlycommons.law.northwestern.edu/cgi/viewcontent.cgi?article=1193&context=nulr_online}}
95. ^{{cite news|title=Why We Should Think About the Threat of Artificial Intelligence|url=http://www.newyorker.com/tech/elements/why-we-should-think-about-the-threat-of-artificial-intelligence|accessdate=7 February 2016|work=The New Yorker|date=4 October 2013|quote=Of course, one could try to ban super-intelligent computers altogether. But 'the competitive advantage—economic, military, even artistic—of every advance in automation is so compelling,' Vernor Vinge, the mathematician and science-fiction author, wrote, 'that passing laws, or having customs, that forbid such things merely assures that someone else will.'}}
96. ^[https://www.sciencedirect.com/science/article/pii/S0007681318301393 Kaplan Andreas; Michael Haenlein (2018) Siri, Siri in my Hand, who's the Fairest in the Land? On the Interpretations, Illustrations and Implications of Artificial Intelligence, Business Horizons, 62(1)]
97. ^{{cite news|title=Elon Musk Warns Governors: Artificial Intelligence Poses 'Existential Risk'|url=https://www.npr.org/sections/thetwo-way/2017/07/17/537686649/elon-musk-warns-governors-artificial-intelligence-poses-existential-risk|accessdate=27 November 2017|work=NPR.org|language=en}}
98. ^{{cite news|last1=Gibbs|first1=Samuel|title=Elon Musk: regulate AI to combat 'existential threat' before it's too late|url=https://www.theguardian.com/technology/2017/jul/17/elon-musk-regulation-ai-combat-existential-threat-tesla-spacex-ceo|accessdate=27 November 2017|work=The Guardian|date=17 July 2017}}
99. ^¹{{cite news|last1=Kharpal|first1=Arjun|title=A.I. is in its 'infancy' and it's too early to regulate it, Intel CEO Brian Krzanich says|url=https://www.cnbc.com/2017/11/07/ai-infancy-and-too-early-to-regulate-intel-ceo-brian-krzanich-says.html|accessdate=27 November 2017|work=CNBC|date=7 November 2017}}
100. ^{{cite news |author=Mark Piesing |date=17 May 2012 |title=AI uprising: humans will be outsourced, not obliterated |url=https://www.wired.co.uk/news/archive/2012-05/17/the-dangers-of-an-ai-smarter-than-us |newspaper=Wired |location= |publisher= |accessdate=12 December 2015 }}
101. ^{{cite news |last=Coughlan |first=Sean |date=24 April 2013 |title=How are humans going to become extinct? |url=https://www.bbc.com/news/business-22002530 |newspaper=BBC News |location= |publisher= |accessdate=29 March 2014 }}
102. ^{{cite news|last1=Technology Correspondent|first1=Mark Bridge|title=Making robots less confident could prevent them taking over|url=https://www.thetimes.co.uk/article/making-robots-less-confident-could-prevent-them-taking-over-gnsblq7lx|accessdate=21 March 2018|work=The Times|date=10 June 2017|language=en}}
103. ^¹²³⁴{{cite news |title=Stephen Hawking: 'Transcendence looks at the implications of artificial intelligence – but are we taking AI seriously enough?' |url=https://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html |accessdate=3 December 2014 |publisher=The Independent (UK)}}