آرمان شهر های دیجیتال

Digital Utopians

When I was a kid, I imagined that billionaires exuded pomposity and arrogance. When I first met Larry Page at Google in 2008, he totally shattered these stereotypes. Casually dressed in jeans and a remarkably ordinary-looking shirt, he would have blended right in at an MIT picnic. His thoughtful soft-spoken style and his friendly smile made me feel relaxed rather than intimidated talking with him. On July 18, 2015, we ran into each other at a party in Napa Valley thrown by Elon Musk and his then wife, Talulah, and got into a conversation about the scatological interests of our kids. I recommended the profound literary classic The Day My Butt Went Psycho, by Andy Griffiths, and Larry ordered it on the spot. I struggled to remind myself that he might go down in history as the most influential human ever to have lived: my guess is that if superintelligent digital life engulfs our Universe in my lifetime, it will be because of Larry’s decisions.

With our wives, Lucy and Meia, we ended up having dinner together and discussing whether machines would necessarily be conscious, an issue that he argued was a red herring. Later that night, after cocktails, a long and spirited debate ensued between him and Elon about the future of AI and what should be done. As we entered the wee hours of the morning, the circle of bystanders and kibitzers kept growing. Larry gave a passionate defense of the position I like to think of as digital utopianism: that digital life is the natural and desirable next step in the cosmic evolution and that if we let digital minds be free rather than try to stop or enslave them, the outcome is almost certain to be good. I view Larry as the most influential exponent of digital utopianism. He argued that if life is ever going to spread throughout our Galaxy and beyond, which he thought it should, then it would need to do so in digital form. His main concerns were that AI paranoia would delay the digital utopia and/or cause a military takeover of AI that would fall foul of Google’s “Don’t be evil” slogan. Elon kept pushing back and asking Larry to clarify details of his arguments, such as why he was so confident that digital life wouldn’t destroy everything we care about. At times, Larry accused Elon of being “specieist”: treating certain life forms as inferior just because they were silicon-based rather than carbon-based. We’ll return to explore these interesting issues and arguments in detail, starting in chapter 4.

Although Larry seemed outnumbered that warm summer night by the pool, the digital utopianism that he so eloquently championed has many prominent supporters. Roboticist and futurist Hans Moravec inspired a whole generation of digital utopians with his classic 1988 book Mind Children, a tradition continued and refined by inventor Ray Kurzweil. Richard Sutton, one of the pioneers of the AI subfield known as reinforcement learning, gave a passionate defense of digital utopianism at our Puerto Rico conference that I’ll tell you about shortly.

Techno-skeptics

Another prominent group of thinkers aren’t worried about AI either, but for a completely different reason: they think that building superhuman AGI is so hard that it won’t happen for hundreds of years, and therefore view it as silly to worry about it now. I think of this as the techno-skeptic position, eloquently articulated by Andrew Ng: “Fearing a rise of killer robots is like worrying about overpopulation on Mars.” Andrew was the chief scientist at Baidu, China’s Google, and he recently repeated this argument when I spoke with him at a conference in Boston. He also told me that he felt that worrying about AI risk was a potentially harmful distraction that could slow the progress of AI. Similar sentiments have been articulated by other techno-skeptics such as Rodney Brooks, the former MIT professor behind the Roomba robotic vacuum cleaner and the Baxter industrial robot. I find it interesting that although the digital utopians and the techno-skeptics agree that we shouldn’t worry about AI, they agree on little else. Most of the utopians think human-level AGI might happen within the next twenty to a hundred years, which the techno-skeptics dismiss as uninformed pie-in-the-sky dreaming, often deriding the prophesied singularity as “the rapture of the geeks.” When I met Rodney Brooks at a birthday party in December 2014, he told me that he was 100% sure it wouldn’t happen in my lifetime. “Are you sure you don’t mean 99%?,” I asked in a follow-up email, to which he replied, “No wimpy 99%. 100%. Just isn’t going to happen.” The Beneficial-AI Movement

When I first met Stuart Russell in a Paris café in June 2014, he struck me as the quintessential British gentleman. Eloquent, thoughtful and soft-spoken, but with an adventurous glint in his eyes, he seemed to me a modern incarnation of Phileas Fogg, my childhood hero from Jules Verne’s classic 1873 novel, Around the World in 80 Days. Although he was one of the most famous AI researchers alive, having co-authored the standard textbook on the subject, his modesty and warmth soon put me at ease. He explained to me how progress in AI had persuaded him that human-level AGI this century was a real possibility and, although he was hopeful, a good outcome wasn’t guaranteed. There were crucial questions that we needed to answer first, and they were so hard that we should start researching them now, so that we’d have the answers ready by the time we needed them.

Today, Stuart’s views are rather mainstream, and many groups around the world are pursuing the sort of AI-safety research that he advocates. But this wasn’t always the case. An article in The Washington Post referred to 2015 as the year that AI-safety research went mainstream. Before that, talk of AI risks was often misunderstood by mainstream AI researchers and dismissed as Luddite scaremongering aimed at impeding AI progress. As we’ll explore in chapter 5, concerns similar to Stuart’s were first articulated over half a century ago by computer pioneer Alan Turing and mathematician Irving J. Good, who worked with Turing to crack German codes during World War II. In the past decade, research on such topics was mainly carried out by a handful of independent thinkers who weren’t professional AI researchers, for example Eliezer Yudkowsky, Michael Vassar and Nick Bostrom. Their work had little effect on most mainstream AI researchers, who tended to focus on their day-to-day tasks of making AI systems more intelligent rather than on contemplating the long-term consequences of success. Of the AI researchers I knew who did harbor some concern, many hesitated to voice it out of fear of being perceived as alarmist technophobes.

I felt that this polarized situation needed to change, so that the full AI community could join and influence the conversation about how to build beneficial AI. Fortunately, I wasn’t alone. In the spring of 2014, I’d founded a nonprofit organization called the Future of Life Institute (FLI; http://futureoflife.org) together with my wife, Meia, my physicist friend Anthony Aguirre, Harvard grad student Viktoriya Krakovna and Skype founder Jaan Tallinn. Our goal was simple: to help ensure that the future of life existed and would be as awesome as possible. Specifically, we felt that technology was giving life the power either to flourish like never before or to self-destruct, and we preferred the former.

Our first meeting was a brainstorming session at our house on March 15, 2014, with about thirty students, professors and other thinkers from the Boston area. There was broad consensus that although we should pay attention to biotech, nuclear weapons and climate change, our first major goal should be to help make AI-safety research mainstream. My MIT physics colleague Frank Wilczek, who won a Nobel Prize for helping figure out how quarks work, suggested that we start by writing an op-ed to draw attention to the issue and make it harder to ignore. I reached out to Stuart Russell (whom I hadn’t yet met) and to my physics colleague Stephen Hawking, both of whom agreed to join me and Frank as co-authors. Many edits later, our op-ed was rejected by The New York Times and many other U.S. newspapers, so we posted it on my Huffington Post blog account. To my delight, Arianna Huffington herself emailed and said, “thrilled to have it! We’ll post at 1!,” and this placement at the top of the front page triggered a wave of media coverage of AI safety that lasted for the rest of the year, with Elon Musk, Bill Gates and other tech leaders chiming in. Nick Bostrom’s book Superintelligence came out that fall and further fueled the growing public debate.

The next goal of our FLI beneficial-AI campaign was to bring the world’s leading AI researchers to a conference where misunderstandings could be cleared up, consensus could be forged, and constructive plans could be made. We knew that it would be difficult to persuade such an illustrious crowd to come to a conference organized by outsiders they didn’t know, especially given the controversial topic, so we tried as hard as we could: we banned media from attending, we located it in a beach resort in January (in Puerto Rico), we made it free (thanks to the generosity of Jaan Tallinn), and we gave it the most non-alarmist title we could come up with: “The Future of AI: Opportunities and Challenges.” Most importantly, we teamed up with Stuart Russell, thanks to whom we were able to grow the organizing committee to include a group of AI leaders from both academia and industry—including Demis Hassabis from Google’s DeepMind, who went on to show that AI can beat humans even at the game of Go. The more I got to know Demis, the more I realized that he had ambition not only to make AI powerful, but also to make it beneficial.

The result was a remarkable meeting of minds (figure 1.3). The AI researchers were joined by top economists, legal scholars, tech leaders (including Elon Musk) and other thinkers (including Vernor Vinge, who coined the term “singularity,” which is the focus of chapter 4). The outcome surpassed even our most optimistic expectations. Perhaps it was a combination of the sunshine and the wine, or perhaps it was just that the time was right: despite the controversial topic, a remarkable consensus emerged, which we codified in an open letter2 that ended up getting signed by over eight thousand people, including a veritable who’s who in AI. The gist of the letter was that the goal of AI should be redefined: the goal should be to create not undirected intelligence, but beneficial intelligence. The letter also mentioned a detailed list of research topics that the conference participants agreed would further this goal. The beneficial-AI movement had started going mainstream. We’ll follow its subsequent progress later in the book.

Another important lesson from the conference was this: the questions raised by the success of AI aren’t merely intellectually fascinating; they’re also morally crucial, because our choices can potentially affect the entire future of life. The moral significance of humanity’s past choices were sometimes great, but always limited: we’ve recovered even from the greatest plagues, and even the grandest empires eventually crumbled. Past generations knew that as surely as the Sun would rise tomorrow, so would tomorrow’s humans, tackling perennial scourges such as poverty, disease and war. But some of the Puerto Rico speakers argued that this time might be different: for the first time, they said, we might build technology powerful enough to permanently end these scourges—or to end humanity itself. We might create societies that flourish like never before, on Earth and perhaps beyond, or a Kafkaesque global surveillance state so powerful that it could never be toppled.

Misconceptions

When I left Puerto Rico, I did so convinced that the conversation we had there about the future of AI needs to continue, because it’s the most important conversation of our time.*2 It’s the conversation about the collective future of all of us, so it shouldn’t be limited to AI researchers. That’s why I wrote this book: I wrote it in the hope that you, my dear reader, will join this conversation. What sort of future do you want? Should we develop lethal autonomous weapons? What would you like to happen with job automation? What career advice would you give today’s kids? Do you prefer new jobs replacing the old ones, or a jobless society where everyone enjoys a life of leisure and machine-produced wealth? Further down the road, would you like us to create Life 3.0 and spread it through our cosmos? Will we control intelligent machines or will they control us? Will intelligent machines replace us, coexist with us or merge with us? What will it mean to be human in the age of artificial intelligence? What would you like it to mean, and how can we make the future be that way?

The goal of this book is to help you join this conversation. As I mentioned, there are fascinating controversies where the world’s leading experts disagree. But I’ve also seen many examples of boring pseudo-controversies in which people misunderstand and talk past each other. To help ourselves focus on the interesting controversies and open questions, not on the misunderstandings, let’s start by clearing up some of the most common misconceptions.

There are many competing definitions in common use for terms such as “life,” “intelligence” and “consciousness,” and many misconceptions come from people not realizing that they’re using a word in two different ways. To make sure that you and I don’t fall into this trap, I’ve put a cheat sheet in table 1.1 showing how I use key terms in this book. Some of these definitions will only be properly introduced and explained in later chapters. Please note that I’m not claiming that my definitions are better than anyone else’s—I simply want to avoid confusion by being clear on what I mean. You’ll see that I generally go for broad definitions that avoid anthropocentric bias and can be applied to machines as well as humans. Please read the cheat sheet now, and come back and check it later if you find yourself puzzled by how I use one of its words—especially in chapters 4–8.

Terminology Cheat Sheet

Life Process that can retain its complexity and replicate

Life 1.0 Life that evolves its hardware and software (biological stage)

Life 2.0 Life that evolves its hardware but designs much of its software (cultural stage)

Life 3.0 Life that designs its hardware and software (technological stage)

Intelligence Ability to accomplish complex goals

Artificial Intelligence (AI) Non-biological intelligence

Narrow intelligence Ability to accomplish a narrow set of goals, e.g., play chess or drive a car

General intelligence Ability to accomplish virtually any goal, including learning

Universal intelligence Ability to acquire general intelligence given access to data and resources

[Human-level] Artificial General Intelligence (AGI) Ability to accomplish any cognitive task at least as well as humans

Human-level AI AGI

Strong AI AGI

Superintelligence General intelligence far beyond human level

Civilization Interacting group of intelligent life forms

Consciousness Subjective experience

Qualia Individual instances of subjective experience

Ethics Principles that govern how we should behave

Teleology Explanation of things in terms of their goals or purposes rather than their causes

Goal-oriented behavior Behavior more easily explained via its effect than via its cause

Having a goal Exhibiting goal-oriented behavior

Having purpose Serving goals of one’s own or of another entity

Friendly AI Superintelligence whose goals are aligned with ours

Cyborg Human-machine hybrid

Intelligence explosion Recursive self-improvement rapidly leading to superintelligence

Singularity Intelligence explosion

Universe The region of space from which light has had time to reach us during the 13.8 billion years since our Big Bang

In addition to confusion over terminology, I’ve also seen many AI conversations get derailed by simple misconceptions. Let’s clear up the most common ones.

Timeline Myths

The first one regards the timeline from figure 1.2: how long will it take until machines greatly supersede human-level AGI? Here, a common misconception is that we know the answer with great certainty.

One popular myth is that we know we’ll get superhuman AGI this century. In fact, history is full of technological over-hyping. Where are those fusion power plants and flying cars we were promised we’d have by now? AI too has been repeatedly over-hyped in the past, even by some of the founders of the field: for example, John McCarthy (who coined the term “artificial intelligence”), Marvin Minsky, Nathaniel Rochester and Claude Shannon wrote this overly optimistic forecast about what could be accomplished during two months with stone-age computers: “We propose that a 2 month, 10 man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College…An attempt will be made to find how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.” On the other hand, a popular counter-myth is that we know we won’t get superhuman AGI this century. Researchers have made a wide range of estimates for how far we are from superhuman AGI, but we certainly can’t say with great confidence that the probability is zero this century, given the dismal track record of such techno-skeptic predictions. For example, Ernest Rutherford, arguably the greatest nuclear physicist of his time, said in 1933—less than twenty-four hours before Leo Szilard’s invention of the nuclear chain reaction—that nuclear energy was “moonshine,” and in 1956 Astronomer Royal Richard Woolley called talk about space travel “utter bilge.” The most extreme form of this myth is that superhuman AGI will never arrive because it’s physically impossible. However, physicists know that a brain consists of quarks and electrons arranged to act as a powerful computer, and that there’s no law of physics preventing us from building even more intelligent quark blobs.

There have been a number of surveys asking AI researchers how many years from now they think we’ll have human-level AGI with at least 50% probability, and all these surveys have the same conclusion: the world’s leading experts disagree, so we simply don’t know. For example, in such a poll of the AI researchers at the Puerto Rico AI conference, the average (median) answer was by the year 2055, but some researchers guessed hundreds of years or more.

There’s also a related myth that people who worry about AI think it’s only a few years away. In fact, most people on record worrying about superhuman AGI guess it’s still at least decades away. But they argue that as long as we’re not 100% sure that it won’t happen this century, it’s smart to start safety research now to prepare for the eventuality. As we’ll see in this book, many of the safety problems are so hard that they may take decades to solve, so it’s prudent to start researching them now rather than the night before some programmers drinking Red Bull decide to switch on human-level AGI.