Kevin Kelly, Wired, December 24, 2012
Imagine that 7 out of 10 working Americans got fired tomorrow. What would they all do?
It’s hard to believe you’d have an economy at all if you gave pink slips to more than half the labor force. But that—in slow motion—is what the industrial revolution did to the workforce of the early 19th century. Two hundred years ago, 70 percent of American workers lived on the farm. Today automation has eliminated all but 1 percent of their jobs, replacing them (and their work animals) with machines. But the displaced workers did not sit idle. Instead, automation created hundreds of millions of jobs in entirely new fields. Those who once farmed were now manning the legions of factories that churned out farm equipment, cars, and other industrial products. Since then, wave upon wave of new occupations have arrived—appliance repairman, offset printer, food chemist, photographer, web designer—each building on previous automation. Today, the vast majority of us are doing jobs that no farmer from the 1800s could have imagined.
It may be hard to believe, but before the end of this century, 70 percent of today’s occupations will likewise be replaced by automation. Yes, dear reader, even you will have your job taken away by machines. In other words, robot replacement is just a matter of time. This upheaval is being led by a second wave of automation, one that is centered on artificial cognition, cheap sensors, machine learning, and distributed smarts. This deep automation will touch all jobs, from manual labor to knowledge work.
First, machines will consolidate their gains in already-automated industries. After robots finish replacing assembly line workers, they will replace the workers in warehouses. Speedy bots able to lift 150 pounds all day long will retrieve boxes, sort them, and load them onto trucks. Fruit and vegetable picking will continue to be robotized until no humans pick outside of specialty farms. Pharmacies will feature a single pill-dispensing robot in the back while the pharmacists focus on patient consulting. Next, the more dexterous chores of cleaning in offices and schools will be taken over by late-night robots, starting with easy-to-do floors and windows and eventually getting to toilets. The highway legs of long-haul trucking routes will be driven by robots embedded in truck cabs.
All the while, robots will continue their migration into white-collar work. We already have artificial intelligence in many of our machines; we just don’t call it that. Witness one piece of software by Narrative Science (profiled in issue 20.05) that can write newspaper stories about sports games directly from the games’ stats or generate a synopsis of a company’s stock performance each day from bits of text around the web. Any job dealing with reams of paperwork will be taken over by bots, including much of medicine. Even those areas of medicine not defined by paperwork, such as surgery, are becoming increasingly robotic. The rote tasks of any information-intensive job can be automated. It doesn’t matter if you are a doctor, lawyer, architect, reporter, or even programmer: The robot takeover will be epic.
And it has already begun.
Here’s why we’re at the inflection point: Machines are acquiring smarts.
We have preconceptions about how an intelligent robot should look and act, and these can blind us to what is already happening around us. To demand that artificial intelligence be humanlike is the same flawed logic as demanding that artificial flying be birdlike, with flapping wings. Robots will think different. To see how far artificial intelligence has penetrated our lives, we need to shed the idea that they will be humanlike.
Consider Baxter, a revolutionary new workbot from Rethink Robotics. Designed by Rodney Brooks, the former MIT professor who invented the best-selling Roomba vacuum cleaner and its descendants, Baxter is an early example of a new class of industrial robots created to work alongside humans. Baxter does not look impressive. It’s got big strong arms and a flatscreen display like many industrial bots. And Baxter’s hands perform repetitive manual tasks, just as factory robots do. But it’s different in three significant ways.
First, it can look around and indicate where it is looking by shifting the cartoon eyes on its head. It can perceive humans working near it and avoid injuring them. And workers can see whether it sees them.
Previous industrial robots couldn’t do this, which means that working robots have to be physically segregated from humans. The typical factory robot is imprisoned within a chain-link fence or caged in a glass case. They are simply too dangerous to be around, because they are oblivious to others. This isolation prevents such robots from working in a small shop, where isolation is not practical. Optimally, workers should be able to get materials to and from the robot or to tweak its controls by hand throughout the workday; isolation makes that difficult. Baxter, however, is aware. Using force-feedback technology to feel if it is colliding with a person or another bot, it is courteous. You can plug it into a wall socket in your garage and easily work right next to it.
Second, anyone can train Baxter. It is not as fast, strong, or precise as other industrial robots, but it is smarter. To train the bot you simply grab its arms and guide them in the correct motions and sequence. It’s a kind of “watch me do this” routine. Baxter learns the procedure and then repeats it. Any worker is capable of this show-and-tell; you don’t even have to be literate. Previous workbots required highly educated engineers and crack programmers to write thousands of lines of code (and then debug them) in order to instruct the robot in the simplest change of task. The code has to be loaded in batch mode, i.e., in large, infrequent batches, because the robot cannot be reprogrammed while it is being used. Turns out the real cost of the typical industrial robot is not its hardware but its operation. Industrial robots cost $100,000-plus to purchase but can require four times that amount over a lifespan to program, train, and maintain. The costs pile up until the average lifetime bill for an industrial robot is half a million dollars or more.
The third difference, then, is that Baxter is cheap. Priced at $22,000, it’s in a different league compared with the $500,000 total bill of its predecessors. It is as if those established robots, with their batch-mode programming, are the mainframe computers of the robot world, and Baxter is the first PC robot. It is likely to be dismissed as a hobbyist toy, missing key features like sub-millimeter precision, and not serious enough. But as with the PC, and unlike the mainframe, the user can interact with it directly, immediately, without waiting for experts to mediate—and use it for nonserious, even frivolous things. It’s cheap enough that small-time manufacturers can afford one to package up their wares or custom paint their product or run their 3-D printing machine. Or you could staff up a factory that makes iPhones.
Baxter was invented in a century-old brick building near the Charles River in Boston. In 1895 the building was a manufacturing marvel in the very center of the new manufacturing world. It even generated its own electricity. For a hundred years the factories inside its walls changed the world around us. Now the capabilities of Baxter and the approaching cascade of superior robot workers spur Brooks to speculate on how these robots will shift manufacturing in a disruption greater than the last revolution. Looking out his office window at the former industrial neighborhood, he says, “Right now we think of manufacturing as happening in China. But as manufacturing costs sink because of robots, the costs of transportation become a far greater factor than the cost of production. Nearby will be cheap. So we’ll get this network of locally franchised factories, where most things will be made within 5 miles of where they are needed.”
That may be true of making stuff, but a lot of jobs left in the world for humans are service jobs. I ask Brooks to walk with me through a local McDonald’s and point out the jobs that his kind of robots can replace. He demurs and suggests it might be 30 years before robots will cook for us. “In a fast food place you’re not doing the same task very long. You’re always changing things on the fly, so you need special solutions. We are not trying to sell a specific solution. We are building a general-purpose machine that other workers can set up themselves and work alongside.” And once we can cowork with robots right next to us, it’s inevitable that our tasks will bleed together, and soon our old work will become theirs—and our new work will become something we can hardly imagine.
To understand how robot replacement will happen, it’s useful to break down our relationship with robots into four categories, as summed up in this chart:
The rows indicate whether robots will take over existing jobs or make new ones, and the columns indicate whether these jobs seem (at first) like jobs for humans or for machines.
Let’s begin with quadrant A: jobs humans can do but robots can do even better. Humans can weave cotton cloth with great effort, but automated looms make perfect cloth, by the mile, for a few cents. The only reason to buy handmade cloth today is because you want the imperfections humans introduce. We no longer value irregularities while traveling 70 miles per hour, though—so the fewer humans who touch our car as it is being made, the better.
And yet for more complicated chores, we still tend to believe computers and robots can’t be trusted. That’s why we’ve been slow to acknowledge how they’ve mastered some conceptual routines, in some cases even surpassing their mastery of physical routines. A computerized brain known as the autopilot can fly a 787 jet unaided, but irrationally we place human pilots in the cockpit to babysit the autopilot “just in case.” In the 1990s, computerized mortgage appraisals replaced human appraisers wholesale. Much tax preparation has gone to computers, as well as routine x-ray analysis and pretrial evidence-gathering—all once done by highly paid smart people. We’ve accepted utter reliability in robot manufacturing; soon we’ll accept it in robotic intelligence and service.
Next is quadrant B: jobs that humans can’t do but robots can. A trivial example: Humans have trouble making a single brass screw unassisted, but automation can produce a thousand exact ones per hour. Without automation, we could not make a single computer chip—a job that requires degrees of precision, control, and unwavering attention that our animal bodies don’t possess. Likewise no human, indeed no group of humans, no matter their education, can quickly search through all the web pages in the world to uncover the one page revealing the price of eggs in Katmandu yesterday. Every time you click on the search button you are employing a robot to do something we as a species are unable to do alone.
While the displacement of formerly human jobs gets all the headlines, the greatest benefits bestowed by robots and automation come from their occupation of jobs we are unable to do. We don’t have the attention span to inspect every square millimeter of every CAT scan looking for cancer cells. We don’t have the millisecond reflexes needed to inflate molten glass into the shape of a bottle. We don’t have an infallible memory to keep track of every pitch in Major League Baseball and calculate the probability of the next pitch in real time.
We aren’t giving “good jobs” to robots. Most of the time we are giving them jobs we could never do. Without them, these jobs would remain undone.
Now let’s consider quadrant C, the new jobs created by automation—including the jobs that we did not know we wanted done. This is the greatest genius of the robot takeover: With the assistance of robots and computerized intelligence, we already can do things we never imagined doing 150 years ago. We can remove a tumor in our gut through our navel, make a talking-picture video of our wedding, drive a cart on Mars, print a pattern on fabric that a friend mailed to us through the air. We are doing, and are sometimes paid for doing, a million new activities that would have dazzled and shocked the farmers of 1850. These new accomplishments are not merely chores that were difficult before. Rather they are dreams that are created chiefly by the capabilities of the machines that can do them. They are jobs the machines make up.
Before we invented automobiles, air-conditioning, flatscreen video displays, and animated cartoons, no one living in ancient Rome wished they could watch cartoons while riding to Athens in climate-controlled comfort. Two hundred years ago not a single citizen of Shanghai would have told you that they would buy a tiny slab that allowed them to talk to faraway friends before they would buy indoor plumbing. Crafty AIs embedded in first-person-shooter games have given millions of teenage boys the urge, the need, to become professional game designers—a dream that no boy in Victorian times ever had. In a very real way our inventions assign us our jobs. Each successful bit of automation generates new occupations—occupations we would not have fantasized about without the prompting of the automation.
To reiterate, the bulk of new tasks created by automation are tasks only other automation can handle. Now that we have search engines like Google, we set the servant upon a thousand new errands. Google, can you tell me where my phone is? Google, can you match the people suffering depression with the doctors selling pills? Google, can you predict when the next viral epidemic will erupt? Technology is indiscriminate this way, piling up possibilities and options for both humans and machines.
It is a safe bet that the highest-earning professions in the year 2050 will depend on automations and machines that have not been invented yet. That is, we can’t see these jobs from here, because we can’t yet see the machines and technologies that will make them possible. Robots create jobs that we did not even know we wanted done.
Finally, that leaves us with quadrant D, the jobs that only humans can do—at first. The one thing humans can do that robots can’t (at least for a long while) is to decide what it is that humans want to do. This is not a trivial trick; our desires are inspired by our previous inventions, making this a circular question.
When robots and automation do our most basic work, making it relatively easy for us to be fed, clothed, and sheltered, then we are free to ask, “What are humans for?” Industrialization did more than just extend the average human lifespan. It led a greater percentage of the population to decide that humans were meant to be ballerinas, full-time musicians, mathematicians, athletes, fashion designers, yoga masters, fan-fiction authors, and folks with one-of-a kind titles on their business cards. With the help of our machines, we could take up these roles; but of course, over time, the machines will do these as well. We’ll then be empowered to dream up yet more answers to the question “What should we do?” It will be many generations before a robot can answer that.
This postindustrial economy will keep expanding, even though most of the work is done by bots, because part of your task tomorrow will be to find, make, and complete new things to do, new things that will later become repetitive jobs for the robots. In the coming years robot-driven cars and trucks will become ubiquitous; this automation will spawn the new human occupation of trip optimizer, a person who tweaks the traffic system for optimal energy and time usage. Routine robo-surgery will necessitate the new skills of keeping machines sterile. When automatic self-tracking of all your activities becomes the normal thing to do, a new breed of professional analysts will arise to help you make sense of the data. And of course we will need a whole army of robot nannies, dedicated to keeping your personal bots up and running. Each of these new vocations will in turn be taken over by robots later.
The real revolution erupts when everyone has personal workbots, the descendants of Baxter, at their beck and call. Imagine you run a small organic farm. Your fleet of worker bots do all the weeding, pest control, and harvesting of produce, as directed by an overseer bot, embodied by a mesh of probes in the soil. One day your task might be to research which variety of heirloom tomato to plant; the next day it might be to update your custom labels. The bots perform everything else that can be measured.
Right now it seems unthinkable: We can’t imagine a bot that can assemble a stack of ingredients into a gift or manufacture spare parts for our lawn mower or fabricate materials for our new kitchen. We can’t imagine our nephews and nieces running a dozen workbots in their garage, churning out inverters for their friend’s electric-vehicle startup. We can’t imagine our children becoming appliance designers, making custom batches of liquid-nitrogen dessert machines to sell to the millionaires in China. But that’s what personal robot automation will enable.
Everyone will have access to a personal robot, but simply owning one will not guarantee success. Rather, success will go to those who innovate in the organization, optimization, and customization of the process of getting work done with bots and machines. Geographical clusters of production will matter, not for any differential in labor costs but because of the differential in human expertise. It’s human-robot symbiosis. Our human assignment will be to keep making jobs for robots—and that is a task that will never be finished. So we will always have at least that one “job.”
In the coming years our relationships with robots will become ever more complex. But already a recurring pattern is emerging. No matter what your current job or your salary, you will progress through these Seven Stages of Robot Replacement, again and again:
· 1. A robot/computer cannot possibly do the tasks I do.
· 2. OK, it can do a lot of them, but it can’t do everything I do.
· 3. OK, it can do everything I do, except it needs me when it breaks down, which is often.
· 4. OK, it operates flawlessly on routine stuff, but I need to train it for new tasks.
· 5. OK, it can have my old boring job, because it’s obvious that was not a job that humans were meant to do.
· 6. Wow, now that robots are doing my old job, my new job is much more fun and pays more!
· 7. I am so glad a robot/computer cannot possibly do what I do now.
This is not a race against the machines. If we race against them, we lose. This is a race with the machines. You’ll be paid in the future based on how well you work with robots. Ninety percent of your coworkers will be unseen machines. Most of what you do will not be possible without them. And there will be a blurry line between what you do and what they do. You might no longer think of it as a job, at least at first, because anything that seems like drudgery will be done by robots.
We need to let robots take over. They will do jobs we have been doing, and do them much better than we can. They will do jobs we can’t do at all. They will do jobs we never imagined even needed to be done. And they will help us discover new jobs for ourselves, new tasks that expand who we are. They will let us focus on becoming more human than we were.
Let the robots take the jobs, and let them help us dream up new work that matters.
Kevin Kelly (kk.org) is senior maverick of Wired and the author, most recently, of What Technology Wants.
Steve Lohr, The New York Times, December 29, 2012
IT was the bold title of a conference this month at the Massachusetts Institute of Technology, and of a widely read article in The Harvard Business Review last October: “Big Data: The Management Revolution.”
Andrew McAfee, principal research scientist at the M.I.T. Center for Digital Business, led off the conference by saying that Big Data would be “the next big chapter of our business history.” Next on stage was Erik Brynjolfsson, a professor and director of the M.I.T. center and a co-author of the article with Dr. McAfee. Big Data, said Professor Brynjolfsson, will “replace ideas, paradigms, organizations and ways of thinking about the world.”
These drumroll claims rest on the premise that data like Web-browsing trails, sensor signals, GPS tracking, and social network messages will open the door to measuring and monitoring people and machines as never before. And by setting clever computer algorithms loose on the data troves, you can predict behavior of all kinds: shopping, dating and voting, for example.
The results, according to technologists and business executives, will be a smarter world, with more efficient companies, better-served consumers and superior decisions guided by data and analysis.
I’ve written about what is now being called Big Data a fair bit over the years, and I think it’s a powerful tool and an unstoppable trend. But a year-end column, I thought, might be a time for reflection, questions and qualms about this technology.
The quest to draw useful insights from business measurements is nothing new. Big Data is a descendant of Frederick Winslow Taylor’s “scientific management” of more than a century ago. Taylor’s instrument of measurement was the stopwatch, timing and monitoring a worker’s every movement. Taylor and his acolytes used these time-and-motion studies to redesign work for maximum efficiency. The excesses of this approach would become satirical grist for Charlie Chaplin’s “Modern Times.” The enthusiasm for quantitative methods has waxed and waned ever since.
Big Data proponents point to the Internet for examples of triumphant data businesses, notably Google. But many of the Big Data techniques of math modeling, predictive algorithms and artificial intelligence software were first widely applied on Wall Street.
At the M.I.T. conference, a panel was asked to cite examples of big failures in Big Data. No one could really think of any. Soon after, though, Roberto Rigobon could barely contain himself as he took to the stage. Mr. Rigobon, a professor at M.I.T.’s Sloan School of Management, said that the financial crisis certainly humbled the data hounds. “Hedge funds failed all over the world,” he said.
THE problem is that a math model, like a metaphor, is a simplification. This type of modeling came out of the sciences, where the behavior of particles in a fluid, for example, is predictable according to the laws of physics.
In so many Big Data applications, a math model attaches a crisp number to human behavior, interests and preferences. The peril of that approach, as in finance, was the subject of a recent book by Emanuel Derman, a former quant at Goldman Sachs and now a professor at Columbia University. Its title is “Models. Behaving. Badly.”
Claudia Perlich, chief scientist at Media6Degrees, an online ad-targeting start-up in New York, puts the problem this way: “You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.”
The bubble that concerns Ms. Perlich is not so much a surge of investment, with new companies forming and then failing in large numbers. That’s capitalism, she says. She is worried about a rush of people calling themselves “data scientists,” doing poor work and giving the field a bad name.
Indeed, Big Data does seem to be facing a work-force bottleneck.
“We can’t grow the skills fast enough,” says Ms. Perlich, who formerly worked for I.B.M. Watson Labs and is an adjunct professor at the Stern School of Business at New York University.
A report last year by the McKinsey Global Institute, the research arm of the consulting firm, projected that the United States needed 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired.
Thomas H. Davenport, a visiting professor at the Harvard Business School, is writing a book called “Keeping Up With the Quants” to help managers cope with the Big Data challenge. A major part of managing Big Data projects, he says, is asking the right questions: How do you define the problem? What data do you need? Where does it come from? What are the assumptions behind the model that the data is fed into? How is the model different from reality?
Society might be well served if the model makers pondered the ethical dimensions of their work as well as studying the math, according to Rachel Schutt, a senior statistician at Google Research.
“Models do not just predict, but they can make things happen,” says Ms. Schutt, who taught a data science course this year at Columbia. “That’s not discussed generally in our field.”
Models can create what data scientists call a behavioral loop. A person feeds in data, which is collected by an algorithm that then presents the user with choices, thus steering behavior.
Consider Facebook. You put personal data on your Facebook page, and Facebook’s software tracks your clicks and your searches on the site. Then, algorithms sift through that data to present you with “friend” suggestions.
Understandably, the increasing use of software that microscopically tracks and monitors online behavior has raised privacy worries. Will Big Data usher in a digital surveillance state, mainly serving corporate interests?
Personally, my bigger concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart. That was a theme of a book by Eli Pariser, titled “The Filter Bubble: What the Internet Is Hiding From You.”
It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?
At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.
“I don’t worship the machine,” she said.
Brett Caine, Forbes, December 24
Brett Caine is senior vice president and general manager for the Citrix Online Services division.
So much has been said and written about being social – we tweet, we pin, we like. We have become a society that communicates and shares just about everything we do, with one notable exception – work. Work is the place where social firewalls go up when they really should come down. After all, our teams are about teamwork. Social is the perfect tool to get our teams to work more collaboratively.
And as it catches on, productivity is improving – people can work and play from anywhere and (finally) debunking the notion that workers need to be in an office to produce. The number of work-at-home employees is increasing dramatically and not just day-extenders. For the first time we are seeing companies implement work-at-home policies and practices that make it possible to work from home as a full member of the team. Everyone wants flexibility, more and more ask for it and the millennials will demand it. What does this changing workforce (and workplace) mean for leaders and managers in the workplace?
In full openness, I work for a company whose business, in part, is to help other businesses collaborate socially. We also use the tools ourselves. In embracing these trends, our company has delivered extremely high employee engagement scores over the past few years – increasing each year despite the challenges and complexities of an ever increasing distributed and global workforce.
With these trends as the backdrop, I predict that in 2013, we’ll start seeing some distinct changes in the workplace – changes that mark a leap forward in the “social” work revolution.
Here are five areas where the changes will take place – areas that all managers and employees need to be ready to address.
· Email evolves into obscurity
Email is not the right tool for office communications. It’s been default as the best way to do it for too long. It’s broken and it (we) can’t keep up in a fast paced, continuous adaption of teams in the workforce.
Email was never intended to be used as a collaboration tool. It slows us down, kills productivity and leaves critical information in silos rather than being shared. This argument was made profoundly by a recent study, which found that the average worker spends nearly a third of his or her work week managing email and nearly 20 percent of his or her time looking for internal information and/or tracking down colleagues who can help with specific tasks. Leaders of major companies are shutting down internal email – simply to improve communication and get more work done.
In 2013, we will see a shift in how emails are used in the workplace. Increasingly, emails will evolve into notifications that will signal to readers that something is happening somewhere else – e.g., a brainstorming call at 2 p.m. The content and reference materials will be in the cloud and continuously updated, synchronized and easily available from anywhere.
· Social collaboration becomes invisible
With more and more businesses turning to the cloud for effective collaboration solutions, the social enterprise isn’t the latest idea for companies. Microsoft, Salesforce and IBM proved that with major acquisitions of social networking companies for enterprises. When multiple, large industry players enter a space, it’s the beginning of a major trend and unlikely to fade away. In my view, the trend will become so common, social becomes the frame for collaboration in the future.
If you really want to social enable your company, figure out how to actually get to “work” with social collaboration instead of just talking about it. This will be the tipping point for users and companies to get the benefit in 2013 and beyond.
· Needs for collaboration increase as we get better at preparing for unlikely events
Unfortunately, natural and man-made disasters are likely to increase in the years ahead. Companies therefore are updating their crisis management plans. As part of that preparation, it makes perfect sense for companies to invest in social collaboration tools that allow them to maintain business under any circumstances. Increasingly, companies and employee commutes and routines are getting disrupted by storms, transit strikes, outages and fires to name a few. In 2013, get ahead of this and invest in both tools and policies to enable work from anywhere business models. Certainly, social collaboration tools, remote access and support for employees regardless of location are smart steps to take to prepare your business to stay open even when the most unlikely events occur.
· Your personal cloud will have a single access point
As businesses move to the cloud, this will also be the year where you will start seeing one point of access to your data regardless of where it’s stored. You will no longer have to login and check five, six or even seven different storage repositories for the information you need for work. One app will connect you with all your social channels – personal, professional, you name it. That ease-of-use is going to drive adoption in droves.
· Casual Fridays will become Work-from-Home Fridays
The tablet generation has already redefined the workplace. This on-the-go generation is not defined by four walls and a desktop computer. They are mobile. Mobile is the new normal. It’s not the exception. We will see more small- and medium sized companies take this generation into consideration and consider virtual offices and workshifting to incentivize their employees.
With rising gas prices and the high costs of commercial real estate, it makes financial sense for SMBs in particular to consider alternatives to traditional corporate headquarters and hubs. Office-sharing in some regions has become quite popular, particularly among start-ups. In return, we will have a more dynamic workforce and work environment instead of the same old stagnant cube nation.
The future is indeed bright for going social in the workplace. It’s great to see work catch up to everyday life. It’s something we can all like.
The private sector has all the tools we need to flag rapid weapons build-ups and suspicious purchases. All that's needed is the political will to build the most basic database.
Marc Parrish, The Atlantic, December 27, 2012
Big data might have stopped the massacres in Newtown, Aurora, and Oak Creek. But it didn't, because there is no national database of gun owners, and no national record-keeping of firearm and ammunition purchases. Most states don't even require a license to buy or keep a gun.
That's a tragedy, because combining simple math and the power of crowds could give us the tools we need to red flag potential killers even without new restrictions on the guns anyone can buy. Privacy advocates may hate the idea, but an open national database of ammunition and gun purchases may be what America needs if we're ever going to get our mass shooting problem under control.
Just look at the gun-acquiring backgrounds of some of our more recent mass killers to see what I mean. James Holmes, the Aurora shooting suspect, went to three different locations spread out over 30 miles to legally buy his four weapons. All three were reputable outdoors retail chain stores. He then went online, and bought thousands of rounds of ammunition along with assault gear. UPS delivered around 90 packages to Holmes at his medical campus in that short period. It doesn't take a PhD in statistics to see that a quick, massive buildup of arms like this by a private individual -- especially one, like Holmes, who was known in his community for having growing mental health issues -- should raise a red flag.
In Newtown, Adam Lanza carried hundreds of rounds -- enough to kill every student in the Sandy Hook Elementary school if he had not been stopped. But he also attempted to destroy his hard drives to cover his pre-rampage digital tracks. Clearly he feared the data he left behind.
The list of examples can go on. Seung-Hui Cho, the Virginia Tech student who committed the worst mass shooting in American history, bought two semi-automatic handguns, along with hollow point bullets, from dealers in just over a month. A few weeks later, he purchased 10-round magazines from a seller in Idaho through eBay. All this was after he failed to disclose information about his mental health on the gun-purchasing background questionnaire (specifically, that he had been court-ordered to outpatient treatment at a mental health facility).
Though we call this kind of thing big data, a database tracking ammunition and gun purchases would be, in fact, tiny. As someone who deals daily with the deluge of data currently inundating the marketing world, I can say based on experience that this kind of record-keeping would be an inconsequential task to set up, and the data science to analyze it trivial. Massive efforts are going into far smaller things, such as which TV program is most engaging for soap buyers who have DVRs, and which pitcher/batter combinations lead to better baseball.
Let me give you some numbers. According to the ATF, about 4.5 million new firearms, including approximately 2 million handguns, are sold in the United States every year, along with roughly 2 million more secondhand ones. No one knows how much ammunition is sold yearly in the country, but as a yardstick, the U.S. Military bought 1.8 billion rounds in 2005.
That sounds like a lot -- but it isn't. Not in today's data-driven private sector. This data would fit on a thumb drive you can buy at any Best Buy and still leave room for the transcripts of every Rush Limbaugh show ever.
By numerical contrast, Netflix, with 24 million users, will stream several billion movies this year, and Walmart sees many times this in transactions daily. Twitter had 150 million tweets about the 2012 London Olympics, and 12 million in the first hour after Osama bin Laden was killed.
Big, complicated data, this is not. But the political will to create a firearm and ammunition purchasing database will need to be massive if we're ever going to have one. Michael Bloomberg, in his anger over the Sandy Hook rampage and armed with his reputation and personal bankroll, may be the only person able (and willing) to counter the NRA, whose opposition to any minor gun "restriction" has kept us in political paralysis for decades as tens of thousands of Americans die every year from gun violence. The NRA's opposing argument will be that it is an invasion of privacy for gun owners. But in our post-9/11 world, we already have ample precedents to do this. Go into any airport and see what happens when you try and buy a ticket with cash on the next flight out. You will not board the flight without security calling you aside for questions. Go into a pharmacy in dozens of states and buy cold medicine and you will be asked for ID and tracked in the NPLEX database. Go on the Internet and you can read that the cellphone carriers told Congress that U.S. law enforcement made a staggering 1.3 millions requests for customer text messages, caller locations, and other information just in 2011. And that the number of requests had doubled in the last five years. Most of this cell phone data is requested without even a warrant issued by a judge.
There is no outrage by the American public over any of this, even though it causes citizens inconvenience and invades their privacy. We are willing to permit much when we are convinced that it is in the interest of government making us safer. Getting carded for cold medicine does not bother Fox News pundits as a freedom limiter. But government goes even further. Most states limit the number of Sudafed you can buy in a month and keep quietly reducing this quantity. This is more restrictive than the size of soda in New York City. Ammo, handguns and rifle purchases have been excluded from the simple tracking mechanisms and scrutiny of algorithms that these other pursuits are subjected to.
Why?
After all, flying on a plane, buying cold medicine, and using your cell phone are much more common (so more people are tracked) than purchasing a weapon, and much less dangerous. To keep us safe, our government has decided they need the brightest mathematical minds to analyze records on the former and not the latter.
Imagine if that were to change. Armed only with data, we could begin to see the patterns between guns and ammunition purchases and violence, and to flag those people most at risk of killing dozens of their neighbors.
The digital pioneer and visionary behind virtual reality has turned against the very culture he helped create
Ron Rosenbaum, Smithsonian, January 2013
I couldn’t help thinking of John Le Carré’s spy novels as I awaited my rendezvous with Jaron Lanier in a corner of the lobby of the stylish W Hotel just off Union Square in Manhattan. Le Carré’s espionage tales, such as The Spy Who Came In From the Cold, are haunted by the spectre of the mole, the defector, the double agent, who, from a position deep inside, turns against the ideology he once professed fealty to.
And so it is with Jaron Lanier and the ideology he helped create, Web 2.0 futurism, digital utopianism, which he now calls “digital Maoism,” indicting “internet intellectuals,” accusing giants like Facebook and Google of being “spy agencies.” Lanier was one of the creators of our current digital reality and now he wants to subvert the “hive mind,” as the web world’s been called, before it engulfs us all, destroys political discourse, economic stability, the dignity of personhood and leads to “social catastrophe.” Jaron Lanier is the spy who came in from the cold 2.0.
To understand what an important defector Lanier is, you have to know his dossier. As a pioneer and publicizer of virtual-reality technology (computer-simulated experiences) in the ’80s, he became a Silicon Valley digital-guru rock star, later renowned for his giant bushel-basket-size headful of dreadlocks and Falstaffian belly, his obsession with exotic Asian musical instruments, and even a big-label recording contract for his modernist classical music. (As he later told me, he once “opened for Dylan.” )
The colorful, prodigy-like persona of Jaron Lanier—he was in his early 20s when he helped make virtual reality a reality—was born among a small circle of first-generation Silicon Valley utopians and artificial-intelligence visionaries. Many of them gathered in, as Lanier recalls, “some run-down bungalows [I rented] by a stream in Palo Alto” in the mid-’80s, where, using capital he made from inventing the early video game hit Moondust, he’d started building virtual-reality machines. In his often provocative and astute dissenting book You Are Not a Gadget, he recalls one of the participants in those early mind-melds describing it as like being “in the most interesting room in the world.” Together, these digital futurists helped develop the intellectual concepts that would shape what is now known as Web 2.0—“information wants to be free,” “the wisdom of the crowd” and the like.
And then, shortly after the turn of the century, just when the rest of the world was turning on to Web 2.0, Lanier turned against it. With a broadside in Wired called “One-Half of a Manifesto,” he attacked the idea that “the wisdom of the crowd” would result in ever-upward enlightenment. It was just as likely, he argued, that the crowd would devolve into an online lynch mob.
Lanier became the fiercest and weightiest critic of the new digital world precisely because he came from the Inside. He was a heretic, an apostate rebelling against the ideology, the culture (and the cult) he helped found, and in effect, turning against himself.
***
And despite his apostasy, he’s still very much in the game. People want to hear his thoughts even when he’s castigating them. He’s still on the Davos to Dubai, SXSW to TED Talks conference circuit. Indeed, Lanier told me that after our rendezvous, he was off next to deliver the keynote address at the annual meeting of the Ford Foundation uptown in Manhattan. Following which he was flying to Vienna to address a convocation of museum curators, then, in an overnight turnaround, back to New York to participate in the unveiling of Microsoft’s first tablet device, the Surface.
Lanier freely admits the contradictions; he’s a kind of research scholar at Microsoft, he was on a first-name basis with “Sergey” and “Steve” (Brin, of Google, and Jobs, of Apple, respectively). But he uses his lecture circuit earnings to subsidize his obsession with those extremely arcane wind instruments. Following his Surface appearance he gave a concert downtown at a small venue in which he played some of them.
Lanier is still in the game in part because virtual reality has become, virtually, reality these days. “If you look out the window,” he says pointing to the traffic flowing around Union Square, “there’s no vehicle that wasn’t designed in a virtual-reality system first. And every vehicle of every kind built—plane, train—is first put in a virtual-reality machine and people experience driving it [as if it were real] first.”
I asked Lanier about his decision to rebel against his fellow Web 2.0 “intellectuals.”
“I think we changed the world,” he replies, “but this notion that we shouldn’t be self-critical and that we shouldn’t be hard on ourselves is irresponsible.”
For instance, he said, “I’d been an early advocate of making information free,” the mantra of the movement that said it was OK to steal, pirate and download the creative works of musicians, writers and other artists. It’s all just “information,” just 1’s and 0’s.
Indeed, one of the foundations of Lanier’s critique of digitized culture is the very way its digital transmission at some deep level betrays the essence of what it tries to transmit. Take music.
“MIDI,” Lanier wrote, of the digitizing program that chops up music into one-zero binaries for transmission, “was conceived from a keyboard player’s point of view...digital patterns that represented keyboard events like ‘key-down’ and ‘key-up.’ That meant it could not describe the curvy, transient expressions a singer or a saxophone note could produce. It could only describe the tile mosaic world of the keyboardist, not the watercolor world of the violin.”
Quite eloquent, an aspect of Lanier that sets him apart from the HAL-speak you often hear from Web 2.0 enthusiasts (HAL was the creepy humanoid voice of the talking computer in Stanley Kubrick’s prophetic 2001: A Space Odyssey). But the objection that caused Lanier’s turnaround was not so much to what happened to the music, but to its economic foundation.
I asked him if there was a single development that gave rise to his defection.
“I’d had a career as a professional musician and what I started to see is that once we made information free, it wasn’t that we consigned all the big stars to the bread lines.” (They still had mega-concert tour profits.)
“Instead, it was the middle-class people who were consigned to the bread lines. And that was a very large body of people. And all of a sudden there was this weekly ritual, sometimes even daily: ‘Oh, we need to organize a benefit because so and so who’d been a manager of this big studio that closed its doors has cancer and doesn’t have insurance. We need to raise money so he can have his operation.’
“And I realized this was a hopeless, stupid design of society and that it was our fault. It really hit on a personal level—this isn’t working. And I think you can draw an analogy to what happened with communism, where at some point you just have to say there’s too much wrong with these experiments.”
His explanation of the way Google translator works, for instance, is a graphic example of how a giant just takes (or “appropriates without compensation”) and monetizes the work of the crowd. “One of the magic services that’s available in our age is that you can upload a passage in English to your computer from Google and you get back the Spanish translation. And there’s two ways to think about that. The most common way is that there’s some magic artificial intelligence in the sky or in the cloud or something that knows how to translate, and what a wonderful thing that this is available for free.
“But there’s another way to look at it, which is the technically true way: You gather a ton of information from real live translators who have translated phrases, just an enormous body, and then when your example comes in, you search through that to find similar passages and you create a collage of previous translations.”
“So it’s a huge, brute-force operation?” “It’s huge but very much like Facebook, it’s selling people [their advertiser-targetable personal identities, buying habits, etc.] back to themselves. [With translation] you’re producing this result that looks magical but in the meantime, the original translators aren’t paid for their work—their work was just appropriated. So by taking value off the books, you’re actually shrinking the economy.”
The way superfast computing has led to the nanosecond hedge-fund-trading stock markets? The “Flash Crash,” the “London Whale” and even the Great Recession of 2008?
“Well, that’s what my new book’s about. It’s called The Fate of Power and the Future of Dignity, and it doesn’t focus as much on free music files as it does on the world of finance—but what it suggests is that a file-sharing service and a hedge fund are essentially the same things. In both cases, there’s this idea that whoever has the biggest computer can analyze everyone else to their advantage and concentrate wealth and power. [Meanwhile], it’s shrinking the overall economy. I think it’s the mistake of our age.”
The mistake of our age? That’s a bold statement (as someone put it in Pulp Fiction). “I think it’s the reason why the rise of networking has coincided with the loss of the middle class, instead of an expansion in general wealth, which is what should happen. But if you say we’re creating the information economy, except that we’re making information free, then what we’re saying is we’re destroying the economy.”
The connection Lanier makes between techno-utopianism, the rise of the machines and the Great Recession is an audacious one. Lanier is suggesting we are outsourcing ourselves into insignificant advertising-fodder. Nanobytes of Big Data that diminish our personhood, our dignity. He may be the first Silicon populist.
“To my mind an overleveraged unsecured mortgage is exactly the same thing as a pirated music file. It’s somebody’s value that’s been copied many times to give benefit to some distant party. In the case of the music files, it’s to the benefit of an advertising spy like Google [which monetizes your search history], and in the case of the mortgage, it’s to the benefit of a fund manager somewhere. But in both cases all the risk and the cost is radiated out toward ordinary people and the middle classes—and even worse, the overall economy has shrunk in order to make a few people more.”
Lanier has another problem with the techno-utopians, though. It’s not just that they’ve crashed the economy, but that they’ve made a joke out of spirituality by creating, and worshiping, “the Singularity”—the “Nerd Rapture,” as it’s been called. The belief that increasing computer speed and processing power will shortly result in machines acquiring “artificial intelligence,” consciousness, and that we will be able to upload digital versions of ourselves into the machines and achieve immortality. Some say as early as 2020, others as late as 2045. One of its chief proponents, Ray Kurzweil, was on NPR recently talking about his plans to begin resurrecting his now dead father digitally.
Some of Lanier’s former Web 2.0 colleagues—for whom he expresses affection, not without a bit of pity—take this prediction seriously. “The first people to really articulate it did so right about the late ’70s, early ’80s and I was very much in that conversation. I think it’s a way of interpreting technology in which people forgo taking responsibility,” he says. “‘Oh, it’s the computer did it not me.’ ‘There’s no more middle class? Oh, it’s not me. The computer did it.’
“I was talking last year to Vernor Vinge, who coined the term ‘singularity,’” Lanier recalls, “and he was saying, ‘There are people around who believe it’s already happened.’ And he goes, ‘Thank God, I’m not one of those people.’”
In other words, even to one of its creators, it’s still just a thought experiment—not a reality or even a virtual-reality hot ticket to immortality. It’s a surreality.
Lanier says he’ll regard it as faith-based, “Unless of course, everybody’s suddenly killed by machines run amok.”
“Skynet!” I exclaim, referring to the evil machines in the Terminator films.
At last we come to politics, where I believe Lanier has been most farsighted—and which may be the deep source of his turning into a digital Le Carré figure. As far back as the turn of the century, he singled out one standout aspect of the new web culture—the acceptance, the welcoming of anonymous commenters on websites—as a danger to political discourse and the polity itself. At the time, this objection seemed a bit extreme. But he saw anonymity as a poison seed. The way it didn’t hide, but, in fact, brandished the ugliness of human nature beneath the anonymous screen-name masks. An enabling and foreshadowing of mob rule, not a growth of democracy, but an accretion of tribalism.
It’s taken a while for this prophecy to come true, a while for this mode of communication to replace and degrade political conversation, to drive out any ambiguity. Or departure from the binary. But it slowly is turning us into a nation of hate-filled trolls.
Surprisingly, Lanier tells me it first came to him when he recognized his own inner troll—for instance, when he’d find himself shamefully taking pleasure when someone he knew got attacked online. “I definitely noticed it happening to me,” he recalled. “We’re not as different from one another as we’d like to imagine. So when we look at this pathetic guy in Texas who was just outed as ‘Violentacrez’...I don’t know if you followed it?”
“I did.” “Violentacrez” was the screen name of a notorious troll on the popular site Reddit. He was known for posting “images of scantily clad underage girls...[and] an unending fountain of racism, porn, gore” and more, according to the Gawker.com reporter who exposed his real name, shaming him and evoking consternation among some Reddit users who felt that this use of anonymity was inseparable from freedom of speech somehow.
“So it turns out Violentacrez is this guy with a disabled wife who’s middle-aged and he’s kind of a Walter Mitty—someone who wants to be significant, wants some bit of Nietzschean spark to his life.”
Only Lanier would attribute Nietzschean longings to Violentacrez. “And he’s not that different from any of us. The difference is that he’s scared and possibly hurt a lot of people.”
Well, that is a difference. And he couldn’t have done it without the anonymous screen name. Or he wouldn’t have.
And here’s where Lanier says something remarkable and ominous about the potential dangers of anonymity.
“This is the thing that continues to scare me. You see in history the capacity of people to congeal—like social lasers of cruelty. That capacity is constant.”
“Social lasers of cruelty?” I repeat.
“I just made that up,” Lanier says. “Where everybody coheres into this cruelty beam....Look what we’re setting up here in the world today. We have economic fear combined with everybody joined together on these instant twitchy social networks which are designed to create mass action. What does it sound like to you? It sounds to me like the prequel to potential social catastrophe. I’d rather take the risk of being wrong than not be talking about that.”
Here he sounds less like a Le Carré mole than the American intellectual pessimist who surfaced back in the ’30s and criticized the Communist Party he left behind: someone like Whittaker Chambers.
But something he mentioned next really astonished me: “I’m sensitive to it because it murdered most of my parents’ families in two different occasions and this idea that we’re getting unified by people in these digital networks—”
“Murdered most of my parents’ families.” You heard that right. Lanier’s mother survived an Austrian concentration camp but many of her family died during the war—and many of his father’s family were slaughtered in prewar Russian pogroms, which led the survivors to flee to the United States.
It explains, I think, why his father, a delightfully eccentric student of human nature, brought up his son in the New Mexico desert—far from civilization and its lynch mob potential. We read of online bullying leading to teen suicides in the United States and, in China, there are reports of well-organized online virtual lynch mobs forming...digital Maoism.
He gives me one detail about what happened to his father’s family in Russia. “One of [my father’s] aunts was unable to speak because she had survived the pogrom by remaining absolutely mute while her sister was killed by sword in front of her [while she hid] under a bed. She was never able to speak again.”
It’s a haunting image of speechlessness. A pogrom is carried out by a “crowd,” the true horrific embodiment of the purported “wisdom of the crowd.” You could say it made Lanier even more determined not to remain mute. To speak out against the digital barbarism he regrets he helped create.
Andrew Hacker, New York Review of Books, January 10, 2013
The Signal and the Noise: Why So Many Predictions Fail—But Some Don't
by Nate Silver
Penguin, 534 pp., $27.95
The Physics of Wall Street: A Brief History of Predicting the Unpredictable
by James Owen Weatherall
Houghton Mifflin Harcourt, 286 pp., $27.00
Antifragile: Things That Gain from Disorder
by Nassim Nicholas Taleb
Random House, 519 pp., $30.00
1.
Nate Silver called every state correctly in the last presidential race, and was wrong about only one in 2008. In 2012 he predicted Obama's total of the popular vote within one tenth of a percent of the actual figure. His powers of prediction seemed uncanny. In his early and sustained prediction of an Obama victory, he was ahead of most polling organizations and my fellow political scientists. But buyers of his book, The Signal and the Noise, now a deserved best seller, may be in for something of a surprise. There's only a short chapter on predicting elections, briefer than ones on baseball, weather, and chess. In fact, he's written a serious treatise about the craft of prediction—without academic mathematics—cheerily aimed at lay readers. Silver's coverage is polymathic, ranging from poker and earthquakes to climate change and terrorism.
We learn that while more statistics per capita are collected for baseball than perhaps any other human activity, seasoned scouts still surpass algorithms in predicting the performance of players. Since poker depends as much on luck as on skill, professionals make a living by having well-heeled amateurs at the table. The lesson from a long chapter on earthquakes is that while we're good at measuring them, they're "not really predictable at all." Much the same caution holds for economists, whose forecasts of next year's growth are seldom correct. Their models may be elegant, Silver says, but "their raw data isn't much good."
The most striking success has been in forecasting where hurricanes will hit. Over the last twenty-five years, the ability to pinpoint landfalls has increased twelvefold. At the same time, Silver says, newscasts purposely overpredict rain, since they know their listeners will be grateful when they find they don't need umbrellas. While he doesn't dismiss "highly mathematical and data-driven techniques," he cautions climate modelers not to give out precise changes in temperature and ocean levels. He tells of attending a conference on terrorism at which a Coca-Cola marketing executive and a dating service consultant were asked for hints on how to identify suicide bombers.
Much is made of ours being an era of Big Data. Silver passes on an estimate from IBM that 2.5 quintillion (that's seventeen zeros) new bytes (sequences of eight binary digits that each encode a single character of text in a computer) of data are being created every day, representing everything from the brand of toothpaste you bought yesterday to your location when you called a friend this morning. Such information can be put together to fashion personal profiles, which Amazon and Google are already doing in order to target advertisements more accurately. Obama's tech-savvy workers did something similar, notably in identifying voters who needed extra prompting to go to the polls.1
Those daily quintillions are what led to Silver's title. "Signals" are facts we want and need, such as those that will help us detect incipient shoe bombers. "Noise" is everything else, usually extraneous information that impedes or misleads our search for signals. Silver makes the failure to forecast September 11 a telling example.
But first, The Signal and the Noise is in large part a homage to Thomas Bayes (1701–1761), a long-neglected statistical scholar, especially by the university departments concerned with statistical methods. The Bayesian approach to probability is essentially simple: start by approximating the odds of something happening, then alter that figure as more findings come in. So it's wholly empirical, rather than building edifices of equations.2 Silver has a diverting example on whether your spouse may be cheating. You might start with an out-of-the-air 4 percent likelihood. But a strange undergarment could raise it to 50 percent, after which the game's afoot. This has importance, Silver suggests, because officials charged with anticipating terrorist acts had not conjured a Bayesian "prior" about the possible use of airplanes.
Silver is prepared to say, "We had some reason to think that an attack on the scale of September 11 was possible." His Bayseian "prior" is that airplanes were targeted in the cases of an Air India flight in 1985 and Pan Am's over Lockerbie three years later, albeit using secreted bombs, plus in later attempts that didn't succeed. At the least, a chart with, say, a 4 percent likelihood of an attack should have been on someone's wall. Granted, what comes in as intelligence is largely "noise." (Most intercepted conversations are about plans for dinner.) Still, in the summer of 2001, staff members at a Minnesota flight school told FBI agents of a Moroccan-born student who wanted to learn to pilot a Boeing 747 in midair, skipping lessons on taking off and landing. Some FBI agents took the threat of Zacarias Moussaoui seriously, but several requests for search and wiretap warrants were denied. In fact, an instructor added that a fuel-laden plane could make a horrific weapon. At the least, these "signals" should have raised the probability of an attack using an airplane, say, to 15 percent, prompting visits to other flight schools.
Silver's "mathematics of terrorism" may be stretching the odds a bit. Many of those daily quintillion digits flow into the FBI and CIA, not to mention the departments of State and Defense. To follow all of them up is patently impossible, with only a small fraction getting even a cursory second look. It's bemusing that two recent revelations of marital infidelity—Eliot Spitzer and David Petraeus—arose from inquiries having other purposes. Plus there's the question of how many investigators and investigations we want to have, as more searching will inevitably touch more of us.
Yet in the end, Silver's claims are quite modest. Indeed, he could have well phrased his subtitle "why most predictions fail." It's simply because "the volume of information is increasing exponentially."
There is no reason to conclude that the affairs of man are becoming more predictable. The opposite may well be true. The same sciences that uncover the laws of nature are making the organization of society more complex.
I'd only add that it's not just what sciences are finding that makes the world seem more complex. Shifts in the structure of occupations, abetted by more college degrees, have increased the number of positions deemed to be professional. If entrepreneurs tend to be assessed by how much money they amass, professionals are rated by the presumed complexity of what they know and do. So to retain or raise an occupation's status, tasks are made more mysterious, usually by taking what's really simple and adding obfuscating layers. The very sciences Silver cites—especially those of a social sort—rank among the culprits.
2.
Nate Silver is known not so much for predicting who will win elections, but for how close he comes to the actual results. His final 2012 forecast gave Obama 50.8 percent of the popular vote, almost identical with his eventual figure of 50.9 percent. This kind of precision is striking. A more typical projection may warn that it has a three-point margin of error either way, meaning a candidate accorded 52 percent could end anywhere between 55 percent and 49 percent. Or, fearful of making a wrong call, as in 2000, polling agencies will claim that the outcome is too close to foretell. Still, it's too early to hail a new statistical science. As can be seen in Table A, Rasmussen's and Gallup's final polls predicted that Romney would be the winner, while the Boston Herald gave its state's senate race to Scott Brown.
In fact, I am impressed when polls come even close. To start, what's needed is a reliable cross-section of people who will actually vote. In 2008, only 62 percent of eligible citizens cast ballots. In 2012, even fewer did. Not surprisingly, some people who seldom or never vote will still claim they'll be turning out. Testing them ("can you tell me where your polling place is?") can be time-consuming and expensive. And there are those who don't report their real choices. But much more vexing is finding people willing to cooperate. According to a recent Pew Research Center report, only several years ago, in 1997, about 90 percent of a desired sample could be reached in person or at home by telephone, and 36 percent of them were amenable to an interview.
Today, with fewer people at home or picking up calls, and increasing refusals from those who do, the rates are down to 62 percent and 9 percent.3 So the polls must create a model of an electorate from the slender slice willing to give them time. Yet despite these hurdles, the Columbus Dispatch called Ohio's result perfectly, using 1,501 respondents from the state's 5,362,236 voters (the figures available on December 7).
Election polls are unique in at least two ways. First, they aim to tell us about a concrete act—a cast ballot—to be performed in an impending period of time. (Each year, more of us vote early.) It's hard to think of other surveys that try to anticipate what a huge pool of adults will do. Second, how well a poll did becomes known once the votes are counted. So we find Nate Silver got it right and Rasmussen and Gallup didn't. But a poll's accuracy is only a historic curiosity after the returns are in. That is, it didn't tell us anything lasting; just about a foray into forecasting during some months when a lot of us were wondering how events would turn out.
Other polls tell us about something less fleeting: the opinions people hold on public issues and personal matters.
But with polls on opinions—military spending, say, or the provision of contraceptives—there's seldom a subsequent vote that can validate findings. (To an extent, this is possible when there are statewide votes on issues like affirmative action and gay marriage.) A recourse is to compare a series of surveys that ask similar questions.
Yet as Table B shows, responses on abortion have been quite varied. What could be called the "pro-choice" side ranges across twenty-three percentage points. Certainly, how the question is phrased can skew the answers. CBS's 42 percent agreed that abortion should be "generally available," while Gallup's 25 percent were supporting the view that abortion should be "always legal," and The Washington Post's 19 percent were for abortion to be "legal in all cases." The short answer is that apart from the severe anti side, polling can't give us specific figures on where most adults line up on abortion. Or, for that matter, any issue.
What goes on in the American mind remains a mystery that sampling is unlikely to unlock. In my estimate, the 65,075,450 people who chose Barack Obama and Joseph Biden over Mitt Romney and Paul Ryan were mainly expressing a moral mood, a feeling about the kind of country they want. I'd like to see Nate Silver using his statistical talents to explore such surmises.
We've been informed that 55 percent of women supported Obama, rising to 67 percent of those who are single, divorced, or widowed. Obama also secured 55 percent among holders of postgraduate degrees, and 69 percent of Jewish voters. But how can we know? Voting forms don't ask for marital status or religion. The answer is that these and similar figures were extrapolated from a national sample of 26,563 voters, approached just after they cast their ballots or telephoned later in the day, by an organization called Edison Research.
The figures I've cited and others on the list look plausible to me. Still, there's no way to check them; moreover, the Edison survey is the only post-election one that was done. So here's a caveat: Jews are so small a fraction of the electorate that there were only 241 in the sample. Thus the abovementioned 69 percent comes with a seven-point margin of error either way, a caveat not noted in most media accounts.
Nate Silver doesn't conduct his own polls. Rather, he collects a host of state and national reports, and enters them in a database of his own devising. Combining samples from varied surveys gives him a much larger pool of respondents and the potential for a more reliable profile. Of course, Silver doesn't simply crunch whatever comes in. He factors in past predictions and looks for slipshod work, as when the Florida Times-Union on election eve gave the state to Romney, based on 681 interviews. He pays special attention to demographic shifts, such as a surge in registrations with Hispanic names. His model also draws on the Cook Political Report, which actually meets informally with candidates to assess their electoral appeal. In September, Silver set the odds of Obama's winning at 85 percent, enough to withstand a dismal performance in the first debate, which hadn't yet occurred.
3.
Early in The Signal and the Noise, Silver alludes to Isaiah Berlin's trope about hedgehogs and foxes. Hedgehogs know "one big thing," while foxes know "many little things." But there's more. Hedgehogs display a disconcerting certainty that their one idea will put everything straight, whether on intellectual questions or in the working world. Moreover, Berlin warned, hedgehogs can cause a lot of damage when their nostrums are applied. Silver sees himself as a more modest fox, willing to draw on varied approaches to get his job done. So The Signal and the Noise doesn't end with a crescendo, but actually stresses the quite limited ambit of our power to predict.
James Weatherall is an unabashed hedgehog, propounding a single idea with an uncommon confidence. After training in physics, philosophy, and mathematics, he now teaches logic and the philosophy of science at the University of California's Irvine campus. With The Physics of Wall Street, he is taking his training even further: to finance in its preeminent location.
He is a man with a mission: to bring a heightened rationality to investment decisions. His book opens with an admiring visit to a hedge fund where a third of the employees have doctorates in physics, mathematics, statistics, even astronomy. Indeed, their rarefied insights are what's wanted; "PhDs in finance need not apply." In fact, there's a niche Wall Street sector called "quant firms," which reserve key positions for holders of advanced degrees.
Weatherall would have this perspective pervade the entire financial industry. He succinctly states his one big idea: "Insights that are commonplace in physics…are useful in studying virtually anything." In one sense, Wall Street's products have a physical character. Collateralized debt obligations, credit default swaps, and initial public offerings appear on paper or as electronic impulses. But Weatherall means more than this. Markets, he believes, are subject to physical laws. His star witness is Louis Bachelier, a French mathematician at the turn of the last century, who used Brownian motion to evaluate stock options. After that, we hear how ideas from such mathematicians as Jacob Bernoulli and Benoît Mandelbrot can be applied to mitigating risks and minimizing uncertainty. Not least of their influence has been to entrench mathematics in MBA programs, Wall Street's principal recruiting pool.
But the book is less a celebration of the past than a prospectus for the future. Weatherall anticipates "a breakthrough in our ability to identify the underlying chaotic patterns lurking in market data," that is, to impose order on the "noise" that bedevils Silver. In a similar vein, he foresees strides "in predicting financial calamity using mathematical techniques," perhaps like Russia's default in 1998, which some Nobel economists didn't see coming. He also hopes that "studies of psychology and human behavior" can be framed so that they are "symbiotic with mathematical approaches to economics." Here he foresees doing better than Newton, who confessed, "I can calculate the movements of stars, but not the madness of man." Almost everyone favors rigor and unlocking more mysteries. That's why most of us support science. Can there be anything amiss in feeling optimistic about crossing new frontiers?
What isn't explained is whether melding physics with finance will bring more benefits for everyone, or only give an advantage to those who use the techniques—like high-speed trading, if you own a mainframe computer. We can agree there's a lot of irrationality—not to say exuberance—in the investment world. But it's not clear if Weatherall is saying that decisions based on Bernoulli will allocate capital more efficiently, and hence serve the commonweal. When physics enters medicine, as with MRIs, we can have a reasonable hope that patients will benefit as much as their doctors. But when quants were riding high on Wall Street, they were hired only to give their own firms an edge over the competition.
Alluding to the collapse of Bear Stearns and Lehman Brothers, the housing bubble, and the October 2008 crash, Weatherall concedes that "the misuse of mathematical models played a role in this crisis." Still, his implication here is that the models themselves didn't contribute to the downfall; it was that they were somehow mishandled. The ultimate problem with hedgehogs is hubris. In this case, it's the assumption that the quality of our thought can be enhanced by new methodologies. The word "sophistication" recurs on almost every page of The Physics of Wall Street, as if to affirm that higher powers are present.
True, the discovery of the calculus enabled us to build planes that travel faster than the speed of sound. But thus far I've found scant evidence that mathematics and physics have a capacity to give us a deeper understanding of human and social behavior. Of course, we should be open to new findings. Still, that differs from proclaiming that "what we do know for sure is that there will be a next major advance, and…we will understand markets more clearly than we do today." Here he seems to be saying that the physical sciences can tell us how to avoid the crashes and crises we now periodically face. If that's so, I wish Weatherall had listed some warnings of coming "calamities" based on his physical laws, such as the impending college loan bubble: When will it burst, and how widespread will the fallout be?
Nassim Nicholas Taleb in Antifragile calls such certainty "the error of naive rationalism." And it's a naiveté with consequences. There's no doubt that the quants who bundled bad mortages, adding algorithms to rate them AAA, helped to give us the current recession.4 But just as culpable were their nonmathematical superiors who allowed them such rein. So there's a broader issue. Financial firms want to be at the cutting edge, which now means having a bevy of Ph.D.s, just as at other times and places, enterprises might feel they should have an accredited gypsy with tarot cards. We are coming close to deifying anything smacking of STEM—science, technology, engineering, and mathematics—whether for staying ahead of China or cutting-edge careers for our young people. If firms need workers adept in algebra, then such instruction should be available. But to rely on physics and mathematics for deciphering human behavior, in markets or elsewhere, can only bring blind alleys. Less of the world than we might like is, as Taleb puts it, "academizable, rationalizable, formalizable, theoretizable." When such rubrics crowd out more discursive thinking, we all lose.
1.
See Michael Scherer, "Inside the Secret World of the Data Crunchers Who Helped Obama Win," Time, November 7, 2012, and Nate Silver, "In Silicon Valley, Technology Talent Gap Threatens GOP Campaigns," The New York Times, November 28, 2012. ↩
2.
See Sharon Bertsch McGrayne's superb The Theory That Would Not Die (Yale University Press, 2011). ↩
3.
"Assessing the Representativeness of Public Opinion Surveys," The Pew Research Center for the People and the Press, May 15, 2012. ↩
4.
Weatherall barely mentions Scott Patterson's indispensable The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It (Crown Business, 2010). And some stories worth reading: Felix Salmon, "Recipe for Disaster: The Formula That Killed Wall Street," Wired, March 2009; Dennis Overbye, "They Tried to Outsmart Wall Street," The New York Times, March 9, 2009; Julie Creswell, "The Quants are Reeling," The New York Times, August 20, 2010; Pablo Triana, "The Flawed Maths of Financial Models," Financial Times, November 29, 2010. ↩
