Thaler: Applause for the Numbers Machine

Richard H. Thaler, The New York Times, November 18, 2012

THE biggest winners on Election Day weren’t politicians; they were numbers folks.

Computer scientists, behavioral scientists, statisticians and everyone who works with data should be proud. They told us who was going to win, but they also helped to make many of those victories happen.

Three groups of geeks deserve the love they rarely receive: people who run political polls, those who analyze the polls and those who figure out how to help campaigns connect with voters.

Many people doubted the accuracy of political polling this year. Part of the skepticism was based on the wide range of predictions, with some showing President Obama in the lead, and others Mitt Romney. But there were additional, structural reasons to worry whether pollsters would be able to find representative samples of voters.

One problem is that people are harder to reach on the telephone these days. About a third of voters no longer have a land line, and many of those who have them don’t pick up calls from strangers. So modern polling companies have to work harder to find voters willing to answer questions, then have to guess which of these respondents will actually show up and vote.

So it may come as a surprise that, collectively, polling companies did quite well during this election season. Although there was a small tendency for the pollsters to overestimate Mr. Romney’s share of the vote, a simple average of the polls in swing states produced a very accurate prediction of the Electoral College outcome. Notably, the most accurate polls tended to be done via the Internet, many by companies new to this field. That’s geek victory No. 1.

This relatively accurate polling data provided the raw material for the second group of election pioneers: poll analysts like Nate Silver, who writes the FiveThirtyEight blog for The New York Times, as well as Simon Jackman at Stanford, Sam Wang at Princeton and Drew Linzer at Emory University.

What do poll analysts do? They are like the meteorologists who forecast hurricanes. Data for meteorologists comes from satellites and other tracking stations; data for the poll analysts comes from polling companies. The analysts’ job is to take the often conflicting data from the polls and explain what it all means.

Worry about the reliability of the polling data led to widespread skepticism, or even outright hostility, toward poll analysts. The phrase “garbage in, garbage out” was one of the more polite criticisms bouncing around the Internet in the days before the election.

Because the polls were not, in fact, garbage, the first job of a poll analyst was quite easy: to average the results of the various polls, weighing more reliable and recent polls more heavily and correcting for known biases. (Some polls consistently project higher voter shares for one party or the other.)

A harder but more valuable task is to help readers translate the polling data into forecasts of the probability of victory. In Florida, where the final polls showed essentially a tie, according to Mr. Silver’s weighting method, it’s easy to see why he said the chance of either candidate winning the state was 50 percent. Ultimately, President Obama would very narrowly carry the state.

But what about North Carolina, where Mr. Silver projected that Mitt Romney would get 50.6 percent of the vote and President Obama, 48.9 percent? Looking at that very small difference, what probability would you have assigned to a Romney victory in that state?

Most people would guess something very close to 50-50. But not a good numbers guy. By looking back at previous elections with polling data this close, Mr. Silver estimated that Mr. Romney’s chances of winning North Carolina were 74 percent, a number that may seem surprisingly high. (Mr. Romney won the state.)

The slightly larger but still seemingly tiny lead that the president held in Ohio, another swing state, led poll analysts to predict that the chance of an Obama victory in Ohio was around 90 percent. And because Mr. Romney would have to win several such states with small Obama leads in order to prevail in the Electoral College, the analysts ended up with similarly high degrees of confidence in an overall Obama victory. They ended up predicting the Electoral College outcome almost exactly right, especially if you consider the final outcome in Florida to be a virtual tie, as they had projected.

Pundits making forecasts, some of whom had mocked the poll analysts, didn’t fare as well, and many failed miserably. George F. Will predicted that Mr. Romney would win 321 electoral votes, which turned out to be very close to President Obama’s actual total of 332. Jim Cramer from CNBC was nearly as wrong in the opposite direction, projecting that the president would win 440 electoral votes.

There is a lesson here. When it comes to assessing the chances of some complicated combination of events, gut feelings are pretty much useless. Pundits are no better at forecasting election outcomes than they would be at predicting the final path of a hurricane. Smart pundits should consider either abandoning this activity, or consulting with the geeks before rendering their guesses.

The third set of folks who deserve recognition in this election cycle were a group of young people working in a windowless room at Obama headquarters, affectionately known as the cave. They were part of the effort by the numbers-oriented campaign manager, Jim Messina, to maximize turnout.

THERE are two basic parts of an election campaign. The first comes under the category of messaging — deciding what a candidate should say and what ads to run. Most of the commentary we read about elections focuses on this component.

The second part is turnout, and in some ways is even more important. Here is a simple bit of math that you don’t have to be a geek to understand: It doesn’t matter which candidate a person prefers unless that person shows up and votes.

Pundits will debate for eternity which campaign did a better job of communicating its message, but there is no doubt which campaign won the turnout contest. Young, black and Hispanic voters all turned out in higher numbers than expected, and they often supported President Obama.

Much was made of the big Obama advantage in field offices in swing states. But those field offices would have been little good to the campaign without modern tools to find potential voters, have them register and encourage them to vote. In the weeks leading up to the election, the Obama canvassers had accurate lists of potential voters and field-tested scripts for their contacts with voters. This explains in part why Democrats were such heavy users of early voting.

By contrast, Project Orca, a get-out-the-vote computer program for the Romney campaign that wasn’t designed to be used until Election Day, reportedly had some bugs.

There should be something reassuring about this Obama campaign efficiency to all Americans, even those who supported Mr. Romney based on his success in business. When it came to the business of running a campaign, it was the former professor and community organizer who had the more technologically savvy organization and made more effective use of its resources, including geek power.

Richard H. Thaler is a professor of economics and behavioral science at the Booth School of Business at the University of Chicago. He was an informal adviser to the Obama campaign.

Beware the Smart Campaign

Zeynep Tufekci, The New York Times, November 16, 2012


“I AM not a number. I am a free man!” was the famous cry of prisoner Number Six, who could never escape his Kafkaesque village on the 1960s television show “The Prisoner.” This is a prescient cry for an era when numbers follow us everywhere. Jim Messina, the victorious Obama campaign manager, probably agrees that you are not a number. That’s because you are four numbers.

The Obama campaign assigned all potential swing-state voters one number, on a scale of 1 to 100, that represented the likelihood that they would support Mr. Obama, and another number for the prospect that they would show up at the polls. A third metric evaluated the odds that an Obama supporter who was an inconsistent voter could be nudged to the polls, and a fourth score estimated how persuadable someone was by a conversation on a particular issue (which was, of course, also determined by crunching more numbers).
Mr. Messina is understandably proud of his team, which included an unprecedented number of data analysts and social scientists. As a social scientist and a former computer programmer, I enjoy the recognition my kind are getting. But I am nervous about what these powerful tools may mean for the health of our democracy, especially since we know so little about it all.

For all the bragging on the winning side — and an explicit coveting of these methods on the losing side — there are many unanswered questions. What data, exactly, do campaigns have on voters? How exactly do they use it? What rights, if any, do voters have over this data, which may detail their online browsing habits, consumer purchases and social media footprints?

How did Mr. Obama win? The message and the candidate matter, of course; it’s easier to persuade voters if your policies are more popular and your candidate more appealing. But a modern winning campaign requires more. As Mr. Messina explained, his campaign made an “unparalleled” $100 million investment in technology, demanded “data on everything,” “measured everything” and ran 66,000 computer simulations every day. In contrast, Mitt Romney’s campaign’s data operations were lagging, buggy and nowhere as sophisticated. A senior Romney aide described the shock he experienced in seeing the Obama campaign turn out “voters they never even knew existed.” And that kind of ability matters: while Mr. Obama did win decisively, the size of his lead in four states that determined the outcome, Florida, Ohio, Virginia and Colorado, was about 400,000 votes — or about 1.2 percent of the eligible voters.

The confluence of marketing and politics goes back a long way. A blizzard of direct mail engineered by political consultants is credited with defeating President Harry S. Truman’s national health care proposal after World War II. The new methods, however, are not just better direct mail. Noxious TV ads and slick mailers are like machetes compared with the scalpels of social-science-based big-data. The crude methods may still work to soften the ground and drown out other voices, but in the end they are still very big sticks. Sometimes they kill the patient — just ask swing-state voters about the TV ads they were bombarded with.

The scalpels, on the other hand, can be precise and effective in a quiet, un-public way. They take persuasion into a private, invisible realm. Misleading TV ads can be countered and fact-checked. A misleading message sent in just the kind of e-mail you will open or ad you will click on remains hidden from challenge by the other campaign or the media. Or someone who visits evangelical Web sites might be carefully shielded from messages about gay rights, and someone who has hostile views toward environmentalism may receive messages stroking that sentiment even if the broader campaign woos the green vote elsewhere.

What I really worry about, though, is that these new methods are more effective in manipulating people. Social scientists increasingly understand that much of our decision making is irrational and emotional. For example, the Obama campaign used pictures of the president’s family at every opportunity. This was no accident. The campaign field-tested this as early as 2007 through a rigorous randomized experiment, the kind used in clinical trials for medical drugs, and settled on the winning combination of image, message and button placement. I agree that his family is wonderful and his daughters are cute. But an increasing role of “likability” factors, which we now understand better how to manipulate, is not good for democracy.

These methods will also end up empowering better-financed campaigns. The databases are expensive, the algorithms are proprietary, the results of experiments by campaigns are secret, and the analytics require special expertise. The Democrats have an early advantage partly because academics and data analysts tend to be Democrats. Money will solve that problem. This will shift power in both parties even more toward the richer campaigns and may well be the final nail in the coffin of public financing for presidential campaigns.
What is to be done? Campaigns should make public every outreach message so we at least know what they are saying. These messages can be placed in a public database like campaign contributions so the other side can be aware of, and have the right to respond to, false claims. Political access to proprietary databases should be regulated to provide an even playing field.

I’m not claiming that the Obama campaign used these methods to mislead. However, the fact that the winning campaign’s “chief data scientist” was previously employed to “maximize the efficiency of supermarket sales promotions” does not thrill me. You should be worried even if your candidate is — for the moment — better at these methods. Democracy should not just be about how to persuade people to vote for one candidate over another by any means necessary.

Zeynep Tufekci is a fellow at the Center for Information Technology Policy at Princeton University.