Smartphones May Enable Smart Cities

Bernie Meyerson, The Financial Times, March 4, 2013

Connected and predictive cities will transform city living, says IBM Fellow Bernie Meyerson

People who live in cities are starting to get a lot more information to help plan their daily lives. Smarter City systems are telling residents about traffic congestion, the time the next bus will arrive, the potential for flooding on a block-by-block basis, and a host of other useful updates.

The same wealth of data enables city service leaders to proactively send crews out to mitigate future damage, such as going out to replace water lines before small leaks become catastrophic sinkholes.

This is happening because cities are harnessing the streams of information from sensors to manage municipal systems better. Cities are complex systems of systems that have always generated vast amounts of information. Information flowing in from many small subsystems, such as local flood catch-basins, is integrated with other sources to provide guidance on the overall operation of a city's storm sewer system. This integrated data, from such systems of systems, is then given to city administrators in useful formats that can transform city living for the better.

The need for cities to maximise the efficient use of infrastructure and help residents be more productive without spending vast sums on new facilities is driving this process. The more efficient use of data and predictive analytics promises to help accomplish that goal.

In some cases, cities have moved well beyond just monitoring events in real time. They have begun to accurately predict future events likely to cause damage, and take action to avoid it. Predicting and altering the future may sound like science fiction, but it is here today and represents the future for urban management.
This feat begins with gathering as much data as possible on the issues that need to be addressed. Using the "internet of things", sensors embedded in a wide array of systems serving the public (traffic signals, buses, trains, parking spaces, water systems) report the status of the system they are monitoring via the Internet.
The number of devices such as these exceeded the number of people on the internet in 2008, and will reach 50bn in 2020, according to an article in the 8th November issue of the Journal of Sensor and Actuator Networks. Cities now gather the critical data required to build the models and predictive abilities to understand and alter the future for the better.

All these devices can constantly send information to central command posts where digital streams of data can be analysed and used to alert emergency responders or divert traffic in case of trouble.

Sensors that instantly detect a broken down subway car on a busy track can give repair crews valuable extra minutes of work time to correct matters. However, a true game-changer is the use of sensors to show that a bearing on a train car is running warmer than usual, allowing analytics software to alert maintenance workers and eliminate an impending problem long before a dangerous failure occurs.

IBM, has been using predictive analytics to help cities function better for almost a decade. For example, we have helped city police in Richmond, Virginia deploy their forces to predicted trouble spots and avert crimes. We have helped schools in Mobile, Alabama identify potential dropouts and target them for additional remediation.

In Rio de Janeiro, the city has built a sophisticated central command center in preparation for the World Cup in 2014, and Rio's mayor is counting on advanced analytics to monitor the deluge of data coming in from the region and spot problems long before any one individual could.

Rio currently employs sophisticated weather prediction systems capable of forecasting weather for areas as small as one square kilometer to alert city officials to the potential for heavy rainfall to cause highly localised mud slides in a specific neighborhood located on a steep mountain side. As the predictions are very precise, public safety officials can concentrate their safety teams at key locations, alerting residents to the potential danger and moving to evacuate them if deemed appropriate.

With super-cities of tens of millions of people now emerging, at a scale and complexity never before encountered, this ability to focus resources using deep understanding of the "big picture" becomes ever more critical. In 2010, for the first time in history, more people on the planet lived in cities than lived in rural areas. More importantly, in the next 20 years, some 2bn more people will move into cities, marking one of the most significant migrations in human history.

In some countries, India and China as examples, governments are building new cities to accommodate this transformation. However, on a worldwide basis, most migrants will move to existing cities where they may have personal relationships, prospects of a job, or some other incentive.

In those existing cities, the coming population growth will challenge even the most skilled city managers, and the legacy infrastructure of many such cities is not equipped with the ability to gather the data needed to guide city operations. Despite the seeming lack of data gathering infrastructure, another option exists given the shift from desktop computing to mobile devices and tablets.

Mobile technology has advanced dramatically in the past decade to the point where a lightweight smartphone may gather all manner of data, such as its location, velocity, local temperature and background noise levels. Furthermore, in the near future, additional capabilities are easily envisioned, where sensors could be embedded to detect both particulate and chemical pollution, local lighting conditions, vibration levels, and more.
This will enable a new era of participatory engagement by a city's population, where the very data required to optimise city operations is in part willingly provided by its citizens via smartphone apps.

It also raises complex societal issues that must be addressed as we face this new era, but similarly provides an unprecedented opportunity for the citizenry of a community to take an active role in the betterment of their community, not only in the reporting of critical environmental and transportation data, but in taking an active role in mitigating the very challenges their own data identifies.

It will be a cooperative and informed effort, with information technology enabling societal transformation, to address the growing challenges facing our cities in the coming decades.

Bernie Meyerson is an IBM fellow and vice president for innovation

OF INTEREST: Max Levchin (Former CTO of Paypal)'s keynot at DLD on Big Data

This is the approximate text of the keynote I gave at DLD13 today in Munich. I felt my delivery of this (admittedly, relatively dense) material was not the best, but the content is crucially important. To that end, I am posting the notes here. They were edited for grammar beyond the basics, so you will have to forgive the occasional fifth-grade prose.

Max Levchin, January 21, 2013
Many of today’s big data companies are trying to tackle problems that just aren’t nearly big enough. 

Most are focused on marginally improving existing, digital businesses, but I believe the next big wave of opportunities exists in centralized processing of data gathered from primarily analog systems.

At PayPal, where I was the CTO, we succeeded because we gained deep understanding of the immense quantities of behavioral data that we captured in processing millions of transactions per day. We learned so much about our customers, that we could predict their intentions, and prevent vast majority of intentional fraud. 

At HVF, the project I began in 2011, we seek to create businesses that improve the analog, real world through deep understanding of data. I will tell you more about that in a bit, but first let me expand on what I mean by “digital sensors over analog data.”

Collaborative consumption is a huge current trend. To me, it started to make sense with Über — idling black limo cars put to better use. Über (where I am now an investor) created a simple piece of software to send the idle ones to price-insensitive consumers in need of a cab with a bit of flair. 

The world of real things is very inefficient: slack resources are abundant, so are the companies trying to rationalize their use. Über, AirBnB, Exec, GetAround, PostMates, ZipCar, Cherry, Housefed, Skyara, ToolSpinner, Snapgoods, Vayable, Swifto…it’s an explosion! What enabled this? Why now? It’s not like we suddenly have a larger surplus of black cars than ever before.

Examine the DNA of these businesses: resource availability and demand requests — highly analog, as this is about cars, drivers, and passengers — is captured at the edge, automatically where possible, then transmitted and stored, then processed centrally. Requests are queued at the smart center, and a marketplace/auction is used to allocate them, matches are made and feedback is given in real time. 

Utilization per dollar goes up, because there is simply less idle time! So efficient is this new approach in fact that limo companies started hiring drivers to drive for Über only.  

I am ignoring some key details, like the need for mutual trust between participants that is at least initially enabled by the presence of a trusted third party, and the feedback loop from consumers of the service, but at its core, these business all look very similar.

A key revolutionary insight here is not that the market-based distribution of resources is a great idea — it is the digitalization of analog data, and its management in a centralized queue to create amazing new efficiencies.

Consider the cab calling experience vs Über. When dialing up a cab, you are managing your own spot in the queue. If you hang up in anger, you go back to the end. If you stay on, listening to the on-call tune, you may be an infinite loop, as there is no feedback! 

And even if you are willing to pay a hundred times more than everyone else waiting ahead of you in line to speak to dispatch, you never get to express that demand. The data exists in an analog-only format, and it moves at analog-only speeds.

With Über, the queue can be managed centrally (because the information is converted to a digital format at the edge) with nearly complete transparency to you — you know when the resources you want become available, you know how long you have to wait for them, and most importantly, you are generally assured through feedback that you have not been forgotten or ignored.

So why is all this now? Cheap digital sensors over analog resources (cars, houses, humans, etc) — AT&T pays for your GPS… so you’d consume more data of course. Mobile broadband, of course. Even more key is the critical mass of pretty smart devices that are clients for real-time participation in queue management with feedback. It’s the smart-enough terminal model. 

As an aside, consumers want sexy sensors that give visual feedback to motivate action (and more data) — like the Nike Fuel band. Machines, on the other hand, just want cheap sensors to get more data from.

I sometimes imagine the low-use troughs of sinusoidal curves utilization of all these analog resources being pulled up, filling up with happy digital usage. 

Private jets spend 1h per day on average in the air. BlackJet promises to make that number closer to the commercial average — 10h/day. And not everyone in a suburban neighborhood needs their own lawn mower — they need an app to schedule one of the local kids to come by and take care of their overgrown grass. 

The defensibility of these businesses lies in their ability to build a network effect — a network effect of data. Once a business understands substantially more than any one of the resources managed in their queue, it’s effectively impossible to compete with them on price — they can always see more of the usage as it happens, and price it more efficiently, pushing any competition out. 

So what other businesses can we expect to emerge in analog-data-driven, central-intelligence queue marketplace businesses? Some interesting ones are probably already being built: a market for private neighborhood security (off-duty cops)? An auction for short-term patent licenses (litigator included)? 

Technology already enables efficient redistribution for your spare change: it’s Kickstarter and AngelList. We will definitely see dynamically-priced queues for confession-taking priests, and therapists!

How about dynamic pricing for brain cycles? We have been maximizing utilization of very high-value, very low-frequency specialists — today you can already rent the brain of a data-mining genius via Kaggle by the hour, tomorrow by brain-hour. Just like the SETI@Home screensaver “steals” CPU cycles to sift through cosmic radio noise for alien voices, your brain plug firmware will earn you a little extra cash while you sleep, by being remotely programmed to solve hard problems, like factoring products of large primes.

There is also a neat symmetry to this analog-to-digtail transformation — enabling centralization of unique analog capacities. As soon as the general public is ready for it, many things handled by a human at the edge of consumption will be controlled by the best currently available human at the center of the system, real time sensors bringing the necessary data to them in real time. The freshest, smartest pilot, most familiar with the particular complicated airport will land your plane — via remote control.   

So what’s after that? This is where it gets really interesting. These new modes of operation — remote controlled cars and planes flown by pilots you can’t see, rides in quasi-cabs with people you have never met, legal advice from lawyers whose license you cannot really check. This is going to add a huge amount of new kinds of risks. 

But as a species, we simply must take these risks, to continue advancing, to use all available resources to their maximum. Yet these risks are real, and they cannot be ignored.

The way to deal with risk is of course some form of insurance. Modeling loss from observed past events is hardly news, but dynamically changing the price of the service to reflect individual risk is a big deal. My expectation is that next decade we will see an explosion of insurance and insurance-like products and services — leveraging those very same network effects of data, providing truly dynamic resource pricing and allocation. 

This is the purpose of my new project, HVF — to bring these new products to those that will benefit from them the most. We see data-driven understanding and pricing of risk as the great opportunity to improve lives. The reason the notion of analog is so important here is because it ultimately also means “human.”

Understanding the changing risk profile of a person can deliver to them amazing opportunities they wouldn’t have in today’s world: inferring that a particular college grad is financially responsible by looking at their tweets could allow them to buy their first house on credit, at 21, without any history, and looking at someone’s heart rate monitor data could make their cardiovascular healthcare cost-free. 

These are not non-controversial topics — privacy, unfair discrimination, built-in biases are all possible, and we must be thoughtful and diligent in how we go about bringing this future. But I believe that what we can enable with data insights greatly outweighs the downsides. 

Here is what I mean, by way of a simple example.

On a Sat morning, I load my two toddlers into their respective child seats, and my car’s in-wheel strain gauges detect the weight difference and reports that the kids are with me in a moving vehicle to my insurance via a secure message through my iPhone. The insurance company duly increases today’s premium by a few dollars. 

My keepHonest app sees this too and immediately offers me up as a customer to a few competing insurance companies in the background, but nobody is willing to charge me less right now, and the phone chirps sadly to let me know I’m now paying a higher premium. Safer, but more expensive. 

But In a few hours, my car’s GPS duly reports to my insurer that I only drove two miles to the park, never sped and, and observed all traffic signs. My phone now chirps happily: not only has my rate been discounted, several companies are offering me a deal on insurance!

So to conclude: I believe that in the next decades we will see huge number of inherently analog processes captured digitally. Opportunities to build businesses that process this data and improve lives will abound. 

NYT: Sure, Big Data Is Great. But So Is Intuition.

Steve Lohr, The New York Times, December 29, 2012

IT was the bold title of a conference this month at the Massachusetts Institute of Technology, and of a widely read article in The Harvard Business Review last October: “Big Data: The Management Revolution.”

Andrew McAfee, principal research scientist at the M.I.T. Center for Digital Business, led off the conference by saying that Big Data would be “the next big chapter of our business history.” Next on stage was Erik Brynjolfsson, a professor and director of the M.I.T. center and a co-author of the article with Dr. McAfee. Big Data, said Professor Brynjolfsson, will “replace ideas, paradigms, organizations and ways of thinking about the world.”

These drumroll claims rest on the premise that data like Web-browsing trails, sensor signals, GPS tracking, and social network messages will open the door to measuring and monitoring people and machines as never before. And by setting clever computer algorithms loose on the data troves, you can predict behavior of all kinds: shopping, dating and voting, for example.

The results, according to technologists and business executives, will be a smarter world, with more efficient companies, better-served consumers and superior decisions guided by data and analysis.

I’ve written about what is now being called Big Data a fair bit over the years, and I think it’s a powerful tool and an unstoppable trend. But a year-end column, I thought, might be a time for reflection, questions and qualms about this technology.

The quest to draw useful insights from business measurements is nothing new. Big Data is a descendant of Frederick Winslow Taylor’s “scientific management” of more than a century ago. Taylor’s instrument of measurement was the stopwatch, timing and monitoring a worker’s every movement. Taylor and his acolytes used these time-and-motion studies to redesign work for maximum efficiency. The excesses of this approach would become satirical grist for Charlie Chaplin’s “Modern Times.” The enthusiasm for quantitative methods has waxed and waned ever since.

Big Data proponents point to the Internet for examples of triumphant data businesses, notably Google. But many of the Big Data techniques of math modeling, predictive algorithms and artificial intelligence software were first widely applied on Wall Street.

At the M.I.T. conference, a panel was asked to cite examples of big failures in Big Data. No one could really think of any. Soon after, though, Roberto Rigobon could barely contain himself as he took to the stage. Mr. Rigobon, a professor at M.I.T.’s Sloan School of Management, said that the financial crisis certainly humbled the data hounds. “Hedge funds failed all over the world,” he said.

THE problem is that a math model, like a metaphor, is a simplification. This type of modeling came out of the sciences, where the behavior of particles in a fluid, for example, is predictable according to the laws of physics.

In so many Big Data applications, a math model attaches a crisp number to human behavior, interests and preferences. The peril of that approach, as in finance, was the subject of a recent book by Emanuel Derman, a former quant at Goldman Sachs and now a professor at Columbia University. Its title is “Models. Behaving. Badly.”

Claudia Perlich, chief scientist at Media6Degrees, an online ad-targeting start-up in New York, puts the problem this way: “You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.”

The bubble that concerns Ms. Perlich is not so much a surge of investment, with new companies forming and then failing in large numbers. That’s capitalism, she says. She is worried about a rush of people calling themselves “data scientists,” doing poor work and giving the field a bad name.

Indeed, Big Data does seem to be facing a work-force bottleneck.

“We can’t grow the skills fast enough,” says Ms. Perlich, who formerly worked for I.B.M. Watson Labs and is an adjunct professor at the Stern School of Business at New York University.

A report last year by the McKinsey Global Institute, the research arm of the consulting firm, projected that the United States needed 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired.

Thomas H. Davenport, a visiting professor at the Harvard Business School, is writing a book called “Keeping Up With the Quants” to help managers cope with the Big Data challenge. A major part of managing Big Data projects, he says, is asking the right questions: How do you define the problem? What data do you need? Where does it come from? What are the assumptions behind the model that the data is fed into? How is the model different from reality?

Society might be well served if the model makers pondered the ethical dimensions of their work as well as studying the math, according to Rachel Schutt, a senior statistician at Google Research.

“Models do not just predict, but they can make things happen,” says Ms. Schutt, who taught a data science course this year at Columbia. “That’s not discussed generally in our field.”

Models can create what data scientists call a behavioral loop. A person feeds in data, which is collected by an algorithm that then presents the user with choices, thus steering behavior.

Consider Facebook. You put personal data on your Facebook page, and Facebook’s software tracks your clicks and your searches on the site. Then, algorithms sift through that data to present you with “friend” suggestions.

Understandably, the increasing use of software that microscopically tracks and monitors online behavior has raised privacy worries. Will Big Data usher in a digital surveillance state, mainly serving corporate interests?
Personally, my bigger concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart. That was a theme of a book by Eli Pariser, titled “The Filter Bubble: What the Internet Is Hiding From You.”

It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?

At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.

“I don’t worship the machine,” she said.

NYRB: How He Got It Right

Andrew Hacker, New York Review of Books, January 10, 2013

The Signal and the Noise: Why So Many Predictions Fail—But Some Don't
by Nate Silver
Penguin, 534 pp., $27.95                                                  
The Physics of Wall Street: A Brief History of Predicting the Unpredictable
by James Owen Weatherall
Houghton Mifflin Harcourt, 286 pp., $27.00                                                  
Antifragile: Things That Gain from Disorder
by Nassim Nicholas Taleb
Random House, 519 pp., $30.00         
                                         
1.
Nate Silver called every state correctly in the last presidential race, and was wrong about only one in 2008. In 2012 he predicted Obama's total of the popular vote within one tenth of a percent of the actual figure. His powers of prediction seemed uncanny. In his early and sustained prediction of an Obama victory, he was ahead of most polling organizations and my fellow political scientists. But buyers of his book, The Signal and the Noise, now a deserved best seller, may be in for something of a surprise. There's only a short chapter on predicting elections, briefer than ones on baseball, weather, and chess. In fact, he's written a serious treatise about the craft of prediction—without academic mathematics—cheerily aimed at lay readers. Silver's coverage is polymathic, ranging from poker and earthquakes to climate change and terrorism.

We learn that while more statistics per capita are collected for baseball than perhaps any other human activity, seasoned scouts still surpass algorithms in predicting the performance of players. Since poker depends as much on luck as on skill, professionals make a living by having well-heeled amateurs at the table. The lesson from a long chapter on earthquakes is that while we're good at measuring them, they're "not really predictable at all." Much the same caution holds for economists, whose forecasts of next year's growth are seldom correct. Their models may be elegant, Silver says, but "their raw data isn't much good."

The most striking success has been in forecasting where hurricanes will hit. Over the last twenty-five years, the ability to pinpoint landfalls has increased twelvefold. At the same time, Silver says, newscasts purposely overpredict rain, since they know their listeners will be grateful when they find they don't need umbrellas. While he doesn't dismiss "highly mathematical and data-driven techniques," he cautions climate modelers not to give out precise changes in temperature and ocean levels. He tells of attending a conference on terrorism at which a Coca-Cola marketing executive and a dating service consultant were asked for hints on how to identify suicide bombers.

Much is made of ours being an era of Big Data. Silver passes on an estimate from IBM that 2.5 quintillion (that's seventeen zeros) new bytes (sequences of eight binary digits that each encode a single character of text in a computer) of data are being created every day, representing everything from the brand of toothpaste you bought yesterday to your location when you called a friend this morning. Such information can be put together to fashion personal profiles, which Amazon and Google are already doing in order to target advertisements more accurately. Obama's tech-savvy workers did something similar, notably in identifying voters who needed extra prompting to go to the polls.¹

Those daily quintillions are what led to Silver's title. "Signals" are facts we want and need, such as those that will help us detect incipient shoe bombers. "Noise" is everything else, usually extraneous information that impedes or misleads our search for signals. Silver makes the failure to forecast September 11 a telling example.

But first, The Signal and the Noise is in large part a homage to Thomas Bayes (1701–1761), a long-neglected statistical scholar, especially by the university departments concerned with statistical methods. The Bayesian approach to probability is essentially simple: start by approximating the odds of something happening, then alter that figure as more findings come in. So it's wholly empirical, rather than building edifices of equations.² Silver has a diverting example on whether your spouse may be cheating. You might start with an out-of-the-air 4 percent likelihood. But a strange undergarment could raise it to 50 percent, after which the game's afoot. This has importance, Silver suggests, because officials charged with anticipating terrorist acts had not conjured a Bayesian "prior" about the possible use of airplanes.

Silver is prepared to say, "We had some reason to think that an attack on the scale of September 11 was possible." His Bayseian "prior" is that airplanes were targeted in the cases of an Air India flight in 1985 and Pan Am's over Lockerbie three years later, albeit using secreted bombs, plus in later attempts that didn't succeed. At the least, a chart with, say, a 4 percent likelihood of an attack should have been on someone's wall. Granted, what comes in as intelligence is largely "noise." (Most intercepted conversations are about plans for dinner.) Still, in the summer of 2001, staff members at a Minnesota flight school told FBI agents of a Moroccan-born student who wanted to learn to pilot a Boeing 747 in midair, skipping lessons on taking off and landing. Some FBI agents took the threat of Zacarias Moussaoui seriously, but several requests for search and wiretap warrants were denied. In fact, an instructor added that a fuel-laden plane could make a horrific weapon. At the least, these "signals" should have raised the probability of an attack using an airplane, say, to 15 percent, prompting visits to other flight schools.

Silver's "mathematics of terrorism" may be stretching the odds a bit. Many of those daily quintillion digits flow into the FBI and CIA, not to mention the departments of State and Defense. To follow all of them up is patently impossible, with only a small fraction getting even a cursory second look. It's bemusing that two recent revelations of marital infidelity—Eliot Spitzer and David Petraeus—arose from inquiries having other purposes. Plus there's the question of how many investigators and investigations we want to have, as more searching will inevitably touch more of us.
Yet in the end, Silver's claims are quite modest. Indeed, he could have well phrased his subtitle "why most predictions fail." It's simply because "the volume of information is increasing exponentially."

There is no reason to conclude that the affairs of man are becoming more predictable. The opposite may well be true. The same sciences that uncover the laws of nature are making the organization of society more complex.

I'd only add that it's not just what sciences are finding that makes the world seem more complex. Shifts in the structure of occupations, abetted by more college degrees, have increased the number of positions deemed to be professional. If entrepreneurs tend to be assessed by how much money they amass, professionals are rated by the presumed complexity of what they know and do. So to retain or raise an occupation's status, tasks are made more mysterious, usually by taking what's really simple and adding obfuscating layers. The very sciences Silver cites—especially those of a social sort—rank among the culprits.

2.
Nate Silver is known not so much for predicting who will win elections, but for how close he comes to the actual results. His final 2012 forecast gave Obama 50.8 percent of the popular vote, almost identical with his eventual figure of 50.9 percent. This kind of precision is striking. A more typical projection may warn that it has a three-point margin of error either way, meaning a candidate accorded 52 percent could end anywhere between 55 percent and 49 percent. Or, fearful of making a wrong call, as in 2000, polling agencies will claim that the outcome is too close to foretell. Still, it's too early to hail a new statistical science. As can be seen in Table A, Rasmussen's and Gallup's final polls predicted that Romney would be the winner, while the Boston Herald gave its state's senate race to Scott Brown.

In fact, I am impressed when polls come even close. To start, what's needed is a reliable cross-section of people who will actually vote. In 2008, only 62 percent of eligible citizens cast ballots. In 2012, even fewer did. Not surprisingly, some people who seldom or never vote will still claim they'll be turning out. Testing them ("can you tell me where your polling place is?") can be time-consuming and expensive. And there are those who don't report their real choices. But much more vexing is finding people willing to cooperate. According to a recent Pew Research Center report, only several years ago, in 1997, about 90 percent of a desired sample could be reached in person or at home by telephone, and 36 percent of them were amenable to an interview.

Today, with fewer people at home or picking up calls, and increasing refusals from those who do, the rates are down to 62 percent and 9 percent.³ So the polls must create a model of an electorate from the slender slice willing to give them time. Yet despite these hurdles, the Columbus Dispatch called Ohio's result perfectly, using 1,501 respondents from the state's 5,362,236 voters (the figures available on December 7).
Election polls are unique in at least two ways. First, they aim to tell us about a concrete act—a cast ballot—to be performed in an impending period of time. (Each year, more of us vote early.) It's hard to think of other surveys that try to anticipate what a huge pool of adults will do. Second, how well a poll did becomes known once the votes are counted. So we find Nate Silver got it right and Rasmussen and Gallup didn't. But a poll's accuracy is only a historic curiosity after the returns are in. That is, it didn't tell us anything lasting; just about a foray into forecasting during some months when a lot of us were wondering how events would turn out. 
Other polls tell us about something less fleeting: the opinions people hold on public issues and personal matters.

But with polls on opinions—military spending, say, or the provision of contraceptives—there's seldom a subsequent vote that can validate findings. (To an extent, this is possible when there are statewide votes on issues like affirmative action and gay marriage.) A recourse is to compare a series of surveys that ask similar questions.

Yet as Table B shows, responses on abortion have been quite varied. What could be called the "pro-choice" side ranges across twenty-three percentage points. Certainly, how the question is phrased can skew the answers. CBS's 42 percent agreed that abortion should be "generally available," while Gallup's 25 percent were supporting the view that abortion should be "always legal," and The Washington Post's 19 percent were for abortion to be "legal in all cases." The short answer is that apart from the severe anti side, polling can't give us specific figures on where most adults line up on abortion. Or, for that matter, any issue.

What goes on in the American mind remains a mystery that sampling is unlikely to unlock. In my estimate, the 65,075,450 people who chose Barack Obama and Joseph Biden over Mitt Romney and Paul Ryan were mainly expressing a moral mood, a feeling about the kind of country they want. I'd like to see Nate Silver using his statistical talents to explore such surmises.

We've been informed that 55 percent of women supported Obama, rising to 67 percent of those who are single, divorced, or widowed. Obama also secured 55 percent among holders of postgraduate degrees, and 69 percent of Jewish voters. But how can we know? Voting forms don't ask for marital status or religion. The answer is that these and similar figures were extrapolated from a national sample of 26,563 voters, approached just after they cast their ballots or telephoned later in the day, by an organization called Edison Research.

The figures I've cited and others on the list look plausible to me. Still, there's no way to check them; moreover, the Edison survey is the only post-election one that was done. So here's a caveat: Jews are so small a fraction of the electorate that there were only 241 in the sample. Thus the abovementioned 69 percent comes with a seven-point margin of error either way, a caveat not noted in most media accounts.
Nate Silver doesn't conduct his own polls. Rather, he collects a host of state and national reports, and enters them in a database of his own devising. Combining samples from varied surveys gives him a much larger pool of respondents and the potential for a more reliable profile. Of course, Silver doesn't simply crunch whatever comes in. He factors in past predictions and looks for slipshod work, as when the Florida Times-Union on election eve gave the state to Romney, based on 681 interviews. He pays special attention to demographic shifts, such as a surge in registrations with Hispanic names. His model also draws on the Cook Political Report, which actually meets informally with candidates to assess their electoral appeal. In September, Silver set the odds of Obama's winning at 85 percent, enough to withstand a dismal performance in the first debate, which hadn't yet occurred.

3.
Early in The Signal and the Noise, Silver alludes to Isaiah Berlin's trope about hedgehogs and foxes. Hedgehogs know "one big thing," while foxes know "many little things." But there's more. Hedgehogs display a disconcerting certainty that their one idea will put everything straight, whether on intellectual questions or in the working world. Moreover, Berlin warned, hedgehogs can cause a lot of damage when their nostrums are applied. Silver sees himself as a more modest fox, willing to draw on varied approaches to get his job done. So The Signal and the Noise doesn't end with a crescendo, but actually stresses the quite limited ambit of our power to predict.

James Weatherall is an unabashed hedgehog, propounding a single idea with an uncommon confidence. After training in physics, philosophy, and mathematics, he now teaches logic and the philosophy of science at the University of California's Irvine campus. With The Physics of Wall Street, he is taking his training even further: to finance in its preeminent location.

He is a man with a mission: to bring a heightened rationality to investment decisions. His book opens with an admiring visit to a hedge fund where a third of the employees have doctorates in physics, mathematics, statistics, even astronomy. Indeed, their rarefied insights are what's wanted; "PhDs in finance need not apply." In fact, there's a niche Wall Street sector called "quant firms," which reserve key positions for holders of advanced degrees.

Weatherall would have this perspective pervade the entire financial industry. He succinctly states his one big idea: "Insights that are commonplace in physics…are useful in studying virtually anything." In one sense, Wall Street's products have a physical character. Collateralized debt obligations, credit default swaps, and initial public offerings appear on paper or as electronic impulses. But Weatherall means more than this. Markets, he believes, are subject to physical laws. His star witness is Louis Bachelier, a French mathematician at the turn of the last century, who used Brownian motion to evaluate stock options. After that, we hear how ideas from such mathematicians as Jacob Bernoulli and Benoît Mandelbrot can be applied to mitigating risks and minimizing uncertainty. Not least of their influence has been to entrench mathematics in MBA programs, Wall Street's principal recruiting pool.

But the book is less a celebration of the past than a prospectus for the future. Weatherall anticipates "a breakthrough in our ability to identify the underlying chaotic patterns lurking in market data," that is, to impose order on the "noise" that bedevils Silver. In a similar vein, he foresees strides "in predicting financial calamity using mathematical techniques," perhaps like Russia's default in 1998, which some Nobel economists didn't see coming. He also hopes that "studies of psychology and human behavior" can be framed so that they are "symbiotic with mathematical approaches to economics." Here he foresees doing better than Newton, who confessed, "I can calculate the movements of stars, but not the madness of man." Almost everyone favors rigor and unlocking more mysteries. That's why most of us support science. Can there be anything amiss in feeling optimistic about crossing new frontiers?

What isn't explained is whether melding physics with finance will bring more benefits for everyone, or only give an advantage to those who use the techniques—like high-speed trading, if you own a mainframe computer. We can agree there's a lot of irrationality—not to say exuberance—in the investment world. But it's not clear if Weatherall is saying that decisions based on Bernoulli will allocate capital more efficiently, and hence serve the commonweal. When physics enters medicine, as with MRIs, we can have a reasonable hope that patients will benefit as much as their doctors. But when quants were riding high on Wall Street, they were hired only to give their own firms an edge over the competition.

Alluding to the collapse of Bear Stearns and Lehman Brothers, the housing bubble, and the October 2008 crash, Weatherall concedes that "the misuse of mathematical models played a role in this crisis." Still, his implication here is that the models themselves didn't contribute to the downfall; it was that they were somehow mishandled. The ultimate problem with hedgehogs is hubris. In this case, it's the assumption that the quality of our thought can be enhanced by new methodologies. The word "sophistication" recurs on almost every page of The Physics of Wall Street, as if to affirm that higher powers are present.
True, the discovery of the calculus enabled us to build planes that travel faster than the speed of sound. But thus far I've found scant evidence that mathematics and physics have a capacity to give us a deeper understanding of human and social behavior. Of course, we should be open to new findings. Still, that differs from proclaiming that "what we do know for sure is that there will be a next major advance, and…we will understand markets more clearly than we do today." Here he seems to be saying that the physical sciences can tell us how to avoid the crashes and crises we now periodically face. If that's so, I wish Weatherall had listed some warnings of coming "calamities" based on his physical laws, such as the impending college loan bubble: When will it burst, and how widespread will the fallout be?

Nassim Nicholas Taleb in Antifragile calls such certainty "the error of naive rationalism." And it's a naiveté with consequences. There's no doubt that the quants who bundled bad mortages, adding algorithms to rate them AAA, helped to give us the current recession.⁴ But just as culpable were their nonmathematical superiors who allowed them such rein. So there's a broader issue. Financial firms want to be at the cutting edge, which now means having a bevy of Ph.D.s, just as at other times and places, enterprises might feel they should have an accredited gypsy with tarot cards. We are coming close to deifying anything smacking of STEM—science, technology, engineering, and mathematics—whether for staying ahead of China or cutting-edge careers for our young people. If firms need workers adept in algebra, then such instruction should be available. But to rely on physics and mathematics for deciphering human behavior, in markets or elsewhere, can only bring blind alleys. Less of the world than we might like is, as Taleb puts it, "academizable, rationalizable, formalizable, theoretizable." When such rubrics crowd out more discursive thinking, we all lose.

1.      

See Michael Scherer, "Inside the Secret World of the Data Crunchers Who Helped Obama Win," Time, November 7, 2012, and Nate Silver, "In Silicon Valley, Technology Talent Gap Threatens GOP Campaigns," The New York Times, November 28, 2012. ↩

2.      

See Sharon Bertsch McGrayne's superb The Theory That Would Not Die (Yale University Press, 2011). ↩

3.      

"Assessing the Representativeness of Public Opinion Surveys," The Pew Research Center for the People and the Press, May 15, 2012. ↩

4.      

Weatherall barely mentions Scott Patterson's indispensable The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It (Crown Business, 2010). And some stories worth reading: Felix Salmon, "Recipe for Disaster: The Formula That Killed Wall Street," Wired, March 2009; Dennis Overbye, "They Tried to Outsmart Wall Street," The New York Times, March 9, 2009; Julie Creswell, "The Quants are Reeling," The New York Times, August 20, 2010; Pablo Triana, "The Flawed Maths of Financial Models," Financial Times, November 29, 2010. ↩