Steve Lohr The New york Times, February 11, 2012
GOOD with numbers? Fascinated by data? The sound you hear is opportunity knocking.
Mo Zhou was snapped up by I.B.M. last summer, as a freshly minted Yale M.B.A., to join the technology company’s fast-growing ranks of data consultants. They help businesses make sense of an explosion of data — Web traffic and social network comments, as well as software and sensors that monitor shipments, suppliers and customers — to guide decisions, trim costs and lift sales. “I’ve always had a love of numbers,” says Ms. Zhou, whose job as a data analyst suits her skills.
To exploit the data flood, America will need many more like her. A report last year by the McKinsey Global Institute, the research arm of the consulting firm, projected that the United States needs 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired.
The impact of data abundance extends well beyond business. Justin Grimmer, for example, is one of the new breed of political scientists. A 28-year-old assistant professor at Stanford, he combined math with political science in his undergraduate and graduate studies, seeing “an opportunity because the discipline is becoming increasingly data-intensive.” His research involves the computer-automated analysis of blog postings, Congressional speeches and press releases, and news articles, looking for insights into how political ideas spread.
The story is similar in fields as varied as science and sports, advertising and public health — a drift toward data-driven discovery and decision-making. “It’s a revolution,” says Gary King, director of Harvard’s Institute for Quantitative Social Science. “We’re really just getting under way. But the march of quantification, made possible by enormous new sources of data, will sweep through academia, business and government. There is no area that is going to be untouched.”
Welcome to the Age of Big Data. The new megarich of Silicon Valley, first at Google and now Facebook, are masters at harnessing the data of the Web — online searches, posts and messages — with Internet advertising. At the World Economic Forum last month in Davos, Switzerland, Big Data was a marquee topic. A report by the forum, “Big Data, Big Impact,” declared data a new class of economic asset, like currency or gold.
Rick Smolan, creator of the “Day in the Life” photography series, is planning a project later this year, “The Human Face of Big Data,” documenting the collection and uses of data. Mr. Smolan is an enthusiast, saying that Big Data has the potential to be “humanity’s dashboard,” an intelligent tool that can help combat poverty, crime and pollution. Privacy advocates take a dim view, warning that Big Data is Big Brother, in corporate clothing.
What is Big Data? A meme and a marketing term, for sure, but also shorthand for advancing trends in technology that open the door to a new approach to understanding the world and making decisions. There is a lot more data, all the time, growing at 50 percent a year, or more than doubling every two years, estimates IDC, a technology research firm. It’s not just more streams of data, but entirely new ones. For example, there are now countless digital sensors worldwide in industrial equipment, automobiles, electrical meters and shipping crates. They can measure and communicate location, movement, vibration, temperature, humidity, even chemical changes in the air.
Link these communicating sensors to computing intelligence and you see the rise of what is called the Internet of Things or the Industrial Internet. Improved access to information is also fueling the Big Data trend. For example, government data — employment figures and other information — has been steadily migrating onto the Web. In 2009, Washington opened the data doors further by starting Data.gov, a Web site that makes all kinds of government data accessible to the public.
Data is not only becoming more available but also more understandable to computers. Most of the Big Data surge is data in the wild — unruly stuff like words, images and video on the Web and those streams of sensor data. It is called unstructured data and is not typically grist for traditional databases.
But the computer tools for gleaning knowledge and insights from the Internet era’s vast trove of unstructured data are fast gaining ground. At the forefront are the rapidly advancing techniques of artificial intelligence like natural-language processing, pattern recognition and machine learning.
Those artificial-intelligence technologies can be applied in many fields. For example, Google’s search and ad business and its experimental robot cars, which have navigated thousands of miles of California roads, both use a bundle of artificial-intelligence tricks. Both are daunting Big Data challenges, parsing vast quantities of data and making decisions instantaneously.
The wealth of new data, in turn, accelerates advances in computing — a virtuous circle of Big Data. Machine-learning algorithms, for example, learn on data, and the more data, the more the machines learn. Take Siri, the talking, question-answering application in iPhones, which Apple introduced last fall. Its origins go back to a Pentagon research project that was then spun off as a Silicon Valley start-up. Apple bought Siri in 2010, and kept feeding it more data. Now, with people supplying millions of questions, Siri is becoming an increasingly adept personal assistant, offering reminders, weather reports, restaurant suggestions and answers to an expanding universe of questions.
To grasp the potential impact of Big Data, look to the microscope, says Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management. The microscope, invented four centuries ago, allowed people to see and measure things as never before — at the cellular level. It was a revolution in measurement.
Data measurement, Professor Brynjolfsson explains, is the modern equivalent of the microscope. Google searches, Facebook posts and Twitter messages, for example, make it possible to measure behavior and sentiment in fine detail and as it happens.
In business, economics and other fields, Professor Brynjolfsson says, decisions will increasingly be based on data and analysis rather than on experience and intuition. “We can start being a lot more scientific,” he observes.
There is plenty of anecdotal evidence of the payoff from data-first thinking. The best-known is still “Moneyball,” the 2003 book by Michael Lewis, chronicling how the low-budget Oakland A’s massaged data and arcane baseball statistics to spot undervalued players. Heavy data analysis had become standard not only in baseball but also in other sports, including English soccer, well before last year’s movie version of “Moneyball,” starring Brad Pitt.
Retailers, like Walmart and Kohl’s, analyze sales, pricing and economic, demographic and weather data to tailor product selections at particular stores and determine the timing of price markdowns. Shipping companies, like U.P.S., mine data on truck delivery times and traffic patterns to fine-tune routing.
Online dating services, like Match.com, constantly sift through their Web listings of personal characteristics, reactions and communications to improve the algorithms for matching men and women on dates. Police departments across the country, led by New York’s, use computerized mapping and analysis of variables like historical arrest patterns, paydays, sporting events, rainfall and holidays to try to predict likely crime “hot spots” and deploy officers there in advance.
Research by Professor Brynjolfsson and two other colleagues, published last year, suggests that data-guided management is spreading across corporate America and starting to pay off. They studied 179 large companies and found that those adopting “data-driven decision making” achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain.
The predictive power of Big Data is being explored — and shows promise — in fields like public health, economic development and economic forecasting. Researchers have found a spike in Google search requests for terms like “flu symptoms” and “flu treatments” a couple of weeks before there is an increase in flu patients coming to hospital emergency rooms in a region (and emergency room reports usually lag behind visits by two weeks or so).
Global Pulse, a new initiative by the United Nations, wants to leverage Big Data for global development. The group will conduct so-called sentiment analysis of messages in social networks and text messages — using natural-language deciphering software — to help predict job losses, spending reductions or disease outbreaks in a given region. The goal is to use digital early-warning signals to guide assistance programs in advance to, for example, prevent a region from slipping back into poverty.
In economic forecasting, research has shown that trends in increasing or de
creasing volumes of housing-related search queries in Google are a more accurate predictor of house sales in the next quarter than the forecasts of real estate economists. The Federal Reserve, among others, has taken notice. In July, the National Bureau of Economic Research is holding a workshop on “Opportunities in Big Data” and its implications for the economics profession.
Big Data is already transforming the study of how social networks function. In the 1960s, Stanley Milgram of Harvard used packages as his research medium in a famous experiment in social connections. He sent packages to volunteers in the Midwest, instructing them to get the packages to strangers in Boston, but not directly; participants could mail a package only to someone they knew. The average number of times a package changed hands was remarkably few, about six. It was a classic demonstration of the “small-world phenomenon,” captured in the popular phrase “six degrees of separation.”
Today, social-network research involves mining huge digital data sets of collective behavior online. Among the findings: people whom you know but don’t communicate with often — “weak ties,” in sociology — are the best sources of tips about job openings. They travel in slightly different social worlds than close friends, so they see opportunities you and your best friends do not.
Researchers can see patterns of influence and peaks in communication on a subject — by following trending hashtags on Twitter, for example. The online fishbowl is a window into the real-time behavior of huge numbers of people. “I look for hot spots in the data, an outbreak of activity that I need to understand,” says Jon Kleinberg, a professor at Cornell. “It’s something you can only do with Big Data.”
Big Data has its perils, to be sure. With huge data sets and fine-grained measurement, statisticians and computer scientists note, there is increased risk of “false discoveries.” The trouble with seeking a meaningful needle in massive haystacks of data, says Trevor Hastie, a statistics professor at Stanford, is that “many bits of straw look like needles.”
Big Data also supplies more raw material for statistical shenanigans and biased fact-finding excursions. It offers a high-tech twist on an old trick: I know the facts, now let’s find ’em. That is, says Rebecca Goldin, a mathematician at George Mason University, “one of the most pernicious uses of data.”
Data is tamed and understood using computer and mathematical models. These models, like metaphors in literature, are explanatory simplifications. They are useful for understanding, but they have their limits. A model might spot a correlation and draw a statistical inference that is unfair or discriminatory, based on online searches, affecting the products, bank loans and health insurance a person is offered, privacy advocates warn.
Despite the caveats, there seems to be no turning back. Data is in the driver’s seat. It’s there, it’s useful and it’s valuable, even hip.
Veteran data analysts tell of friends who were long bored by discussions of their work but now are suddenly curious. “Moneyball” helped, they say, but things have gone way beyond that. “The culture has changed,” says Andrew Gelman, a statistician and political scientist at Columbia University. “There is this idea that numbers and statistics are interesting and fun. It’s cool now.”
Steve Lohr is a technology reporter for The New York Times.
P Please consider the environment before printing this e-mail.
Sunday, February 12, 2012
Friday, February 10, 2012
A Revolution of Machines Quietly Hums
Chrystia Freeland, Reuters, February 9, 2012
NEW YORK — If you want to get a finger-tip feel for one of the most important transformations in our world today, read “The Fear Index,” Robert Harris’s new thriller.
Mr. Harris has been widely praised for his adept portrayal of the hedge fund universe in which his novel is set. “The greatest pleasure of this book is that it gets the finance right,” cooed Felix Salmon, the Reuters finance blogger whose keyboard often oozes acid.
He is right that Mr. Harris’s hedgies are a welcome and realistic departure from the Masters of the Universe of most popular fiction. For one thing, these are the alpha geeks, the nerdy doctorate-holders whose testosterone is channeled into equations instead of frat-house swagger.
And Mr. Harris knows his super-rich. As they are being wooed to make a billion-dollar investment over a 1995 Latour in a fine Geneva restaurant, an international group of investors energetically discusses the ways that their national governments oppress them.
Dr. Alex Hoffmann, the billionaire hedge fund manager protagonist who is making the pitch for their cash, reflects: “He was remembering now why he didn’t like the rich: their self-pity. Persecution was the common ground of their conversation, like sport or the weather was for everyone else.”
But the most valuable intellectual take-away from this riveting novel is the element that has been widely dismissed by reviewers as familiar and unpersuasive: VIXAL, the intelligent machine at the heart of the story that torments its human creator and may be trying to rule the world. Michiko Kakutani, writing in The New York Times, gave “The Fear Index” a thumbs up, but deemed the machine takeover “silly and contrived.” VIXAL, the evil genius computer, “seems less like a plausible villain than like a metaphor for the greed and heedlessness that overtook Wall Street,” she wrote.
Ms. Kakutani’s dismissal of VIXAL as an unoriginal and uncompelling reworking of the Frankenstein template misses the most important point. VIXAL and the parallel world the computer program creates by interacting with other machines is worth reading about — but not for the usual pulp fiction pleasure of entering a vividly rendered imaginary world.
“The Fear Index” is fun to read because Mr. Harris is a great novelist, but VIXAL is worth getting to know not as a fictional character, but because so much of what it does is already happening in real life.
Futurists and fantasists have been dreaming about the rise of intelligent machines for centuries. Now it is actually starting to happen.
For a drier but in many ways even more astonishing account of what is going on, read “The Second Economy,” an essay in McKinsey Quarterly by W. Brian Arthur, a visiting researcher at the Intelligent Systems Lab at the Palo Alto Research Center in California.
Mr. Arthur’s contention is that a second, machine-to-machine economy is emerging and that it will bring deep economic, social and political change comparable to the transformation wrought by the Industrial Revolution.
“Business processes that once took place among human beings are now being executed electronically,” Mr Arthur writes. “They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential — it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.”
Mr. Arthur describes this economy as “vast, silent, connected, unseen and autonomous (meaning that human beings may design it but are not directly involved in running it).” The second economy is manifest in transactions as quotidian as checking in for a flight with a machine and as esoteric as the algorithm hedge funds of Mr. Harris’s thriller, which use information produced by machines to trade with other machines.
Economists and novelists aren’t the only people musing about the rise of the machine-to-machine economy and its transformative potential.
Yuri Milner is one of the savviest technology thinkers in the world; he was a pioneering investor in Facebook, a bet that was wildly vindicated last week.
Mr. Milner has a presentation in which he describes the nine most important changes in the world today. Three of them are about what Mr. Arthur has dubbed the second economy: the rise of what Mr. Milner calls “the Internet of things,” or the machine-to-machine economy; the growing power of artificial intelligence; and the emergence of a “global brain,” which is the network of all of the people and the machines in the world and their connections to one another.
The one aspect of “The Fear Index” that is tired and familiar is its depiction of the rise of the machine-to-machine economy as murderous and menacing. That time-worn worry that the smart machines will turn on their creators isn’t one I share. I am a fan of the machine-to-machine economy and of all the ways it makes my human life cheaper and easier.
But I am concerned about something Mr. Harris only hints at. At the end of “The Fear Index,” VIXAL changes its hedge fund’s mission statement. The first lines of the new, machine-written version, read: “The company of the future will have no workers.”
I’m not afraid our smart machines will try to exterminate us, but I do worry that the second economy may be a jobless one.
Mr. Arthur doesn’t offer much comfort on that score. In an exchange with a reader after his essay was published, the economist wrote: “Since the second economy began, in the early and mid-1990s, we’ve had wave after wave of downsizing and layoffs, and now we have ongoing structural joblessness. I hope jobs will be created, and maybe they will. More likely, the system, as so many times before in history, will have to readjust radically. It needs to find new ways to distribute the wealth.”
Chrystia Freeland is global editor at large at Reuters.
NEW YORK — If you want to get a finger-tip feel for one of the most important transformations in our world today, read “The Fear Index,” Robert Harris’s new thriller.
Mr. Harris has been widely praised for his adept portrayal of the hedge fund universe in which his novel is set. “The greatest pleasure of this book is that it gets the finance right,” cooed Felix Salmon, the Reuters finance blogger whose keyboard often oozes acid.
He is right that Mr. Harris’s hedgies are a welcome and realistic departure from the Masters of the Universe of most popular fiction. For one thing, these are the alpha geeks, the nerdy doctorate-holders whose testosterone is channeled into equations instead of frat-house swagger.
And Mr. Harris knows his super-rich. As they are being wooed to make a billion-dollar investment over a 1995 Latour in a fine Geneva restaurant, an international group of investors energetically discusses the ways that their national governments oppress them.
Dr. Alex Hoffmann, the billionaire hedge fund manager protagonist who is making the pitch for their cash, reflects: “He was remembering now why he didn’t like the rich: their self-pity. Persecution was the common ground of their conversation, like sport or the weather was for everyone else.”
But the most valuable intellectual take-away from this riveting novel is the element that has been widely dismissed by reviewers as familiar and unpersuasive: VIXAL, the intelligent machine at the heart of the story that torments its human creator and may be trying to rule the world. Michiko Kakutani, writing in The New York Times, gave “The Fear Index” a thumbs up, but deemed the machine takeover “silly and contrived.” VIXAL, the evil genius computer, “seems less like a plausible villain than like a metaphor for the greed and heedlessness that overtook Wall Street,” she wrote.
Ms. Kakutani’s dismissal of VIXAL as an unoriginal and uncompelling reworking of the Frankenstein template misses the most important point. VIXAL and the parallel world the computer program creates by interacting with other machines is worth reading about — but not for the usual pulp fiction pleasure of entering a vividly rendered imaginary world.
“The Fear Index” is fun to read because Mr. Harris is a great novelist, but VIXAL is worth getting to know not as a fictional character, but because so much of what it does is already happening in real life.
Futurists and fantasists have been dreaming about the rise of intelligent machines for centuries. Now it is actually starting to happen.
For a drier but in many ways even more astonishing account of what is going on, read “The Second Economy,” an essay in McKinsey Quarterly by W. Brian Arthur, a visiting researcher at the Intelligent Systems Lab at the Palo Alto Research Center in California.
Mr. Arthur’s contention is that a second, machine-to-machine economy is emerging and that it will bring deep economic, social and political change comparable to the transformation wrought by the Industrial Revolution.
“Business processes that once took place among human beings are now being executed electronically,” Mr Arthur writes. “They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential — it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.”
Mr. Arthur describes this economy as “vast, silent, connected, unseen and autonomous (meaning that human beings may design it but are not directly involved in running it).” The second economy is manifest in transactions as quotidian as checking in for a flight with a machine and as esoteric as the algorithm hedge funds of Mr. Harris’s thriller, which use information produced by machines to trade with other machines.
Economists and novelists aren’t the only people musing about the rise of the machine-to-machine economy and its transformative potential.
Yuri Milner is one of the savviest technology thinkers in the world; he was a pioneering investor in Facebook, a bet that was wildly vindicated last week.
Mr. Milner has a presentation in which he describes the nine most important changes in the world today. Three of them are about what Mr. Arthur has dubbed the second economy: the rise of what Mr. Milner calls “the Internet of things,” or the machine-to-machine economy; the growing power of artificial intelligence; and the emergence of a “global brain,” which is the network of all of the people and the machines in the world and their connections to one another.
The one aspect of “The Fear Index” that is tired and familiar is its depiction of the rise of the machine-to-machine economy as murderous and menacing. That time-worn worry that the smart machines will turn on their creators isn’t one I share. I am a fan of the machine-to-machine economy and of all the ways it makes my human life cheaper and easier.
But I am concerned about something Mr. Harris only hints at. At the end of “The Fear Index,” VIXAL changes its hedge fund’s mission statement. The first lines of the new, machine-written version, read: “The company of the future will have no workers.”
I’m not afraid our smart machines will try to exterminate us, but I do worry that the second economy may be a jobless one.
Mr. Arthur doesn’t offer much comfort on that score. In an exchange with a reader after his essay was published, the economist wrote: “Since the second economy began, in the early and mid-1990s, we’ve had wave after wave of downsizing and layoffs, and now we have ongoing structural joblessness. I hope jobs will be created, and maybe they will. More likely, the system, as so many times before in history, will have to readjust radically. It needs to find new ways to distribute the wealth.”
Chrystia Freeland is global editor at large at Reuters.
Wednesday, February 8, 2012
Walter Russell Mead : The Power of Infostructure
Beyond Blue Part Three: The Power of Infostructure
Walter Russell Mead, The American Interest, February 5, 2012
The quest for a new social model has to start with economics. America could survive without growing prosperity and rising standards of living, but it would not flourish — and it would not be living up to its potential to create a better life not only for Americans but for people all over the world. Green dreamers and communitarians disagree, often eloquently but always futilely; the drive for economic prosperity is deeply planted in American politics and society. When the economy isn’t performing well, politicians lose their jobs while the public looks for alternative ideas.
The quest for economic prosperity helped make the blue social model, and the failure of that model to deliver continuing prosperity in contemporary conditions is both a symptom of and a leading reason for its decline. The mass prosperity of “Fordism” depended on economic conditions that no longer hold. High paid manufacturing jobs and many clerical and middle management positions are disappearing. Some jobs are outsourced to cheaper foreign countries; others disappear as automation increases productivity and decreases the number of people needed to accomplish various tasks. The stable oligopolies and monopolies that once dominated the American economy have disappeared in the face of heightened international competition and accelerated technological change. As formerly large and stably profitable corporations had to scramble to survive, they no longer could afford lifetime employment, friendly relations with strong unions, and generous health care and retirement benefits.
Cars on an assembly line
Economically, the decline of the blue social model presents Americans with some urgent questions: how can we generate a rising standard of living for our citizens in a post-Fordist world? If manufacturing and stable oligopolies won’t underpin lifetime employment and rising wages for new generations of Americans, what are we going to do instead?
To start answering these questions we have to think about whether the United States enjoys any comparative advantages or can we develop some that, wisely deployed, could provide us with the kind of rising living standards that we have enjoyed in the past?
It’s clear that many such advantages exist and some are even becoming more prominent and useful. In the energy sector, homegrown shale gas and shale oil plus large new discoveries in the western hemisphere suggest that 21st century America will enjoy secure access to an abundant and varied mix of fuel from nearby and friendly neighbors. Blessed by a favorable climate and rich soil, our agricultural productivity will continue to astound. Our geographical location, which gives us access to both of the world’s greatest ocean trading basins while protecting us against invasion by other great powers, remains a tremendous advantage. The language we speak continues to develop as the global lingua franca, giving every American citizen and American business a significant boost.
The container ship SS Mayaguez
These are advantages we have enjoyed for centuries and they are likely to loom larger in the 21st century even than in the recent past. We have grown proficient at using and enhancing these advantages and we need to continue to do so moving forward. Just based on these factors, fears that we are sinking back to some kind of dystopian, third world future are, I think, overblown. But our greatest advantages are cultural and political and it is to these that we must ultimately look for the prosperity we want in the decades to come.
I’ve written about these advantages at some length in God and Gold: Britain, America and the Making of the Modern World, and I won’t repeat those arguments here. The upshot is that among all the countries in the world, the United States and a handful of English speaking societies have some unique strengths. We do better at combining rapid innovation with social and political stability than others do. Since 1688, the English speaking world has seen only two successful violent revolutions, and even the American Revolution was largely non-violent with royalist governors peacefully replaced until the British army arrived to reconquer the colonies.
(The other successful violent revolution was the secession of Ireland from the United Kingdom in 1919.) The UK, the US, Canada, Australia, New Zealand and, since independence, Ireland have had remarkably stable political histories and show great continuities in culture even as they have remained at the forefront of economic and technological change.
Despite a history of booms and busts from the South Sea Bubble to the 2008 meltdown, their financial histories have also been more stable than those of other countries; since the establishment of the Bank of England in the late seventeenth century, the political stability of the English speaking world has contributed to economic stability. Investors have been able to buy long term bonds in the English-speaking world with confidence since the 18th century. No other significant global economic or political centers have this kind of track record, though the Dutch come close. (Ugly budget pictures today in Ireland, the US and the UK tell us that we should not take these economic advantages for granted. Fortunately, in all three countries there seems to be some real political will to turn things around.)
The English-speaking world has been at the heart of the process of modernization with all the upheaval and uncertainty this process brings in train, but through it all these countries have somehow remained more stable and reliable than others. We change faster than others do in response to new opportunities and new technological possibilities — but we don’t lose our balance in the process.
Joseph Addison described the behavior of Winston Churchill’s famous ancestor the Duke of Marlborough at the battle of Blenheim in a way that captures this ideal of calm in the midst of frenzied activity:
The dreadful burst of cannon rend the skies,And all the thunder of the battle rise.
`Twas then great Marlborough`s mighty soul was prov`d,
That, in the shock of charging hosts unmov`d,
Amidst confusion, horror, and despair,
Examin`d all the dreadful scenes of war:
In peaceful thought the field of death survey`d,
To fainting squadrons sent the timely aid,
Inspir`d repuls`d battalions to engage,
And taught the doubtful battle where to rage.
So when an angel by divine command
With rising tempests shaks a guilty land,
Such as of late o`er pale Britannia past,
Calm and serene he drives the furious blast;
And, pleas`d th` Almighty`s orders to perform,
Rides in the whirlwind, and directs the storm.
And, pleas`d th` Almighty`s orders to perform,
Rides in the whirlwind, and directs the storm.
This angel in the whirlwind quality means more now than ever before. The 21st century is shaping up to be an age of upheaval; change is coming at us from so many directions and at such a pace that cultures and countries around the world are being shaken to their foundations. Those who can keep their calm and balance in the midst of the whirlwind have a serious advantage — and they should use it for all it is worth.
America is good at change. We absorb immigrants better than most. We like new things and like to try them out. We have an optimistic streak in our nature; we believe that change is basically good and that being open to new things will make us happier and better off. Our religious sensibility is future-oriented and believes that God is working through the chaos and uncertainties of life. Our national religious tradition is profoundly influenced by the dynamic vision of a God who calls humanity into an unknown future. While the religious cultures of some parts of the world look back to a real or imagined utopia in the far distant past, or instruct the faithful to resist change and cling to the ancient ways, American religion tends to see the hand of God behind the winds of change. We pursue God into the future, rather than hunting for him in the far-distant past.
America’s critical comparative advantage in the 21st century will be its ability to respond quickly to change: to recognize and exploit new opportunities faster than others, to retool its core institutions and practices to fit the emerging shape of the new world, and to do all that while retaining its political and social equipoise: to ride the whirlwind and direct the storm.
We were the first to build the blue social model and we can be the first to get to the next stage and reap the enormous rewards that come from reaching a more productive and efficient form of social organization before the competition.
Nanogears, a NASA project
No doubt many new opportunities will emerge during the 21st century; nanotechnology and biology look to provide revolutions at least as profound as anything to be found in the IT revolution now transforming the world. But until these new developments come more fully on line, the biggest opportunity and the greatest challenge that faces all the leading economies will be to harness the full power of IT to social needs.
All About Friction
In particular, this means competing with other countries not by the cheapness of our wages or the laxity of our environmental regulations, but by building on our ability to increase the productivity of both capital and labor through the power of IT to reduce the cost of friction in society. The first challenge of the 21st century will be the race to build infostructure — a mix of hardware, bandwidth, software, and government and corporate practices that deliver the greatest possible benefits of IT in ways that dramatically reduce costs and delays throughout the economy. A lot of this will be about friction.
Readers of Clausewitz know that friction was his word for the inexorable entropic pressure that disrupts any plan of campaign. Bad weather, diseases in the camp, miscommunication of orders, the absence of good information, the hazards of a night march: all these could be examples of the kind of friction with which every commander must struggle.
In civilian life as well as military affairs, friction has often been caused by distance: moving people, goods and information through space costs money and time. We will continue to build and repair physical infrastructure in the future, but the cutting edge of society’s effort to reduce frictional costs in time and money will lie in the development of “infostructure” rather than infrastructure: the mix of institutions, practices, hardware, software and bandwidth that allow the inherent productivity of information technology the fullest possible scope to transform the way we do things.
Take the legal system. The legal system arose in order to solve disputes; it consumes a great deal of time and money and is itself one of the great sources of friction in our society, but it is much better than anarchy would be. Yet as society becomes more complex, we need more legal service and not less, and the question of how to make the legal system fast, efficient and cheap grows more important all the time.
Let us imagine how the legal system might work some years down the road with the right kind of infostructure. Some changes are already under way. Routine matters like basic, simple wills, taxes and incorporations can be handled using cheap commercial software. Clearly, this software is going to get better and smarter, and more and more matters can be handled by people working on their own. Governments should be working to facilitate this shift, changing laws where necessary and advisable. Perhaps one day most routine marriage and divorce, residential real estate transactions, and many other activities that historically required lawyers can be done by individuals using software or at most with paralegals. More and more of this will move into the “cloud”; less and less time and money will be chewed up by routine legal affairs.
That is all first stage reform. Deeper change and greater savings will come as more sophisticated software and new institutions and practices that take full advantage of these capabilities transform the inner workings of the legal system. It’s impossible to foresee how this would work out in practice — any more than someone in 1895 could predict how the modern chemical industry would develop over time. But develop it will. One possible way things could go: Every court decision ever given, every brief ever filed and every journal article ever written will some day be available in a single database; intelligent software could sift the facts of any given case and the arguments of both sides against this material to make some predictions about how a given case might come out. Given access to the records and decisions of the judge appointed to hear a particular case, it would be possible to see how this particular judge would be likely to view this particular case with a fair degree of probability. That information would be available to both parties in the case, making a fast settlement a much more likely outcome. Many fewer cases might go to trial; decisions would in any case come much more quickly.
A courthouse in Washington County, Arkansas
Smart policy would speed this transition. Investing in the infostructure that moves the state’s legal system as far into the future as quickly as possible would attract new investment and jobs and enhance the profitability of business already in the state.
Research into ways to use IT to accelerate the speed of legal proceedings while reducing their cost is the kind of thing government should support. Land grant colleges supported agricultural research in the age of the family farm; we developed a host of ways to support research into improved industrial and factory productivity in the age of Fordism. Now it is time to shift to the next stage, for universities and government to move into the kind of research that will transform our social infostructure.
While infrastructure will remain important to the smooth functioning of society, infostructure will likely surpass it. Governor Jerry Brown’s successors may think less about building $100 billion high speed rail programs than about radical restructuring of the way California’s government works so that a much smaller and cheaper government is able to do more and do it much quicker. A superfast and supercheap legal system would dramatically enhance the productivity of capital invested in the state.
Government would work better and faster and cost less; taxes would fall even as services improved. (There is no real reason why many government services can’t improve the way personal computers and software have done: each year they get cheaper and each year they do more.)
The war on friction is one of humanity’s oldest challenges, and it inevitably becomes more important as society becomes more complicated and more interdependent. A small village can handle everything quickly; a modern megalopolis must have a much better developed government. Good infostructure can dramatically reduce the costs and complexities of modern governance.
America’s Secret Weapon
In all societies vested interests will resist efforts to reduce friction. The mule drivers didn’t like the canals and the canals hated the railroads. Pettifogging lawyers will hate the transformation of their profession, and every vested interest in the country will be touched one way or another by the kinds of transformations of our core social institutions that will take place as we build out the infostructure of the 21st century.
This is where America’s advantages count. That process can and should happen faster here than in other places. We have fifty states who compete with one another and the federal government to do things faster, quicker, better. Best practices and new ideas spread. Bureaucrats are less powerful here, and state employees are less well-placed to block changes they don’t like. The public is readier to try new things and take risks for the sake of progress. Our government, our legal system, our health system, our educational system: all of these can be radically and greatly improved by the right infostructure, and Americans are more willing and able to push this process ahead than people in other parts of the world.
The critical advances of the next generation involve the development and construction of a radically new infostructure that will change the way government, the law, education, medicine and many other institutions and industries work. The new infostructures will so dramatically enhance the efficiency of the societies able to build and install them that they will enjoy huge advantages over those who cannot.
America needs to rebuild its infostructure. The alternative to a race to the bottom, to trying to compete on the basis of lower wages or laxer regulations, is a race to the front: to the cutting edge of human progress.
This is a race to build the new kinds of infostructure that advances in communications and information technology have made possible. This is a race that we can win and that is worth winning. America can get to a faster, more efficient future faster than other people, and it is on that basis that we can prosper and raise our living standards in the coming years.
Monday, February 6, 2012
Big Data: Hospitals mine patient records in search of customers
Phil Galewitz, Kaiser Health News, January 18, 2012
By Brett T. Roseman, for USA TODAY
Steven Boyd of Joliet, Ill., and wife Karol received a direct mailing from their local hospital offering lung scans.
Boyd, 59, of Joliet, Ill., had smoked for decades, as had his wife, Karol.
Provena didn't send the mailing to everyone who lived near the hospital, just those who had a stronger likelihood of having smoked based on their age, income, insurance status and other demographic criteria.
The non-profit facility is one of a growing number of hospitals using their patients' health and financial records to help pitch their most lucrative services, such as cancer, heart and orthopedic care. As part of these direct mail campaigns, they are also buying detailed information about local residents compiled by consumer marketing firms — everything from age, income and marital status to shopping habits and whether residents have children or pets at home.
Hospitals say they are promoting needed services, such as cancer screenings and cholesterol tests, but they often use the data to target patients with private health insurance, which typically pay higher rates than government coverage. At an industry conference last year, Provena Health marketing executive Lisa Lagger said such efforts had helped attract higher-paying patients, including those covered by "profitable Blue Cross and less Medicare."
While the strategies are increasing revenues, they are drawing fire from patient advocates and privacy groups who criticize the hospitals for using private medical records to pursue profits.
Doug Heller, executive director of Consumer Watchdog, a California-based consumer advocacy group, says he is bothered by efforts to "cherry pick" the best-paying patients.
"When marketing is picking and choosing based on people's financial status, it is inherently discriminating against patients who have every right and need for medical information," Heller says. Deven McGraw, director of the health privacy project at the Center for Democracy and Technology in Washington, says federal law allows hospitals to use confidential medical records to keep patients informed about services that may help them.
"You want health providers to communicate to patients about health options that may be beneficial," McGraw says. "But sometimes this is about generating business for a new piece of equipment that the hospital just bought."
Using such information for marketing "creeps closer to the line" between what is legal and what is not, she says.
Hospital officials such as Denise Beaudoin of Detroit's Henry Ford Health System say what they do is legal and that the sophisticated targeting approach — called "customer relationship marketing" — simply helps them deliver information to those most likely to use it.
They say hospitals are adopting strategies used for decades by the retail, travel and communications industries, which have flourished with the growth of online companies such as Amazon and Google. For example: Buy a book on Amazon and it will suggest a title with a similar subject. Search for information on Alaskan vacations on Google, and an ad pops up for a cruise line.
At a time when government and private insurers are tightening reimbursements, more hospitals are turning to similar approaches to drive admissions. An estimated 20% now use the strategy, including large academic medical centers and large chains, such as HCA, based in Nashville, and Trinity Health, based in Novi, Mich. The trend is expected to accelerate as more hospitals adopt electronic health records, says Guy Miller, a Chicago health care consultant.
Putting patient data to work
Tess Niehaus, vice president of marketing at St. Anthony's Medical Center in St. Louis, makes no apologies for going after the most lucrative business.
"We are here to serve everybody, but we market for good-paying patients because it preserves our ability to serve everyone," she says.
St. Anthony's marketers use patient data to personalize mailings with an individual's name and a picture of someone of similar age or gender. It is more expensive, but the strategy results in better response rates, she says. From October 2010 through July 2011, St. Anthony's spent $25,000 on a targeted mailing to 40,000 women for mammogram screenings. The letters led 1,000 women to get the test, which generated $530,000 in revenue from screenings, biopsies and other related services, she says.
To help devise such campaigns, St. Anthony's and other hospitals share patient data with marketing staff and outside consultants. Anyone with access to patient records is required by federal law to sign non-disclosure agreements.
While the practice is legal, most people would be shocked to know their records may be shared with non-medical personnel and outside firms to help hospitals attract business, says Pam Dixon, executive director of the World Privacy Forum, an advocacy group based in California.
"I am really bothered by the overabundance of information that is flowing that is unnecessary and risky," she says.
Screenings lead to more tests
While hospitals may profit from offering cholesterol tests and mammograms, the big payoff is in what those screenings may lead to — additional tests and procedures, including surgery.
"The old adage in business is that it's easier to sell an existing customer new services rather than find a new customer," says Patrick Kane, senior vice president of marketing at Cape Cod Healthcare in Massachusetts who used such approaches at Wellmont Health System in Kingsport, Tenn.
Provena's six hospitals in Illinois embraced targeted marketing in 2010, mailing information about screenings and educational events to 293,000 people. The mailings led to more than 50,000 patient visits — a 17% response rate, several times that typically seen in direct mail efforts, according to the industry presentation to hospital officials last year in Orlando. After accounting for marketing costs, those visits netted the system $595,000.
Provena's Lagger says the approach boosted the system's bottom line so it could serve people regardless of insurance status. "This is a means to an end," she says.
One of the biggest pluses for hospital executives is that they can track a campaign's success by comparing the amount of services used by targeted consumers with a control group with the same demographic and economic characteristics but who are not sent mailings.
Mailings get results
When the Henry Ford Health System promoted mammograms last year in mailings to 30,000 women aged 40 or older, more than 5,700 responded — 304 more than in the control group. The mailings generated $268,000 more in profit than the control group — a return of more than four to one on the cost of the campaign, says Beaudoin, vice president of customer engagement.
"Some doctors used to be leery about the effectiveness of these marketing campaigns, but not when we can show them data like this," she says.
Mercy Health Partners in western Michigan, part of the 47-hospital non-profit Trinity Health system, sent a targeted cardiac screening mailing last year to 7,450 people. That resulted in 1,729 patient visits, or 7% more than in a control group. The campaign generated about $1 million in revenue and about $50,000 in profit.
"It's a much more efficient use of marketing dollars," says Preston Gee, Trinity's senior vice president of strategic planning. "People like having information tailored to their own needs."
Much of the expertise for such campaigns is provided by three consulting firms — CPM Marketing of Madison, Wis., Medseek of Birmingham, Ala., and New York-based Thomson Reuters. They typically charge hospitals $100,000 a year or more.
CPM, which merged in November with Denver-based HealthGrades, a health-ratings firm, added 100 new hospitals last year to give it a total of 400. Medseek works with more than 250 hospitals and Thomson Reuters with 150.
"There are a lot of very rich data in health care beyond just age and gender that help steer or guide people to health services," says John Hallick, president of CPM.
Loss of privacy
Boyd, the Joliet home inspector, was not upset that Provena Health used information about him and his wife — both former patients — to pitch screening tests. "We lost our privacy long ago, and I don't like to think about all the information that's out there about us."
Provena marketing manager Richard Matula would not say why the Boyds were included in the mailing, citing patient privacy laws. Patients' smoking status was not used in the criteria, he said.
The targeting worked in the case of Boyd, who called the number on the back and scheduled the CT scan a few days later. The $169 test showed his lungs were clear, but found potential blockages in coronary arteries that his Provena-affiliated doctor is monitoring.
"In hindsight, I'm glad I had the test," he says.
Kaiser Health News is an editorially independent program of the Henry J. Kaiser Family Foundation, a non-profit, non-partisan health policy research and communication organization not affiliated with Kaiser Permanente.
By Brett T. Roseman, for USA TODAY
Steven Boyd of Joliet, Ill., and wife Karol received a direct mailing from their local hospital offering lung scans.
Boyd, 59, of Joliet, Ill., had smoked for decades, as had his wife, Karol.
Provena didn't send the mailing to everyone who lived near the hospital, just those who had a stronger likelihood of having smoked based on their age, income, insurance status and other demographic criteria.
The non-profit facility is one of a growing number of hospitals using their patients' health and financial records to help pitch their most lucrative services, such as cancer, heart and orthopedic care. As part of these direct mail campaigns, they are also buying detailed information about local residents compiled by consumer marketing firms — everything from age, income and marital status to shopping habits and whether residents have children or pets at home.
Hospitals say they are promoting needed services, such as cancer screenings and cholesterol tests, but they often use the data to target patients with private health insurance, which typically pay higher rates than government coverage. At an industry conference last year, Provena Health marketing executive Lisa Lagger said such efforts had helped attract higher-paying patients, including those covered by "profitable Blue Cross and less Medicare."
While the strategies are increasing revenues, they are drawing fire from patient advocates and privacy groups who criticize the hospitals for using private medical records to pursue profits.
Doug Heller, executive director of Consumer Watchdog, a California-based consumer advocacy group, says he is bothered by efforts to "cherry pick" the best-paying patients.
"When marketing is picking and choosing based on people's financial status, it is inherently discriminating against patients who have every right and need for medical information," Heller says. Deven McGraw, director of the health privacy project at the Center for Democracy and Technology in Washington, says federal law allows hospitals to use confidential medical records to keep patients informed about services that may help them.
"You want health providers to communicate to patients about health options that may be beneficial," McGraw says. "But sometimes this is about generating business for a new piece of equipment that the hospital just bought."
Using such information for marketing "creeps closer to the line" between what is legal and what is not, she says.
Hospital officials such as Denise Beaudoin of Detroit's Henry Ford Health System say what they do is legal and that the sophisticated targeting approach — called "customer relationship marketing" — simply helps them deliver information to those most likely to use it.
They say hospitals are adopting strategies used for decades by the retail, travel and communications industries, which have flourished with the growth of online companies such as Amazon and Google. For example: Buy a book on Amazon and it will suggest a title with a similar subject. Search for information on Alaskan vacations on Google, and an ad pops up for a cruise line.
At a time when government and private insurers are tightening reimbursements, more hospitals are turning to similar approaches to drive admissions. An estimated 20% now use the strategy, including large academic medical centers and large chains, such as HCA, based in Nashville, and Trinity Health, based in Novi, Mich. The trend is expected to accelerate as more hospitals adopt electronic health records, says Guy Miller, a Chicago health care consultant.
Putting patient data to work
Tess Niehaus, vice president of marketing at St. Anthony's Medical Center in St. Louis, makes no apologies for going after the most lucrative business.
"We are here to serve everybody, but we market for good-paying patients because it preserves our ability to serve everyone," she says.
St. Anthony's marketers use patient data to personalize mailings with an individual's name and a picture of someone of similar age or gender. It is more expensive, but the strategy results in better response rates, she says. From October 2010 through July 2011, St. Anthony's spent $25,000 on a targeted mailing to 40,000 women for mammogram screenings. The letters led 1,000 women to get the test, which generated $530,000 in revenue from screenings, biopsies and other related services, she says.
To help devise such campaigns, St. Anthony's and other hospitals share patient data with marketing staff and outside consultants. Anyone with access to patient records is required by federal law to sign non-disclosure agreements.
While the practice is legal, most people would be shocked to know their records may be shared with non-medical personnel and outside firms to help hospitals attract business, says Pam Dixon, executive director of the World Privacy Forum, an advocacy group based in California.
"I am really bothered by the overabundance of information that is flowing that is unnecessary and risky," she says.
Screenings lead to more tests
While hospitals may profit from offering cholesterol tests and mammograms, the big payoff is in what those screenings may lead to — additional tests and procedures, including surgery.
"The old adage in business is that it's easier to sell an existing customer new services rather than find a new customer," says Patrick Kane, senior vice president of marketing at Cape Cod Healthcare in Massachusetts who used such approaches at Wellmont Health System in Kingsport, Tenn.
Provena's six hospitals in Illinois embraced targeted marketing in 2010, mailing information about screenings and educational events to 293,000 people. The mailings led to more than 50,000 patient visits — a 17% response rate, several times that typically seen in direct mail efforts, according to the industry presentation to hospital officials last year in Orlando. After accounting for marketing costs, those visits netted the system $595,000.
Provena's Lagger says the approach boosted the system's bottom line so it could serve people regardless of insurance status. "This is a means to an end," she says.
One of the biggest pluses for hospital executives is that they can track a campaign's success by comparing the amount of services used by targeted consumers with a control group with the same demographic and economic characteristics but who are not sent mailings.
Mailings get results
When the Henry Ford Health System promoted mammograms last year in mailings to 30,000 women aged 40 or older, more than 5,700 responded — 304 more than in the control group. The mailings generated $268,000 more in profit than the control group — a return of more than four to one on the cost of the campaign, says Beaudoin, vice president of customer engagement.
"Some doctors used to be leery about the effectiveness of these marketing campaigns, but not when we can show them data like this," she says.
Mercy Health Partners in western Michigan, part of the 47-hospital non-profit Trinity Health system, sent a targeted cardiac screening mailing last year to 7,450 people. That resulted in 1,729 patient visits, or 7% more than in a control group. The campaign generated about $1 million in revenue and about $50,000 in profit.
"It's a much more efficient use of marketing dollars," says Preston Gee, Trinity's senior vice president of strategic planning. "People like having information tailored to their own needs."
Much of the expertise for such campaigns is provided by three consulting firms — CPM Marketing of Madison, Wis., Medseek of Birmingham, Ala., and New York-based Thomson Reuters. They typically charge hospitals $100,000 a year or more.
CPM, which merged in November with Denver-based HealthGrades, a health-ratings firm, added 100 new hospitals last year to give it a total of 400. Medseek works with more than 250 hospitals and Thomson Reuters with 150.
"There are a lot of very rich data in health care beyond just age and gender that help steer or guide people to health services," says John Hallick, president of CPM.
Loss of privacy
Boyd, the Joliet home inspector, was not upset that Provena Health used information about him and his wife — both former patients — to pitch screening tests. "We lost our privacy long ago, and I don't like to think about all the information that's out there about us."
Provena marketing manager Richard Matula would not say why the Boyds were included in the mailing, citing patient privacy laws. Patients' smoking status was not used in the criteria, he said.
The targeting worked in the case of Boyd, who called the number on the back and scheduled the CT scan a few days later. The $169 test showed his lungs were clear, but found potential blockages in coronary arteries that his Provena-affiliated doctor is monitoring.
"In hindsight, I'm glad I had the test," he says.
Kaiser Health News is an editorially independent program of the Henry J. Kaiser Family Foundation, a non-profit, non-partisan health policy research and communication organization not affiliated with Kaiser Permanente.
Thursday, February 2, 2012
Why Big Data Won’t Make You Smart, Rich, Or Pretty
FC Expert Blogger Daniel W. Rasmus, Fast Company, January 27, 2012
If 2012 is the year of Big Data, it will likely be the year vendors and consultants start to over-promise, under-deliver, and put processes in motion that will generate insights and potential risks for years to come.
This year will be the year of Big Data. The Data Warehousing Institute (TDWI) reported that 90 percent of the IT professionals it surveyed said they were familiar with big data analytics. And 34 percent said they already applied analytics to Big Data. The vast hordes of data collection during e-commerce transactions, from loyalty programs, employment records, supply chain and ERP systems are, or are about to get, cozy. Uncomfortably cozy.
Let me start by saying there is nothing inherently wrong with Big Data. Big Data is a thing, and like anything, it can be used for good or for evil. It can be used appropriately given known limitations, or stretched wantonly until its principles fray. For now, the identification, consolidation, and governance of data is an appropriate step, as Forbes's Tom Groenfeldt recently documented with Michigan's $19 million in data center consolidation savings. Dirk Helbing of the Swiss Federal Institute of Technology in Zurich is more ambitious. His €1-billion project, the topic of the December 2011 Scientific American cover story, seeks to do nothing less than foretell the future.
The meaningful use of Big Data lies somewhere between these two extremes. For Big Data to move from anything more than an instantiation of databases running in logical or physical proximity, to data that can be meaningfully mined for insight, requires new skills, new perspectives, and new cautions.
The Big Data Dream
Dirk Helbing seeks a system that is akin to Asimov's Psychohistory as imagined in the Foundation series. In broad swaths, it would anticipate the future by linking social, scientific, and economic data. This system could be used to help advise world governments on the most salient choices to make.
Reading the article in Scientific American reminded me of a science fiction story by Tribble-inventor David Gerrold—When Harlie Was One. In this book, Harlie, which stands for Human Analog Robot Life Input Equivalents, decides that he needs answers, and that he isn't sophisticated enough to solve his own problem and therefore keep the corporate interests that built him interested enough to keep him plugged in. So he designs a new computer, the Graphic Omniscient Device, or GOD, as a proof of his value. GOD will answer all questions submitted to it. Unfortunately, as the human engineers building GOD eventually realize, the processing capacity is so vast, that GOD will not be able to provide an answer to any question during the lifetime of a human.
Harlie, of course, knew this all along. He needed the humans for three reasons: to keep him running, for engineering labor to build GOD, and to ask the questions that GOD will answer.
Given the woes of Europe, spending €1-billion on such a project will likely prove to be wasted money. We, of course, don't have a mechanical futurist to evaluate that position, but we do have history. Whenever there is an existential problem facing the world, charlatans appear to dazzle the masses with feats of magic and wonder. I don't see this proposal being anything more than the latest version of apocalyptic sorcery. It's not that a big science project can't yield interesting outcomes, but if you look at the Microelectronics and Computer Technology Corporation (MCC), late of Austin, Texas, we find Cyc, a system conceived at the beginning of the computer era, to combat Japan's Fifth Generation Project as it supposedly threatened to out-innovate America's nascent lead in computer technology.
Although Cyc has yielded some use, it has not yet become the artificial human mind it was intended to be, able to converse naturally with anyone about the events, concepts, and objects in the world. And artificial intelligence, as imagined in the 1980s, has yet to transform the human condition.
As Big Data becomes the next great savior of business and humanity, we need to remain skeptical of its promises as well as its applications and aspirations.
Existential Issues With Big Data
Determinism teaches that what will be, will be. Existentialism deals with a humanity in the throes of chaos. Big Data can be seen as either a lens through which determinism is revealed, or a tool for navigating an existential world.
As a scenario planner, I take the existential position and see a number of existential threats to the success of Big Data and its applications.
Overconfidence: Many managers creating a project plan, drawing up a budget, or managing a hedge fund trust their forecasts based on personal abilities and confidence in their knowledge and experience. As University of Chicago professor Richard H. Thaler recently pointed out in the New York Times ("The Overconfidence Problem in Forecasting"), most managers are overconfident and miscalibrated. In other words, they don't recognize their own inability to forecast the future, nor do they recognize the inherent volatility of markets. Both of these traits portend big problems for Big Data as humans code their assumptions about the world into algorithms: people don't understand their personal limitations, nor do they recognize if a model is good or not.
When learning happens: Even in a field as seemingly physical and visceral as fossil hunting, Big Data is playing a role. Geologic data has been fed into a model that helps pinpoint good fossil-hunting fields. On the surface that appears a useful discovery, but if you dig a bit deeper, you find a lesson for would-be Big Data modelers. As technology and data sophistication increases, the underlying assumptions in the model must change. Current data, derived from the analysis of Landsat photos, can direct field workers toward a fairly large, but promising area with multiple types of rock exposures. Eventually the team hopes to increase their 15-meter resolution to 15-centimeter resolution by acquiring higher-resolution data. As they examine the new data, they will need to change their analysis approach to recognize features not previously available (for more see "Artificial intelligence joins the fossil hunt" in New Scientist). Learning will mean reinterpreting the model.
On a more abstract level, recent work conducted by ETH Zurich looked at 43,000 transnational companies seeking to understand the relationships between those companies and their potential for influence. This analysis found that 1,318 companies were tightly connected, with an average of 20 connections, representing about 60 percent of global revenues. Deeper analysis revealed a "super-entity" of 147 firms that accounts for about 40 percent of the wealth in the network. This type of analysis has been conducted before, but the Zurich team included indirect ownership, which changed the outcome significantly (for more see "The network of global control" by Bitali, Glattfelder, and Battiston).
If organizations rely on Big Data to connect far-ranging databases--well beyond corporate ownership or maps of certain geologies--who, it must be asked, will understand enough of the model to challenge its underlying assumptions, and re-craft those assumptions when the world, and the data that reflects it, changes?
Complexity: I was sitting with the CIO of a large insurance company in Portland. We were talking about generational hand-offs when he raised the issue of an Excel spreadsheet used to evaluate commercial property underwriting. He said one of the older members of the organization owned that spreadsheet and he was the only one who knew how it worked. The hand-off issue was not one of getting the older employee to collaborate with the younger employee, but one of complexity. That spreadsheet was complex and tightly woven into the employee's worldview. Although the transfer could theoretically take place, it is unknowable how long it would take, if the new employee would stay, or how the process would change as multiple worldviews collided. Combining models full of nuance and obscurity increases complexity. Organizations that plan complex uses of Big Data and the algorithms that analyze the data need to think about continuity and succession planning in order to maintain the accuracy and relevance of their models over time, and they need to be very cautious about the time it will take to integrate, and the value of results achieved, from data and models that border on the cryptic.
Feedback Loops: Big Data isn't just about the size of well-understood data sets, it is about linking disparate data sets and then creating connective tissue, either through design or inference, between these data sets. At the onset of the Great Recession, we experienced a feedback loop failure as David X Li's famous Gaussian copula function, a seemingly well-tested approach to analyzing financial risk, failed to anticipate the risks lurking outside of its models. People kept trading, assuming their risk analysis was still meaningful. No feedback loop existed to inform the bond markets that their credit default swaps were an inverse pyramid teetering on a bed of miscalculation.
Algorithms and a Lack of Theory: It is not only algorithms that can go wrong when a theory proves incorrect or the assumptions underlying the algorithm change. There are places where no theory exists at any level of consensus to be meaningful. The impact of education (and the effectiveness of various approaches), how innovation works, or what triggers a fad are examples of behaviors for which little valid theory exists--it's not that plenty of opinion about various approaches or models is lacking, but that a theory, in the scientific sense, is nonexistent. For Big Data that means a number of things, first and foremost, that if you don't have a working theory, you probably don't know what data you need to test any hypotheses you may posit. It also means that data scientists can't create a model because no reliable underlying logic exists that can be encoded into a model.
Confirmation Bias: Every model is based on historical assumptions and perceptual biases. Regardless of the sophistication of the science, we often create models that help us see what we want to see, using data selected as a good indicator of such a perception. Take a recent debate about how to price futures. Future events are typically discounted using an exponential model that creates a regular discount rate that eventually leads a value of zero for far-flung events. Exponential discounting takes a deterministic view. A more existential view comes from the proponents of hyperbolic discounting, which creates a preference for rewards that arrive sooner than later. With hyperbolic discounting, discounts of future events fall more gradually, leading to what might be called "irrational behavior." Another version is called the declining discount rate.
This "discount" debate points out that even when a model exists that is designed to aid in decision making about the future, that model may involve contentious disagreements about its validity and alternative approaches that yield very different results. These are important debates in the world of Big Data. One group of modelers advocates for one approach, and another group, an alternative approach, both using sophisticated data and black boxes (as far as the uninitiated business person is concerned) to support their cases. The fact is that in cases like this, no one knows the answer definitively as the application may be contextual or it may be incomplete (e.g., a new approach may solve the issue that none of the current approaches solves completely). What can be said, and what must be remembered is, the adage that "a futurist is never wrong today." Who wins these debates today may be meaningless because the implications have no near-term consequences, but companies that accept one approach over the other may be betting their firm's future on wishful thinking and unwillingness to admit what they don't know.
The World Changes: We must remember that all data is historical. There is no data from or about the future. Future context changes cannot be built into a model because they cannot be anticipated. Consider this: 2012 is the 50th anniversary of the 1962 Seattle World's Fair. In 1962, the retail world was dominated by Sears, Montgomery Ward, Woolworth, A&P, and Kresge.
Some of those companies no longer exist, and others have merged to the point that they are unrecognizable from their 1962 incarnations. Also in 1962, thousands of miles away, a small company opened its first location in Rogers, Ark.--the Wal-Mart Discount City. Would models of retail supply chains built in 1962 be able to anticipate the overwhelming disruption that this humble storefront would cause for retail? Did Sam Walton understand the impact of Amazon.com when it went live in 1995?
The answer to all of the above is "no." These innovations are rare and hugely disruptive. They are far from doomsday scenarios except for firms so entrenched in their models that they can't adapt. And there is the problem for Big Data. As with Li's risk analysis algorithm, when the world changed, the model did not. Feedback loops are important as a way of maintaining relevance through incremental improvement, but what happens when the world changes so much that current assumptions become irrelevant and the clock must be started again. Not only must we remember that all data is historical, but we must also remember that at some point historical data becomes irrelevant when the context changes.
Motives: In a recent BusinessWeek article (Palantir, "The War on Terror's Secret Weapon"), Peter Theil is quoted as saying: "We cannot afford to have another 9/11 event in the U.S. or anything bigger than that. That day opened the doors to all sorts of crazy abuses and draconian policies." Theil, characterized in the article as a libertarian, sees the analysis of data as a positive for civil liberties. That position is debatable as groups like NO2ID.net form to campaign against not only the use of Big Data, but its creation. Given the complexity of the data and associated models, along with various intended of unintended biases, organizations have to go out of their way to discern the motives of those developing analytics models, lest they allow programs to manipulate data in a way that may precipitate negative social, legal, or fiduciary outcomes.
Acting on the Model: Consider crime analysis. George Mohler of Santa Clara University in California has applied equations that predict earthquake aftershocks to crime. By using location and data and times of recent crimes, the system predicts "aftercrimes." This kind of anticipatory data may result in bastions of police flooding a neighborhood following one burglary. With no police presence, the anticipated crimes may well take place. If the burglars, however, see an increase in surveillance and police activity, they may abandon planned targets and seek new ones, thus invalidating the models' predictions, potentially in terms of time and location. The proponents of Big Data need to ensure that the users of their models understand the intricacies of trend analysis, what a trend really is, and the implications of acting on a model's recommendations.
Where Big Data Will Work
Some of the emerging Big Data stories don't test the existential limits of technology, nor do they threaten global catastrophe. The worst outcome in dinosaur fossil hunting is not finding dinosaur bones where you expect them, and the worst outcome of a crime prediction is a burglary that doesn't take place.
Big Data will no doubt be used to target advertising, reduce fraud, fight crime, find tax evaders, collect child support payments, create better health outcomes, and myrid other activities from the mundane to the ridiculous.
And along the way, the software companies and those who invested in Big Data will share their stories.
Companies like monumental con
structor Arup use Big Data as a way to better model the use of the buildings they build. The Arup software arm, Oasys, recently acquired MassMotion to help them understand the flow of people through buildings. MassMotion can model the intimacy of a coffee shop or the flow of hundreds of thousands through a terminal or metro system using its agent technology. His models are informed from data about traffic patterns, arrival and departure times for various forms of transportation and the environment, such as shopping locations and information desks where people might stop or congregate. The result is a model, sometimes with thousands of avatars, pushing and shoving, congregating and separating--all based on MassMotion's Erin Morrow and how he perceives the world.
Another movement oriented application of Big Data, Jyotish (Sanskrit for astrology), comes from Boeing's research cener at the University of Illinois in Urbana-Champaign. This application predicts the movement of work crews within Boeing's factories. It will ultimately help them figure out how to save costs and increase satisfaction by ensuring that services, like Wi-Fi, are available where and when they are needed.
Palantir, the Palo Alto-based startup focused on solving the intelligence problem of 9/11, discovers correlations in the data that informs military and intelligence agencies who, what, and when a potential threat turns into an imminent threat. With access to data models and data across government silos, Palantir may well make its hero case more often than not in individual cases. They have to be cautious about applying their ideas to different domains where underlying rules might not be so clear or data so well-defined.
For some fields, like biology, placing large data sets into open source areas may bring a kind of convergence as collaboration ensues. But as Michael Nielsen points out in Reinventing Discovery, scientists have very little motivation to collaborate given the nature of publication, reputation, and tenure.
I seriously doubt that we have the intellectual infrastructure to support the collaborative capabilities of the Internet. We may well be able to connect all sorts of data and run all kinds of analyzes, but in the end, we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data.
A Telling Story
In the Scientific American article, Helbing relates an old story. A drunk man is looking for his keys under a street lamp. When asked why, he responds, "that's where the light is." To Helbing, it appears that he wants to use Big Data to create a brighter light so that scientists can peer beneath if for insight.
Scenario planners tell that story too, but we do so to make the point that while one is looking for keys in the light, they aren't paying any attention to the darkness. The future of Big Data lies not in the stories of anecdotal triumph that report sophisticated, but limited accomplishments--no, the future of Big Data rather lies in the darkness of context change, complexity, and overconfidence.
I will end, as Thaler did in his New York Times article, by quoting Mark Twain: "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."
This blog is written by a member of our expert blogging community and expresses that expert's views alone.
This year will be the year of Big Data. The Data Warehousing Institute (TDWI) reported that 90 percent of the IT professionals it surveyed said they were familiar with big data analytics. And 34 percent said they already applied analytics to Big Data. The vast hordes of data collection during e-commerce transactions, from loyalty programs, employment records, supply chain and ERP systems are, or are about to get, cozy. Uncomfortably cozy.
Let me start by saying there is nothing inherently wrong with Big Data. Big Data is a thing, and like anything, it can be used for good or for evil. It can be used appropriately given known limitations, or stretched wantonly until its principles fray. For now, the identification, consolidation, and governance of data is an appropriate step, as Forbes's Tom Groenfeldt recently documented with Michigan's $19 million in data center consolidation savings. Dirk Helbing of the Swiss Federal Institute of Technology in Zurich is more ambitious. His €1-billion project, the topic of the December 2011 Scientific American cover story, seeks to do nothing less than foretell the future.
The meaningful use of Big Data lies somewhere between these two extremes. For Big Data to move from anything more than an instantiation of databases running in logical or physical proximity, to data that can be meaningfully mined for insight, requires new skills, new perspectives, and new cautions.
The Big Data Dream
Dirk Helbing seeks a system that is akin to Asimov's Psychohistory as imagined in the Foundation series. In broad swaths, it would anticipate the future by linking social, scientific, and economic data. This system could be used to help advise world governments on the most salient choices to make.
Reading the article in Scientific American reminded me of a science fiction story by Tribble-inventor David Gerrold—When Harlie Was One. In this book, Harlie, which stands for Human Analog Robot Life Input Equivalents, decides that he needs answers, and that he isn't sophisticated enough to solve his own problem and therefore keep the corporate interests that built him interested enough to keep him plugged in. So he designs a new computer, the Graphic Omniscient Device, or GOD, as a proof of his value. GOD will answer all questions submitted to it. Unfortunately, as the human engineers building GOD eventually realize, the processing capacity is so vast, that GOD will not be able to provide an answer to any question during the lifetime of a human.
Harlie, of course, knew this all along. He needed the humans for three reasons: to keep him running, for engineering labor to build GOD, and to ask the questions that GOD will answer.
Given the woes of Europe, spending €1-billion on such a project will likely prove to be wasted money. We, of course, don't have a mechanical futurist to evaluate that position, but we do have history. Whenever there is an existential problem facing the world, charlatans appear to dazzle the masses with feats of magic and wonder. I don't see this proposal being anything more than the latest version of apocalyptic sorcery. It's not that a big science project can't yield interesting outcomes, but if you look at the Microelectronics and Computer Technology Corporation (MCC), late of Austin, Texas, we find Cyc, a system conceived at the beginning of the computer era, to combat Japan's Fifth Generation Project as it supposedly threatened to out-innovate America's nascent lead in computer technology.
Although Cyc has yielded some use, it has not yet become the artificial human mind it was intended to be, able to converse naturally with anyone about the events, concepts, and objects in the world. And artificial intelligence, as imagined in the 1980s, has yet to transform the human condition.
As Big Data becomes the next great savior of business and humanity, we need to remain skeptical of its promises as well as its applications and aspirations.
Existential Issues With Big Data
Determinism teaches that what will be, will be. Existentialism deals with a humanity in the throes of chaos. Big Data can be seen as either a lens through which determinism is revealed, or a tool for navigating an existential world.
As a scenario planner, I take the existential position and see a number of existential threats to the success of Big Data and its applications.
Overconfidence: Many managers creating a project plan, drawing up a budget, or managing a hedge fund trust their forecasts based on personal abilities and confidence in their knowledge and experience. As University of Chicago professor Richard H. Thaler recently pointed out in the New York Times ("The Overconfidence Problem in Forecasting"), most managers are overconfident and miscalibrated. In other words, they don't recognize their own inability to forecast the future, nor do they recognize the inherent volatility of markets. Both of these traits portend big problems for Big Data as humans code their assumptions about the world into algorithms: people don't understand their personal limitations, nor do they recognize if a model is good or not.
When learning happens: Even in a field as seemingly physical and visceral as fossil hunting, Big Data is playing a role. Geologic data has been fed into a model that helps pinpoint good fossil-hunting fields. On the surface that appears a useful discovery, but if you dig a bit deeper, you find a lesson for would-be Big Data modelers. As technology and data sophistication increases, the underlying assumptions in the model must change. Current data, derived from the analysis of Landsat photos, can direct field workers toward a fairly large, but promising area with multiple types of rock exposures. Eventually the team hopes to increase their 15-meter resolution to 15-centimeter resolution by acquiring higher-resolution data. As they examine the new data, they will need to change their analysis approach to recognize features not previously available (for more see "Artificial intelligence joins the fossil hunt" in New Scientist). Learning will mean reinterpreting the model.
On a more abstract level, recent work conducted by ETH Zurich looked at 43,000 transnational companies seeking to understand the relationships between those companies and their potential for influence. This analysis found that 1,318 companies were tightly connected, with an average of 20 connections, representing about 60 percent of global revenues. Deeper analysis revealed a "super-entity" of 147 firms that accounts for about 40 percent of the wealth in the network. This type of analysis has been conducted before, but the Zurich team included indirect ownership, which changed the outcome significantly (for more see "The network of global control" by Bitali, Glattfelder, and Battiston).
If organizations rely on Big Data to connect far-ranging databases--well beyond corporate ownership or maps of certain geologies--who, it must be asked, will understand enough of the model to challenge its underlying assumptions, and re-craft those assumptions when the world, and the data that reflects it, changes?
Complexity: I was sitting with the CIO of a large insurance company in Portland. We were talking about generational hand-offs when he raised the issue of an Excel spreadsheet used to evaluate commercial property underwriting. He said one of the older members of the organization owned that spreadsheet and he was the only one who knew how it worked. The hand-off issue was not one of getting the older employee to collaborate with the younger employee, but one of complexity. That spreadsheet was complex and tightly woven into the employee's worldview. Although the transfer could theoretically take place, it is unknowable how long it would take, if the new employee would stay, or how the process would change as multiple worldviews collided. Combining models full of nuance and obscurity increases complexity. Organizations that plan complex uses of Big Data and the algorithms that analyze the data need to think about continuity and succession planning in order to maintain the accuracy and relevance of their models over time, and they need to be very cautious about the time it will take to integrate, and the value of results achieved, from data and models that border on the cryptic.
Feedback Loops: Big Data isn't just about the size of well-understood data sets, it is about linking disparate data sets and then creating connective tissue, either through design or inference, between these data sets. At the onset of the Great Recession, we experienced a feedback loop failure as David X Li's famous Gaussian copula function, a seemingly well-tested approach to analyzing financial risk, failed to anticipate the risks lurking outside of its models. People kept trading, assuming their risk analysis was still meaningful. No feedback loop existed to inform the bond markets that their credit default swaps were an inverse pyramid teetering on a bed of miscalculation.
Algorithms and a Lack of Theory: It is not only algorithms that can go wrong when a theory proves incorrect or the assumptions underlying the algorithm change. There are places where no theory exists at any level of consensus to be meaningful. The impact of education (and the effectiveness of various approaches), how innovation works, or what triggers a fad are examples of behaviors for which little valid theory exists--it's not that plenty of opinion about various approaches or models is lacking, but that a theory, in the scientific sense, is nonexistent. For Big Data that means a number of things, first and foremost, that if you don't have a working theory, you probably don't know what data you need to test any hypotheses you may posit. It also means that data scientists can't create a model because no reliable underlying logic exists that can be encoded into a model.
Confirmation Bias: Every model is based on historical assumptions and perceptual biases. Regardless of the sophistication of the science, we often create models that help us see what we want to see, using data selected as a good indicator of such a perception. Take a recent debate about how to price futures. Future events are typically discounted using an exponential model that creates a regular discount rate that eventually leads a value of zero for far-flung events. Exponential discounting takes a deterministic view. A more existential view comes from the proponents of hyperbolic discounting, which creates a preference for rewards that arrive sooner than later. With hyperbolic discounting, discounts of future events fall more gradually, leading to what might be called "irrational behavior." Another version is called the declining discount rate.
This "discount" debate points out that even when a model exists that is designed to aid in decision making about the future, that model may involve contentious disagreements about its validity and alternative approaches that yield very different results. These are important debates in the world of Big Data. One group of modelers advocates for one approach, and another group, an alternative approach, both using sophisticated data and black boxes (as far as the uninitiated business person is concerned) to support their cases. The fact is that in cases like this, no one knows the answer definitively as the application may be contextual or it may be incomplete (e.g., a new approach may solve the issue that none of the current approaches solves completely). What can be said, and what must be remembered is, the adage that "a futurist is never wrong today." Who wins these debates today may be meaningless because the implications have no near-term consequences, but companies that accept one approach over the other may be betting their firm's future on wishful thinking and unwillingness to admit what they don't know.
The World Changes: We must remember that all data is historical. There is no data from or about the future. Future context changes cannot be built into a model because they cannot be anticipated. Consider this: 2012 is the 50th anniversary of the 1962 Seattle World's Fair. In 1962, the retail world was dominated by Sears, Montgomery Ward, Woolworth, A&P, and Kresge.
Some of those companies no longer exist, and others have merged to the point that they are unrecognizable from their 1962 incarnations. Also in 1962, thousands of miles away, a small company opened its first location in Rogers, Ark.--the Wal-Mart Discount City. Would models of retail supply chains built in 1962 be able to anticipate the overwhelming disruption that this humble storefront would cause for retail? Did Sam Walton understand the impact of Amazon.com when it went live in 1995?
The answer to all of the above is "no." These innovations are rare and hugely disruptive. They are far from doomsday scenarios except for firms so entrenched in their models that they can't adapt. And there is the problem for Big Data. As with Li's risk analysis algorithm, when the world changed, the model did not. Feedback loops are important as a way of maintaining relevance through incremental improvement, but what happens when the world changes so much that current assumptions become irrelevant and the clock must be started again. Not only must we remember that all data is historical, but we must also remember that at some point historical data becomes irrelevant when the context changes.
Motives: In a recent BusinessWeek article (Palantir, "The War on Terror's Secret Weapon"), Peter Theil is quoted as saying: "We cannot afford to have another 9/11 event in the U.S. or anything bigger than that. That day opened the doors to all sorts of crazy abuses and draconian policies." Theil, characterized in the article as a libertarian, sees the analysis of data as a positive for civil liberties. That position is debatable as groups like NO2ID.net form to campaign against not only the use of Big Data, but its creation. Given the complexity of the data and associated models, along with various intended of unintended biases, organizations have to go out of their way to discern the motives of those developing analytics models, lest they allow programs to manipulate data in a way that may precipitate negative social, legal, or fiduciary outcomes.
Acting on the Model: Consider crime analysis. George Mohler of Santa Clara University in California has applied equations that predict earthquake aftershocks to crime. By using location and data and times of recent crimes, the system predicts "aftercrimes." This kind of anticipatory data may result in bastions of police flooding a neighborhood following one burglary. With no police presence, the anticipated crimes may well take place. If the burglars, however, see an increase in surveillance and police activity, they may abandon planned targets and seek new ones, thus invalidating the models' predictions, potentially in terms of time and location. The proponents of Big Data need to ensure that the users of their models understand the intricacies of trend analysis, what a trend really is, and the implications of acting on a model's recommendations.
Where Big Data Will Work
Some of the emerging Big Data stories don't test the existential limits of technology, nor do they threaten global catastrophe. The worst outcome in dinosaur fossil hunting is not finding dinosaur bones where you expect them, and the worst outcome of a crime prediction is a burglary that doesn't take place.
Big Data will no doubt be used to target advertising, reduce fraud, fight crime, find tax evaders, collect child support payments, create better health outcomes, and myrid other activities from the mundane to the ridiculous.
And along the way, the software companies and those who invested in Big Data will share their stories.
Companies like monumental con
structor Arup use Big Data as a way to better model the use of the buildings they build. The Arup software arm, Oasys, recently acquired MassMotion to help them understand the flow of people through buildings. MassMotion can model the intimacy of a coffee shop or the flow of hundreds of thousands through a terminal or metro system using its agent technology. His models are informed from data about traffic patterns, arrival and departure times for various forms of transportation and the environment, such as shopping locations and information desks where people might stop or congregate. The result is a model, sometimes with thousands of avatars, pushing and shoving, congregating and separating--all based on MassMotion's Erin Morrow and how he perceives the world.
Another movement oriented application of Big Data, Jyotish (Sanskrit for astrology), comes from Boeing's research cener at the University of Illinois in Urbana-Champaign. This application predicts the movement of work crews within Boeing's factories. It will ultimately help them figure out how to save costs and increase satisfaction by ensuring that services, like Wi-Fi, are available where and when they are needed.
Palantir, the Palo Alto-based startup focused on solving the intelligence problem of 9/11, discovers correlations in the data that informs military and intelligence agencies who, what, and when a potential threat turns into an imminent threat. With access to data models and data across government silos, Palantir may well make its hero case more often than not in individual cases. They have to be cautious about applying their ideas to different domains where underlying rules might not be so clear or data so well-defined.
For some fields, like biology, placing large data sets into open source areas may bring a kind of convergence as collaboration ensues. But as Michael Nielsen points out in Reinventing Discovery, scientists have very little motivation to collaborate given the nature of publication, reputation, and tenure.
I seriously doubt that we have the intellectual infrastructure to support the collaborative capabilities of the Internet. We may well be able to connect all sorts of data and run all kinds of analyzes, but in the end, we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data.
A Telling Story
In the Scientific American article, Helbing relates an old story. A drunk man is looking for his keys under a street lamp. When asked why, he responds, "that's where the light is." To Helbing, it appears that he wants to use Big Data to create a brighter light so that scientists can peer beneath if for insight.
Scenario planners tell that story too, but we do so to make the point that while one is looking for keys in the light, they aren't paying any attention to the darkness. The future of Big Data lies not in the stories of anecdotal triumph that report sophisticated, but limited accomplishments--no, the future of Big Data rather lies in the darkness of context change, complexity, and overconfidence.
I will end, as Thaler did in his New York Times article, by quoting Mark Twain: "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."
Kessler in the WSJ: The Button That Made Facebook Billions
The power of 'Like' as an emotional sensor is driving the company's exorbitant valuation.
Andy Kessler, The Wall Street Journal, February 2, 2012
As bizarre as this sounds, one of the most valuable innovations in technology over the last several decades is Facebook's "Like" button. That's what has propelled the company to a galaxy-orbit valuation for its forthcoming initial public offering, filed Wednesday.
This is not only because the word "like" is, like, the identifying word of an entire generation. It's because computing has evolved beyond just taking directions from humans—and instead is cozying up to us and sniffing out our emotions and intent.
Advertisers are drooling. Procter & Gamble CEO Robert McDonald, while announcing a layoff of 1,600 people, said this week that the company is trimming its $10 billion annual ad budget as well because, he told analysts, "things like Facebook and Google" can be "much more efficient."
Computing's business model has evolved from selling hardware and software to enticing millions of users to one spot and then selling them something else. The industry has moved from the cool business of directly selling innovation to the age-old media model of selling ads—a $500 billion business.
And behind all the glitz, that's what Facebook is. Its tons of cheap servers sitting in dark rooms store our thoughts, photos and videos, to be delivered and shared to a select group of friends, our social network.
Advertisers rightly took note of the potential. But running ads next to pictures of your buddy Johnny funneling beers at a lacrosse game is not exactly what they had in mind. Then, in mid-2010, Facebook rolled out its Like button, which transformed the company from a somewhat interesting social network into a major media player. The power of Like as an emotional sensor is what's driving Facebook's exorbitant valuation.
Note that no one dominates this next wave of computing—not yet anyway. Instead, lots of companies own a piece of the market, and all are trying to eat the others' lunch.
Apple, worth $425 billion with $125 billion in annual sales, is a huge consumer platform. It keeps hardware and software tightly coupled, selling more than 100 million phones or tablets yearly. This lets it sell other stuff, like apps, and content like music and videos—and some ads. With the ability to track our clicks and location, it knows what all of us are doing minute by minute, sort of—but so far it leverages that by selling ever-better devices which we constantly upgrade.
Apple has already tried and failed to take on Facebook with a homegrown social network, Ping. But Apple is a threat to Facebook in many ways, especially since more than half of Facebook users, I'd guess, access the site via mobile phones.
Amazon, worth $80 billion with $48 billion in annual sales, sells us stuff directly. These sales are relatively low-margin, compared to software anyway. But Amazon leverages its infrastructure and 150 million users by selling excess capacity and Web services to smaller players, by selling ads, and to kick-start its own Kindle, which is now a lot more than an e-book reader. The Kindle Fire, with six million sold in the last three months, is a full-fledged shopping platform.
Google, worth $190 billion with $38 billion in annual sales, is the closest real competitor to Facebook. Google lures you to its site via its search engine and sells ads against results, paid per click. Yes, yes, Google has lots of other services, like email and maps, but basically it runs an ad platform. It's a great business with operating profits of 35%, similar to Facebook's.
Google+ is the next-largest social network to Facebook, and Google is scrambling to integrate your friends' opinions into search results. Google's Android phones are another gateway that Facebook and its users must traverse.
There are lots of other potential competitors trying to leverage our emotions. Twitter with its 140-character messaging. Netflix with streaming video. OpenTable with restaurant reservations. Microsoft with Skype video conferencing and Xbox videogames. Groupon and LivingSocial with social coupons. LinkedIn with business resumes and job search. Each provides some piece of productivity that users or businesses or advertisers will pay for. Each company that fails to dominate will end up being acquired or subsumed by the eventual winners.
Which brings us back to Facebook, which listed 2011 earnings of $3.71 billion in its IPO filing and is believed to have a potential worth of $75 billion to $100 billion—though I hear that some shares traded privately before the filing at an assumed valuation of $150 billion. Facebook doesn't sell phones or tablets, or ship physical products or even do searches. Instead, it has a vibrant, pulsating community of 845 million people willing to share their personal lives with others. Facebook is a giant emotional locker.
That's where the Like button comes in. The adage about advertising is that only half of ads are effective, but no one knows which half. So companies will drop $3.5 million to NBC for a 30-second Super Bowl ad. Or run Keystone Light ads on Comedy Central. They may work, but advertising—ask P&G—is an industry ripe for productive innovation. With the Like button, Facebook is like Bob Eubanks on "The Newlywed Game," who promised contestants "a prize chosen especially for you." Advertising's nirvana is an ad chosen especially for you. Of all the players, Facebook is the closest to delivering.
Others will catch up. Facebook CEO Mark Zuckerberg would do well to take the billions raised from the IPO and go on the offensive. Already—via a new subscribe feature with which you can "follow" someone—it is trying to neutralize Twitter. A Facebook phone? I doubt it, but lucrative deals with other hardware players are almost a must.
My best advice to Facebook is to keep playing poker and betting big while Apple, Amazon and Google move around their chess pieces playing defense. With a potential enterprise value at 25 times sales, a lot of things have to go right for Facebook over the next five years.
Mr. Kessler, a former hedge-fund manager, is the author most recently of "Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs" (Portfolio, 2011).
--------------------------------------------------
P Please consider the environment before printing this e-mail.
Andy Kessler, The Wall Street Journal, February 2, 2012
As bizarre as this sounds, one of the most valuable innovations in technology over the last several decades is Facebook's "Like" button. That's what has propelled the company to a galaxy-orbit valuation for its forthcoming initial public offering, filed Wednesday.
This is not only because the word "like" is, like, the identifying word of an entire generation. It's because computing has evolved beyond just taking directions from humans—and instead is cozying up to us and sniffing out our emotions and intent.
Advertisers are drooling. Procter & Gamble CEO Robert McDonald, while announcing a layoff of 1,600 people, said this week that the company is trimming its $10 billion annual ad budget as well because, he told analysts, "things like Facebook and Google" can be "much more efficient."
Computing's business model has evolved from selling hardware and software to enticing millions of users to one spot and then selling them something else. The industry has moved from the cool business of directly selling innovation to the age-old media model of selling ads—a $500 billion business.
And behind all the glitz, that's what Facebook is. Its tons of cheap servers sitting in dark rooms store our thoughts, photos and videos, to be delivered and shared to a select group of friends, our social network.
Advertisers rightly took note of the potential. But running ads next to pictures of your buddy Johnny funneling beers at a lacrosse game is not exactly what they had in mind. Then, in mid-2010, Facebook rolled out its Like button, which transformed the company from a somewhat interesting social network into a major media player. The power of Like as an emotional sensor is what's driving Facebook's exorbitant valuation.
Note that no one dominates this next wave of computing—not yet anyway. Instead, lots of companies own a piece of the market, and all are trying to eat the others' lunch.
Apple, worth $425 billion with $125 billion in annual sales, is a huge consumer platform. It keeps hardware and software tightly coupled, selling more than 100 million phones or tablets yearly. This lets it sell other stuff, like apps, and content like music and videos—and some ads. With the ability to track our clicks and location, it knows what all of us are doing minute by minute, sort of—but so far it leverages that by selling ever-better devices which we constantly upgrade.
Apple has already tried and failed to take on Facebook with a homegrown social network, Ping. But Apple is a threat to Facebook in many ways, especially since more than half of Facebook users, I'd guess, access the site via mobile phones.
Amazon, worth $80 billion with $48 billion in annual sales, sells us stuff directly. These sales are relatively low-margin, compared to software anyway. But Amazon leverages its infrastructure and 150 million users by selling excess capacity and Web services to smaller players, by selling ads, and to kick-start its own Kindle, which is now a lot more than an e-book reader. The Kindle Fire, with six million sold in the last three months, is a full-fledged shopping platform.
Google, worth $190 billion with $38 billion in annual sales, is the closest real competitor to Facebook. Google lures you to its site via its search engine and sells ads against results, paid per click. Yes, yes, Google has lots of other services, like email and maps, but basically it runs an ad platform. It's a great business with operating profits of 35%, similar to Facebook's.
Google+ is the next-largest social network to Facebook, and Google is scrambling to integrate your friends' opinions into search results. Google's Android phones are another gateway that Facebook and its users must traverse.
There are lots of other potential competitors trying to leverage our emotions. Twitter with its 140-character messaging. Netflix with streaming video. OpenTable with restaurant reservations. Microsoft with Skype video conferencing and Xbox videogames. Groupon and LivingSocial with social coupons. LinkedIn with business resumes and job search. Each provides some piece of productivity that users or businesses or advertisers will pay for. Each company that fails to dominate will end up being acquired or subsumed by the eventual winners.
Which brings us back to Facebook, which listed 2011 earnings of $3.71 billion in its IPO filing and is believed to have a potential worth of $75 billion to $100 billion—though I hear that some shares traded privately before the filing at an assumed valuation of $150 billion. Facebook doesn't sell phones or tablets, or ship physical products or even do searches. Instead, it has a vibrant, pulsating community of 845 million people willing to share their personal lives with others. Facebook is a giant emotional locker.
That's where the Like button comes in. The adage about advertising is that only half of ads are effective, but no one knows which half. So companies will drop $3.5 million to NBC for a 30-second Super Bowl ad. Or run Keystone Light ads on Comedy Central. They may work, but advertising—ask P&G—is an industry ripe for productive innovation. With the Like button, Facebook is like Bob Eubanks on "The Newlywed Game," who promised contestants "a prize chosen especially for you." Advertising's nirvana is an ad chosen especially for you. Of all the players, Facebook is the closest to delivering.
Others will catch up. Facebook CEO Mark Zuckerberg would do well to take the billions raised from the IPO and go on the offensive. Already—via a new subscribe feature with which you can "follow" someone—it is trying to neutralize Twitter. A Facebook phone? I doubt it, but lucrative deals with other hardware players are almost a must.
My best advice to Facebook is to keep playing poker and betting big while Apple, Amazon and Google move around their chess pieces playing defense. With a potential enterprise value at 25 times sales, a lot of things have to go right for Facebook over the next five years.
Mr. Kessler, a former hedge-fund manager, is the author most recently of "Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs" (Portfolio, 2011).
--------------------------------------------------
P Please consider the environment before printing this e-mail.
Wednesday, February 1, 2012
FT: Big innovation needs 'Big Data'
John Hamm, The Financial Times, February 1, 2012
‘Big data’ is all the rage, and for good reason. Companies need to sort through tons of it to understand what products their customers really want, and what they will buy. After all, their future depends on it. Yet, in spite of vast amounts of market intelligence and virtually unlimited information, how well do companies really know their customer?
In a world where, according to the Association of Product Management and product Marketing, more than 50 per cent of new technology products that enter the market fail, and roughly 75 per cent of consumer packaged goods and retail products fail to earn even $7.5m during their first year, we are clearly not doing a good job putting this data to work.
While marketing and manufacturing organisations have begun to take advantage of big data, the same cannot be said of product and service planning, a function most companies still regard as an art. The fact is that reliably creating high-demand products and services requires companies to balance that art with science. That’s where IT comes in.
For senior IT executives, though, the challenges are two-fold: First, to recognize the value of big data in mining customer needs and desires, and second, to devise a data management strategy that integrates big data into the front end of the innovation pipeline.
The third wave
Market research has evolved from the days of discovering customer needs principally on the basis of periodic surveys, focus groups, behavior observation and ethnographic studies. While still important sources of customer reaction, these techniques do not scale to the requirements of global markets or mass customisation.
Next, some companies began to incorporate collaboration technologies, which let customers directly influence product and service features. IBM and Siemens, for example, make use of crowdsourcing. Open innovation initiatives at Proctor & Gamble, Airbus, and Sara Lee are having a notable influence on product development.
Interestingly, a third wave now requires companies to accumulate data from sources residing at close proximity to the customer. In particular, online and brick-and-mortar retail, insurance, transportation and telecommunication companies, healthcare providers, utilities and banks collect large volumes of customer data.
There is transactional data from web searches, purchases, customer service calls and visits to the bank or doctor’s office; social media data from LinkedIn, Facebook, Twitter and blogs, online customer reviews and sentiment analysis, and data produced by the so-called ‘Internet of Things’ which includes location-based services and data from sensors and monitors embedded in a growing array of products, from automobiles to medical devices to household appliances.
With the evolution of software as a service and open source technologies, capturing and managing this data has become more cost effective for companies than ever before. However, sorting through and getting at the right information, and then putting it to work for product development is the real challenge. This requires a fully integrated approach to managing product life cycles so that each data point can be used to paint a full picture of what the customer really wants and needs, which can then be reflected in the end product.
This third wave will account for the biggest growth in big data. A fact made all the more vivid by the recent prediction by Hans Vestberg, Ericsson’s chief executive, that there will be 50bn internet-connected devices by 2020.
IT’s crucial role
That IT has had a positive impact on productivity and efficiency is undeniable. Now, as the pace of innovation quickens and the pressure to remain competitive intensifies, IT can help company innovators build the right products and services. This time, the task is to collect, synthesise, analyse and integrate big data and then merge those results with other sources of market research. There are three areas where IT can contribute.
The first is building the necessary infrastructure. This includes defining the data and computing architectures for extracting and analysing the data, and working with third-party sources of big data. It also means making the data available to existing innovation management and product lifecycle management systems so that those that are best able to act upon it, have easy access. A critical part of this infrastructure also includes developing a system that allows for end-to-end automation of information to avoid siloing and other classic roadblocks in the development cycle.
The second is identifying operational processes. This includes social media analytics and applications for data visualisation, collaboration and business intelligence. And it includes helping to define new business processes and workflows for product and service development organisations.
Third, is advocating for proper policies and governance. The availability of sensitive data to more organisations carries with it new attendant risks. These risks must be assessed in the context of new usage models and existing security, privacy and data access policies.
Eyes on the prize
The barriers to incorporating big data into the innovation process can be significant, ranging from cultural to organisational, technological and budgetary. However, the opportunities are also clear: quicker to market with new products and services; identifying weak product and service candidates earlier in the development cycle; eliminating features that customers don’t want while adding features they are will to pay a premium for; and identifying and prioritising requirements for specific markets.
The bottom line is that the more you know about your customers, and the more you incorporate that knowledge into products and service, the more positive the impact on revenues, margins and market share.
John Hamm is the CEO of Accept Software and the author of Unusually Excellent .
‘Big data’ is all the rage, and for good reason. Companies need to sort through tons of it to understand what products their customers really want, and what they will buy. After all, their future depends on it. Yet, in spite of vast amounts of market intelligence and virtually unlimited information, how well do companies really know their customer?
In a world where, according to the Association of Product Management and product Marketing, more than 50 per cent of new technology products that enter the market fail, and roughly 75 per cent of consumer packaged goods and retail products fail to earn even $7.5m during their first year, we are clearly not doing a good job putting this data to work.
While marketing and manufacturing organisations have begun to take advantage of big data, the same cannot be said of product and service planning, a function most companies still regard as an art. The fact is that reliably creating high-demand products and services requires companies to balance that art with science. That’s where IT comes in.
For senior IT executives, though, the challenges are two-fold: First, to recognize the value of big data in mining customer needs and desires, and second, to devise a data management strategy that integrates big data into the front end of the innovation pipeline.
The third wave
Market research has evolved from the days of discovering customer needs principally on the basis of periodic surveys, focus groups, behavior observation and ethnographic studies. While still important sources of customer reaction, these techniques do not scale to the requirements of global markets or mass customisation.
Next, some companies began to incorporate collaboration technologies, which let customers directly influence product and service features. IBM and Siemens, for example, make use of crowdsourcing. Open innovation initiatives at Proctor & Gamble, Airbus, and Sara Lee are having a notable influence on product development.
Interestingly, a third wave now requires companies to accumulate data from sources residing at close proximity to the customer. In particular, online and brick-and-mortar retail, insurance, transportation and telecommunication companies, healthcare providers, utilities and banks collect large volumes of customer data.
There is transactional data from web searches, purchases, customer service calls and visits to the bank or doctor’s office; social media data from LinkedIn, Facebook, Twitter and blogs, online customer reviews and sentiment analysis, and data produced by the so-called ‘Internet of Things’ which includes location-based services and data from sensors and monitors embedded in a growing array of products, from automobiles to medical devices to household appliances.
With the evolution of software as a service and open source technologies, capturing and managing this data has become more cost effective for companies than ever before. However, sorting through and getting at the right information, and then putting it to work for product development is the real challenge. This requires a fully integrated approach to managing product life cycles so that each data point can be used to paint a full picture of what the customer really wants and needs, which can then be reflected in the end product.
This third wave will account for the biggest growth in big data. A fact made all the more vivid by the recent prediction by Hans Vestberg, Ericsson’s chief executive, that there will be 50bn internet-connected devices by 2020.
IT’s crucial role
That IT has had a positive impact on productivity and efficiency is undeniable. Now, as the pace of innovation quickens and the pressure to remain competitive intensifies, IT can help company innovators build the right products and services. This time, the task is to collect, synthesise, analyse and integrate big data and then merge those results with other sources of market research. There are three areas where IT can contribute.
The first is building the necessary infrastructure. This includes defining the data and computing architectures for extracting and analysing the data, and working with third-party sources of big data. It also means making the data available to existing innovation management and product lifecycle management systems so that those that are best able to act upon it, have easy access. A critical part of this infrastructure also includes developing a system that allows for end-to-end automation of information to avoid siloing and other classic roadblocks in the development cycle.
The second is identifying operational processes. This includes social media analytics and applications for data visualisation, collaboration and business intelligence. And it includes helping to define new business processes and workflows for product and service development organisations.
Third, is advocating for proper policies and governance. The availability of sensitive data to more organisations carries with it new attendant risks. These risks must be assessed in the context of new usage models and existing security, privacy and data access policies.
Eyes on the prize
The barriers to incorporating big data into the innovation process can be significant, ranging from cultural to organisational, technological and budgetary. However, the opportunities are also clear: quicker to market with new products and services; identifying weak product and service candidates earlier in the development cycle; eliminating features that customers don’t want while adding features they are will to pay a premium for; and identifying and prioritising requirements for specific markets.
The bottom line is that the more you know about your customers, and the more you incorporate that knowledge into products and service, the more positive the impact on revenues, margins and market share.
John Hamm is the CEO of Accept Software and the author of Unusually Excellent .
MIT Sloan course to tackle 'Big Data'
Chris Reidy, The Boston Globe, February 01, 2012,
Companies are now so overwhelmed by information overload that the MIT Sloan School of Management said it is gearing up to teach a two-day course that aims to help executives cope with massive data dumps and extract meaningful insights from real-time digital data.
Set for March 27 and March 28, the course is titled, “Big Data: Making Complex Things Simpler.” In an MIT Sloan press release, professor Erik Brynjolfsson, director of MIT’s Center for Digital Business, commented on the phenomenon of “Big Data.” According to Brynjolfsson, companies adept at using “Big Data” are “around 5 percent more productive and profitable than their competitors.”
Professor Alex ‘Sandy’ Pentland, director of MIT’s Media Lab Entrepreneurship Program, also teaches in the Big Data program.
“Many managers undervalue the worth of data, but really it is like money in your bank account and you should be getting a return on it,” Pentland said in a statement. “This program will show managers how to capture the benefits of data such as creating better customer analytics or capturing real-time consumer preferences.”
Mobile phones are one rich source of Big Data, Pentland noted. On Black Friday, the big shopping day on the Friday after Thanksgiving, projections on sales could be based on mobile phone activity. Even more intriguing perhaps, mobile phone activity has the potential to help saloon-keepers to track the movements of potential patrons.
“We could predict Black Friday sales before certain stores make their own announcements based on how many mobile phones were in their parking lots that day,” Pentland said. “We also can find out which bars are popular at 9:00 p.m. versus 11:00 p.m. with this type of mobile phone data.”
Companies are now so overwhelmed by information overload that the MIT Sloan School of Management said it is gearing up to teach a two-day course that aims to help executives cope with massive data dumps and extract meaningful insights from real-time digital data.
Set for March 27 and March 28, the course is titled, “Big Data: Making Complex Things Simpler.” In an MIT Sloan press release, professor Erik Brynjolfsson, director of MIT’s Center for Digital Business, commented on the phenomenon of “Big Data.” According to Brynjolfsson, companies adept at using “Big Data” are “around 5 percent more productive and profitable than their competitors.”
Professor Alex ‘Sandy’ Pentland, director of MIT’s Media Lab Entrepreneurship Program, also teaches in the Big Data program.
“Many managers undervalue the worth of data, but really it is like money in your bank account and you should be getting a return on it,” Pentland said in a statement. “This program will show managers how to capture the benefits of data such as creating better customer analytics or capturing real-time consumer preferences.”
Mobile phones are one rich source of Big Data, Pentland noted. On Black Friday, the big shopping day on the Friday after Thanksgiving, projections on sales could be based on mobile phone activity. Even more intriguing perhaps, mobile phone activity has the potential to help saloon-keepers to track the movements of potential patrons.
“We could predict Black Friday sales before certain stores make their own announcements based on how many mobile phones were in their parking lots that day,” Pentland said. “We also can find out which bars are popular at 9:00 p.m. versus 11:00 p.m. with this type of mobile phone data.”
Subscribe to:
Posts (Atom)