Phil Galewitz, Kaiser Health News, January 18, 2012
By Brett T. Roseman, for USA TODAY
Steven Boyd of Joliet, Ill., and wife Karol received a direct mailing from their local hospital offering lung scans.
Boyd, 59, of Joliet, Ill., had smoked for decades, as had his wife, Karol.
Provena didn't send the mailing to everyone who lived near the hospital, just those who had a stronger likelihood of having smoked based on their age, income, insurance status and other demographic criteria.
The non-profit facility is one of a growing number of hospitals using their patients' health and financial records to help pitch their most lucrative services, such as cancer, heart and orthopedic care. As part of these direct mail campaigns, they are also buying detailed information about local residents compiled by consumer marketing firms — everything from age, income and marital status to shopping habits and whether residents have children or pets at home.
Hospitals say they are promoting needed services, such as cancer screenings and cholesterol tests, but they often use the data to target patients with private health insurance, which typically pay higher rates than government coverage. At an industry conference last year, Provena Health marketing executive Lisa Lagger said such efforts had helped attract higher-paying patients, including those covered by "profitable Blue Cross and less Medicare."
While the strategies are increasing revenues, they are drawing fire from patient advocates and privacy groups who criticize the hospitals for using private medical records to pursue profits.
Doug Heller, executive director of Consumer Watchdog, a California-based consumer advocacy group, says he is bothered by efforts to "cherry pick" the best-paying patients.
"When marketing is picking and choosing based on people's financial status, it is inherently discriminating against patients who have every right and need for medical information," Heller says. Deven McGraw, director of the health privacy project at the Center for Democracy and Technology in Washington, says federal law allows hospitals to use confidential medical records to keep patients informed about services that may help them.
"You want health providers to communicate to patients about health options that may be beneficial," McGraw says. "But sometimes this is about generating business for a new piece of equipment that the hospital just bought."
Using such information for marketing "creeps closer to the line" between what is legal and what is not, she says.
Hospital officials such as Denise Beaudoin of Detroit's Henry Ford Health System say what they do is legal and that the sophisticated targeting approach — called "customer relationship marketing" — simply helps them deliver information to those most likely to use it.
They say hospitals are adopting strategies used for decades by the retail, travel and communications industries, which have flourished with the growth of online companies such as Amazon and Google. For example: Buy a book on Amazon and it will suggest a title with a similar subject. Search for information on Alaskan vacations on Google, and an ad pops up for a cruise line.
At a time when government and private insurers are tightening reimbursements, more hospitals are turning to similar approaches to drive admissions. An estimated 20% now use the strategy, including large academic medical centers and large chains, such as HCA, based in Nashville, and Trinity Health, based in Novi, Mich. The trend is expected to accelerate as more hospitals adopt electronic health records, says Guy Miller, a Chicago health care consultant.
Putting patient data to work
Tess Niehaus, vice president of marketing at St. Anthony's Medical Center in St. Louis, makes no apologies for going after the most lucrative business.
"We are here to serve everybody, but we market for good-paying patients because it preserves our ability to serve everyone," she says.
St. Anthony's marketers use patient data to personalize mailings with an individual's name and a picture of someone of similar age or gender. It is more expensive, but the strategy results in better response rates, she says. From October 2010 through July 2011, St. Anthony's spent $25,000 on a targeted mailing to 40,000 women for mammogram screenings. The letters led 1,000 women to get the test, which generated $530,000 in revenue from screenings, biopsies and other related services, she says.
To help devise such campaigns, St. Anthony's and other hospitals share patient data with marketing staff and outside consultants. Anyone with access to patient records is required by federal law to sign non-disclosure agreements.
While the practice is legal, most people would be shocked to know their records may be shared with non-medical personnel and outside firms to help hospitals attract business, says Pam Dixon, executive director of the World Privacy Forum, an advocacy group based in California.
"I am really bothered by the overabundance of information that is flowing that is unnecessary and risky," she says.
Screenings lead to more tests
While hospitals may profit from offering cholesterol tests and mammograms, the big payoff is in what those screenings may lead to — additional tests and procedures, including surgery.
"The old adage in business is that it's easier to sell an existing customer new services rather than find a new customer," says Patrick Kane, senior vice president of marketing at Cape Cod Healthcare in Massachusetts who used such approaches at Wellmont Health System in Kingsport, Tenn.
Provena's six hospitals in Illinois embraced targeted marketing in 2010, mailing information about screenings and educational events to 293,000 people. The mailings led to more than 50,000 patient visits — a 17% response rate, several times that typically seen in direct mail efforts, according to the industry presentation to hospital officials last year in Orlando. After accounting for marketing costs, those visits netted the system $595,000.
Provena's Lagger says the approach boosted the system's bottom line so it could serve people regardless of insurance status. "This is a means to an end," she says.
One of the biggest pluses for hospital executives is that they can track a campaign's success by comparing the amount of services used by targeted consumers with a control group with the same demographic and economic characteristics but who are not sent mailings.
Mailings get results
When the Henry Ford Health System promoted mammograms last year in mailings to 30,000 women aged 40 or older, more than 5,700 responded — 304 more than in the control group. The mailings generated $268,000 more in profit than the control group — a return of more than four to one on the cost of the campaign, says Beaudoin, vice president of customer engagement.
"Some doctors used to be leery about the effectiveness of these marketing campaigns, but not when we can show them data like this," she says.
Mercy Health Partners in western Michigan, part of the 47-hospital non-profit Trinity Health system, sent a targeted cardiac screening mailing last year to 7,450 people. That resulted in 1,729 patient visits, or 7% more than in a control group. The campaign generated about $1 million in revenue and about $50,000 in profit.
"It's a much more efficient use of marketing dollars," says Preston Gee, Trinity's senior vice president of strategic planning. "People like having information tailored to their own needs."
Much of the expertise for such campaigns is provided by three consulting firms — CPM Marketing of Madison, Wis., Medseek of Birmingham, Ala., and New York-based Thomson Reuters. They typically charge hospitals $100,000 a year or more.
CPM, which merged in November with Denver-based HealthGrades, a health-ratings firm, added 100 new hospitals last year to give it a total of 400. Medseek works with more than 250 hospitals and Thomson Reuters with 150.
"There are a lot of very rich data in health care beyond just age and gender that help steer or guide people to health services," says John Hallick, president of CPM.
Loss of privacy
Boyd, the Joliet home inspector, was not upset that Provena Health used information about him and his wife — both former patients — to pitch screening tests. "We lost our privacy long ago, and I don't like to think about all the information that's out there about us."
Provena marketing manager Richard Matula would not say why the Boyds were included in the mailing, citing patient privacy laws. Patients' smoking status was not used in the criteria, he said.
The targeting worked in the case of Boyd, who called the number on the back and scheduled the CT scan a few days later. The $169 test showed his lungs were clear, but found potential blockages in coronary arteries that his Provena-affiliated doctor is monitoring.
"In hindsight, I'm glad I had the test," he says.
Kaiser Health News is an editorially independent program of the Henry J. Kaiser Family Foundation, a non-profit, non-partisan health policy research and communication organization not affiliated with Kaiser Permanente.
Monday, February 6, 2012
Thursday, February 2, 2012
Why Big Data Won’t Make You Smart, Rich, Or Pretty
FC Expert Blogger Daniel W. Rasmus, Fast Company, January 27, 2012
If 2012 is the year of Big Data, it will likely be the year vendors and consultants start to over-promise, under-deliver, and put processes in motion that will generate insights and potential risks for years to come.
This year will be the year of Big Data. The Data Warehousing Institute (TDWI) reported that 90 percent of the IT professionals it surveyed said they were familiar with big data analytics. And 34 percent said they already applied analytics to Big Data. The vast hordes of data collection during e-commerce transactions, from loyalty programs, employment records, supply chain and ERP systems are, or are about to get, cozy. Uncomfortably cozy.
Let me start by saying there is nothing inherently wrong with Big Data. Big Data is a thing, and like anything, it can be used for good or for evil. It can be used appropriately given known limitations, or stretched wantonly until its principles fray. For now, the identification, consolidation, and governance of data is an appropriate step, as Forbes's Tom Groenfeldt recently documented with Michigan's $19 million in data center consolidation savings. Dirk Helbing of the Swiss Federal Institute of Technology in Zurich is more ambitious. His €1-billion project, the topic of the December 2011 Scientific American cover story, seeks to do nothing less than foretell the future.
The meaningful use of Big Data lies somewhere between these two extremes. For Big Data to move from anything more than an instantiation of databases running in logical or physical proximity, to data that can be meaningfully mined for insight, requires new skills, new perspectives, and new cautions.
The Big Data Dream
Dirk Helbing seeks a system that is akin to Asimov's Psychohistory as imagined in the Foundation series. In broad swaths, it would anticipate the future by linking social, scientific, and economic data. This system could be used to help advise world governments on the most salient choices to make.
Reading the article in Scientific American reminded me of a science fiction story by Tribble-inventor David Gerrold—When Harlie Was One. In this book, Harlie, which stands for Human Analog Robot Life Input Equivalents, decides that he needs answers, and that he isn't sophisticated enough to solve his own problem and therefore keep the corporate interests that built him interested enough to keep him plugged in. So he designs a new computer, the Graphic Omniscient Device, or GOD, as a proof of his value. GOD will answer all questions submitted to it. Unfortunately, as the human engineers building GOD eventually realize, the processing capacity is so vast, that GOD will not be able to provide an answer to any question during the lifetime of a human.
Harlie, of course, knew this all along. He needed the humans for three reasons: to keep him running, for engineering labor to build GOD, and to ask the questions that GOD will answer.
Given the woes of Europe, spending €1-billion on such a project will likely prove to be wasted money. We, of course, don't have a mechanical futurist to evaluate that position, but we do have history. Whenever there is an existential problem facing the world, charlatans appear to dazzle the masses with feats of magic and wonder. I don't see this proposal being anything more than the latest version of apocalyptic sorcery. It's not that a big science project can't yield interesting outcomes, but if you look at the Microelectronics and Computer Technology Corporation (MCC), late of Austin, Texas, we find Cyc, a system conceived at the beginning of the computer era, to combat Japan's Fifth Generation Project as it supposedly threatened to out-innovate America's nascent lead in computer technology.
Although Cyc has yielded some use, it has not yet become the artificial human mind it was intended to be, able to converse naturally with anyone about the events, concepts, and objects in the world. And artificial intelligence, as imagined in the 1980s, has yet to transform the human condition.
As Big Data becomes the next great savior of business and humanity, we need to remain skeptical of its promises as well as its applications and aspirations.
Existential Issues With Big Data
Determinism teaches that what will be, will be. Existentialism deals with a humanity in the throes of chaos. Big Data can be seen as either a lens through which determinism is revealed, or a tool for navigating an existential world.
As a scenario planner, I take the existential position and see a number of existential threats to the success of Big Data and its applications.
Overconfidence: Many managers creating a project plan, drawing up a budget, or managing a hedge fund trust their forecasts based on personal abilities and confidence in their knowledge and experience. As University of Chicago professor Richard H. Thaler recently pointed out in the New York Times ("The Overconfidence Problem in Forecasting"), most managers are overconfident and miscalibrated. In other words, they don't recognize their own inability to forecast the future, nor do they recognize the inherent volatility of markets. Both of these traits portend big problems for Big Data as humans code their assumptions about the world into algorithms: people don't understand their personal limitations, nor do they recognize if a model is good or not.
When learning happens: Even in a field as seemingly physical and visceral as fossil hunting, Big Data is playing a role. Geologic data has been fed into a model that helps pinpoint good fossil-hunting fields. On the surface that appears a useful discovery, but if you dig a bit deeper, you find a lesson for would-be Big Data modelers. As technology and data sophistication increases, the underlying assumptions in the model must change. Current data, derived from the analysis of Landsat photos, can direct field workers toward a fairly large, but promising area with multiple types of rock exposures. Eventually the team hopes to increase their 15-meter resolution to 15-centimeter resolution by acquiring higher-resolution data. As they examine the new data, they will need to change their analysis approach to recognize features not previously available (for more see "Artificial intelligence joins the fossil hunt" in New Scientist). Learning will mean reinterpreting the model.
On a more abstract level, recent work conducted by ETH Zurich looked at 43,000 transnational companies seeking to understand the relationships between those companies and their potential for influence. This analysis found that 1,318 companies were tightly connected, with an average of 20 connections, representing about 60 percent of global revenues. Deeper analysis revealed a "super-entity" of 147 firms that accounts for about 40 percent of the wealth in the network. This type of analysis has been conducted before, but the Zurich team included indirect ownership, which changed the outcome significantly (for more see "The network of global control" by Bitali, Glattfelder, and Battiston).
If organizations rely on Big Data to connect far-ranging databases--well beyond corporate ownership or maps of certain geologies--who, it must be asked, will understand enough of the model to challenge its underlying assumptions, and re-craft those assumptions when the world, and the data that reflects it, changes?
Complexity: I was sitting with the CIO of a large insurance company in Portland. We were talking about generational hand-offs when he raised the issue of an Excel spreadsheet used to evaluate commercial property underwriting. He said one of the older members of the organization owned that spreadsheet and he was the only one who knew how it worked. The hand-off issue was not one of getting the older employee to collaborate with the younger employee, but one of complexity. That spreadsheet was complex and tightly woven into the employee's worldview. Although the transfer could theoretically take place, it is unknowable how long it would take, if the new employee would stay, or how the process would change as multiple worldviews collided. Combining models full of nuance and obscurity increases complexity. Organizations that plan complex uses of Big Data and the algorithms that analyze the data need to think about continuity and succession planning in order to maintain the accuracy and relevance of their models over time, and they need to be very cautious about the time it will take to integrate, and the value of results achieved, from data and models that border on the cryptic.
Feedback Loops: Big Data isn't just about the size of well-understood data sets, it is about linking disparate data sets and then creating connective tissue, either through design or inference, between these data sets. At the onset of the Great Recession, we experienced a feedback loop failure as David X Li's famous Gaussian copula function, a seemingly well-tested approach to analyzing financial risk, failed to anticipate the risks lurking outside of its models. People kept trading, assuming their risk analysis was still meaningful. No feedback loop existed to inform the bond markets that their credit default swaps were an inverse pyramid teetering on a bed of miscalculation.
Algorithms and a Lack of Theory: It is not only algorithms that can go wrong when a theory proves incorrect or the assumptions underlying the algorithm change. There are places where no theory exists at any level of consensus to be meaningful. The impact of education (and the effectiveness of various approaches), how innovation works, or what triggers a fad are examples of behaviors for which little valid theory exists--it's not that plenty of opinion about various approaches or models is lacking, but that a theory, in the scientific sense, is nonexistent. For Big Data that means a number of things, first and foremost, that if you don't have a working theory, you probably don't know what data you need to test any hypotheses you may posit. It also means that data scientists can't create a model because no reliable underlying logic exists that can be encoded into a model.
Confirmation Bias: Every model is based on historical assumptions and perceptual biases. Regardless of the sophistication of the science, we often create models that help us see what we want to see, using data selected as a good indicator of such a perception. Take a recent debate about how to price futures. Future events are typically discounted using an exponential model that creates a regular discount rate that eventually leads a value of zero for far-flung events. Exponential discounting takes a deterministic view. A more existential view comes from the proponents of hyperbolic discounting, which creates a preference for rewards that arrive sooner than later. With hyperbolic discounting, discounts of future events fall more gradually, leading to what might be called "irrational behavior." Another version is called the declining discount rate.
This "discount" debate points out that even when a model exists that is designed to aid in decision making about the future, that model may involve contentious disagreements about its validity and alternative approaches that yield very different results. These are important debates in the world of Big Data. One group of modelers advocates for one approach, and another group, an alternative approach, both using sophisticated data and black boxes (as far as the uninitiated business person is concerned) to support their cases. The fact is that in cases like this, no one knows the answer definitively as the application may be contextual or it may be incomplete (e.g., a new approach may solve the issue that none of the current approaches solves completely). What can be said, and what must be remembered is, the adage that "a futurist is never wrong today." Who wins these debates today may be meaningless because the implications have no near-term consequences, but companies that accept one approach over the other may be betting their firm's future on wishful thinking and unwillingness to admit what they don't know.
The World Changes: We must remember that all data is historical. There is no data from or about the future. Future context changes cannot be built into a model because they cannot be anticipated. Consider this: 2012 is the 50th anniversary of the 1962 Seattle World's Fair. In 1962, the retail world was dominated by Sears, Montgomery Ward, Woolworth, A&P, and Kresge.
Some of those companies no longer exist, and others have merged to the point that they are unrecognizable from their 1962 incarnations. Also in 1962, thousands of miles away, a small company opened its first location in Rogers, Ark.--the Wal-Mart Discount City. Would models of retail supply chains built in 1962 be able to anticipate the overwhelming disruption that this humble storefront would cause for retail? Did Sam Walton understand the impact of Amazon.com when it went live in 1995?
The answer to all of the above is "no." These innovations are rare and hugely disruptive. They are far from doomsday scenarios except for firms so entrenched in their models that they can't adapt. And there is the problem for Big Data. As with Li's risk analysis algorithm, when the world changed, the model did not. Feedback loops are important as a way of maintaining relevance through incremental improvement, but what happens when the world changes so much that current assumptions become irrelevant and the clock must be started again. Not only must we remember that all data is historical, but we must also remember that at some point historical data becomes irrelevant when the context changes.
Motives: In a recent BusinessWeek article (Palantir, "The War on Terror's Secret Weapon"), Peter Theil is quoted as saying: "We cannot afford to have another 9/11 event in the U.S. or anything bigger than that. That day opened the doors to all sorts of crazy abuses and draconian policies." Theil, characterized in the article as a libertarian, sees the analysis of data as a positive for civil liberties. That position is debatable as groups like NO2ID.net form to campaign against not only the use of Big Data, but its creation. Given the complexity of the data and associated models, along with various intended of unintended biases, organizations have to go out of their way to discern the motives of those developing analytics models, lest they allow programs to manipulate data in a way that may precipitate negative social, legal, or fiduciary outcomes.
Acting on the Model: Consider crime analysis. George Mohler of Santa Clara University in California has applied equations that predict earthquake aftershocks to crime. By using location and data and times of recent crimes, the system predicts "aftercrimes." This kind of anticipatory data may result in bastions of police flooding a neighborhood following one burglary. With no police presence, the anticipated crimes may well take place. If the burglars, however, see an increase in surveillance and police activity, they may abandon planned targets and seek new ones, thus invalidating the models' predictions, potentially in terms of time and location. The proponents of Big Data need to ensure that the users of their models understand the intricacies of trend analysis, what a trend really is, and the implications of acting on a model's recommendations.
Where Big Data Will Work
Some of the emerging Big Data stories don't test the existential limits of technology, nor do they threaten global catastrophe. The worst outcome in dinosaur fossil hunting is not finding dinosaur bones where you expect them, and the worst outcome of a crime prediction is a burglary that doesn't take place.
Big Data will no doubt be used to target advertising, reduce fraud, fight crime, find tax evaders, collect child support payments, create better health outcomes, and myrid other activities from the mundane to the ridiculous.
And along the way, the software companies and those who invested in Big Data will share their stories.
Companies like monumental con
structor Arup use Big Data as a way to better model the use of the buildings they build. The Arup software arm, Oasys, recently acquired MassMotion to help them understand the flow of people through buildings. MassMotion can model the intimacy of a coffee shop or the flow of hundreds of thousands through a terminal or metro system using its agent technology. His models are informed from data about traffic patterns, arrival and departure times for various forms of transportation and the environment, such as shopping locations and information desks where people might stop or congregate. The result is a model, sometimes with thousands of avatars, pushing and shoving, congregating and separating--all based on MassMotion's Erin Morrow and how he perceives the world.
Another movement oriented application of Big Data, Jyotish (Sanskrit for astrology), comes from Boeing's research cener at the University of Illinois in Urbana-Champaign. This application predicts the movement of work crews within Boeing's factories. It will ultimately help them figure out how to save costs and increase satisfaction by ensuring that services, like Wi-Fi, are available where and when they are needed.
Palantir, the Palo Alto-based startup focused on solving the intelligence problem of 9/11, discovers correlations in the data that informs military and intelligence agencies who, what, and when a potential threat turns into an imminent threat. With access to data models and data across government silos, Palantir may well make its hero case more often than not in individual cases. They have to be cautious about applying their ideas to different domains where underlying rules might not be so clear or data so well-defined.
For some fields, like biology, placing large data sets into open source areas may bring a kind of convergence as collaboration ensues. But as Michael Nielsen points out in Reinventing Discovery, scientists have very little motivation to collaborate given the nature of publication, reputation, and tenure.
I seriously doubt that we have the intellectual infrastructure to support the collaborative capabilities of the Internet. We may well be able to connect all sorts of data and run all kinds of analyzes, but in the end, we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data.
A Telling Story
In the Scientific American article, Helbing relates an old story. A drunk man is looking for his keys under a street lamp. When asked why, he responds, "that's where the light is." To Helbing, it appears that he wants to use Big Data to create a brighter light so that scientists can peer beneath if for insight.
Scenario planners tell that story too, but we do so to make the point that while one is looking for keys in the light, they aren't paying any attention to the darkness. The future of Big Data lies not in the stories of anecdotal triumph that report sophisticated, but limited accomplishments--no, the future of Big Data rather lies in the darkness of context change, complexity, and overconfidence.
I will end, as Thaler did in his New York Times article, by quoting Mark Twain: "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."
This blog is written by a member of our expert blogging community and expresses that expert's views alone.
This year will be the year of Big Data. The Data Warehousing Institute (TDWI) reported that 90 percent of the IT professionals it surveyed said they were familiar with big data analytics. And 34 percent said they already applied analytics to Big Data. The vast hordes of data collection during e-commerce transactions, from loyalty programs, employment records, supply chain and ERP systems are, or are about to get, cozy. Uncomfortably cozy.
Let me start by saying there is nothing inherently wrong with Big Data. Big Data is a thing, and like anything, it can be used for good or for evil. It can be used appropriately given known limitations, or stretched wantonly until its principles fray. For now, the identification, consolidation, and governance of data is an appropriate step, as Forbes's Tom Groenfeldt recently documented with Michigan's $19 million in data center consolidation savings. Dirk Helbing of the Swiss Federal Institute of Technology in Zurich is more ambitious. His €1-billion project, the topic of the December 2011 Scientific American cover story, seeks to do nothing less than foretell the future.
The meaningful use of Big Data lies somewhere between these two extremes. For Big Data to move from anything more than an instantiation of databases running in logical or physical proximity, to data that can be meaningfully mined for insight, requires new skills, new perspectives, and new cautions.
The Big Data Dream
Dirk Helbing seeks a system that is akin to Asimov's Psychohistory as imagined in the Foundation series. In broad swaths, it would anticipate the future by linking social, scientific, and economic data. This system could be used to help advise world governments on the most salient choices to make.
Reading the article in Scientific American reminded me of a science fiction story by Tribble-inventor David Gerrold—When Harlie Was One. In this book, Harlie, which stands for Human Analog Robot Life Input Equivalents, decides that he needs answers, and that he isn't sophisticated enough to solve his own problem and therefore keep the corporate interests that built him interested enough to keep him plugged in. So he designs a new computer, the Graphic Omniscient Device, or GOD, as a proof of his value. GOD will answer all questions submitted to it. Unfortunately, as the human engineers building GOD eventually realize, the processing capacity is so vast, that GOD will not be able to provide an answer to any question during the lifetime of a human.
Harlie, of course, knew this all along. He needed the humans for three reasons: to keep him running, for engineering labor to build GOD, and to ask the questions that GOD will answer.
Given the woes of Europe, spending €1-billion on such a project will likely prove to be wasted money. We, of course, don't have a mechanical futurist to evaluate that position, but we do have history. Whenever there is an existential problem facing the world, charlatans appear to dazzle the masses with feats of magic and wonder. I don't see this proposal being anything more than the latest version of apocalyptic sorcery. It's not that a big science project can't yield interesting outcomes, but if you look at the Microelectronics and Computer Technology Corporation (MCC), late of Austin, Texas, we find Cyc, a system conceived at the beginning of the computer era, to combat Japan's Fifth Generation Project as it supposedly threatened to out-innovate America's nascent lead in computer technology.
Although Cyc has yielded some use, it has not yet become the artificial human mind it was intended to be, able to converse naturally with anyone about the events, concepts, and objects in the world. And artificial intelligence, as imagined in the 1980s, has yet to transform the human condition.
As Big Data becomes the next great savior of business and humanity, we need to remain skeptical of its promises as well as its applications and aspirations.
Existential Issues With Big Data
Determinism teaches that what will be, will be. Existentialism deals with a humanity in the throes of chaos. Big Data can be seen as either a lens through which determinism is revealed, or a tool for navigating an existential world.
As a scenario planner, I take the existential position and see a number of existential threats to the success of Big Data and its applications.
Overconfidence: Many managers creating a project plan, drawing up a budget, or managing a hedge fund trust their forecasts based on personal abilities and confidence in their knowledge and experience. As University of Chicago professor Richard H. Thaler recently pointed out in the New York Times ("The Overconfidence Problem in Forecasting"), most managers are overconfident and miscalibrated. In other words, they don't recognize their own inability to forecast the future, nor do they recognize the inherent volatility of markets. Both of these traits portend big problems for Big Data as humans code their assumptions about the world into algorithms: people don't understand their personal limitations, nor do they recognize if a model is good or not.
When learning happens: Even in a field as seemingly physical and visceral as fossil hunting, Big Data is playing a role. Geologic data has been fed into a model that helps pinpoint good fossil-hunting fields. On the surface that appears a useful discovery, but if you dig a bit deeper, you find a lesson for would-be Big Data modelers. As technology and data sophistication increases, the underlying assumptions in the model must change. Current data, derived from the analysis of Landsat photos, can direct field workers toward a fairly large, but promising area with multiple types of rock exposures. Eventually the team hopes to increase their 15-meter resolution to 15-centimeter resolution by acquiring higher-resolution data. As they examine the new data, they will need to change their analysis approach to recognize features not previously available (for more see "Artificial intelligence joins the fossil hunt" in New Scientist). Learning will mean reinterpreting the model.
On a more abstract level, recent work conducted by ETH Zurich looked at 43,000 transnational companies seeking to understand the relationships between those companies and their potential for influence. This analysis found that 1,318 companies were tightly connected, with an average of 20 connections, representing about 60 percent of global revenues. Deeper analysis revealed a "super-entity" of 147 firms that accounts for about 40 percent of the wealth in the network. This type of analysis has been conducted before, but the Zurich team included indirect ownership, which changed the outcome significantly (for more see "The network of global control" by Bitali, Glattfelder, and Battiston).
If organizations rely on Big Data to connect far-ranging databases--well beyond corporate ownership or maps of certain geologies--who, it must be asked, will understand enough of the model to challenge its underlying assumptions, and re-craft those assumptions when the world, and the data that reflects it, changes?
Complexity: I was sitting with the CIO of a large insurance company in Portland. We were talking about generational hand-offs when he raised the issue of an Excel spreadsheet used to evaluate commercial property underwriting. He said one of the older members of the organization owned that spreadsheet and he was the only one who knew how it worked. The hand-off issue was not one of getting the older employee to collaborate with the younger employee, but one of complexity. That spreadsheet was complex and tightly woven into the employee's worldview. Although the transfer could theoretically take place, it is unknowable how long it would take, if the new employee would stay, or how the process would change as multiple worldviews collided. Combining models full of nuance and obscurity increases complexity. Organizations that plan complex uses of Big Data and the algorithms that analyze the data need to think about continuity and succession planning in order to maintain the accuracy and relevance of their models over time, and they need to be very cautious about the time it will take to integrate, and the value of results achieved, from data and models that border on the cryptic.
Feedback Loops: Big Data isn't just about the size of well-understood data sets, it is about linking disparate data sets and then creating connective tissue, either through design or inference, between these data sets. At the onset of the Great Recession, we experienced a feedback loop failure as David X Li's famous Gaussian copula function, a seemingly well-tested approach to analyzing financial risk, failed to anticipate the risks lurking outside of its models. People kept trading, assuming their risk analysis was still meaningful. No feedback loop existed to inform the bond markets that their credit default swaps were an inverse pyramid teetering on a bed of miscalculation.
Algorithms and a Lack of Theory: It is not only algorithms that can go wrong when a theory proves incorrect or the assumptions underlying the algorithm change. There are places where no theory exists at any level of consensus to be meaningful. The impact of education (and the effectiveness of various approaches), how innovation works, or what triggers a fad are examples of behaviors for which little valid theory exists--it's not that plenty of opinion about various approaches or models is lacking, but that a theory, in the scientific sense, is nonexistent. For Big Data that means a number of things, first and foremost, that if you don't have a working theory, you probably don't know what data you need to test any hypotheses you may posit. It also means that data scientists can't create a model because no reliable underlying logic exists that can be encoded into a model.
Confirmation Bias: Every model is based on historical assumptions and perceptual biases. Regardless of the sophistication of the science, we often create models that help us see what we want to see, using data selected as a good indicator of such a perception. Take a recent debate about how to price futures. Future events are typically discounted using an exponential model that creates a regular discount rate that eventually leads a value of zero for far-flung events. Exponential discounting takes a deterministic view. A more existential view comes from the proponents of hyperbolic discounting, which creates a preference for rewards that arrive sooner than later. With hyperbolic discounting, discounts of future events fall more gradually, leading to what might be called "irrational behavior." Another version is called the declining discount rate.
This "discount" debate points out that even when a model exists that is designed to aid in decision making about the future, that model may involve contentious disagreements about its validity and alternative approaches that yield very different results. These are important debates in the world of Big Data. One group of modelers advocates for one approach, and another group, an alternative approach, both using sophisticated data and black boxes (as far as the uninitiated business person is concerned) to support their cases. The fact is that in cases like this, no one knows the answer definitively as the application may be contextual or it may be incomplete (e.g., a new approach may solve the issue that none of the current approaches solves completely). What can be said, and what must be remembered is, the adage that "a futurist is never wrong today." Who wins these debates today may be meaningless because the implications have no near-term consequences, but companies that accept one approach over the other may be betting their firm's future on wishful thinking and unwillingness to admit what they don't know.
The World Changes: We must remember that all data is historical. There is no data from or about the future. Future context changes cannot be built into a model because they cannot be anticipated. Consider this: 2012 is the 50th anniversary of the 1962 Seattle World's Fair. In 1962, the retail world was dominated by Sears, Montgomery Ward, Woolworth, A&P, and Kresge.
Some of those companies no longer exist, and others have merged to the point that they are unrecognizable from their 1962 incarnations. Also in 1962, thousands of miles away, a small company opened its first location in Rogers, Ark.--the Wal-Mart Discount City. Would models of retail supply chains built in 1962 be able to anticipate the overwhelming disruption that this humble storefront would cause for retail? Did Sam Walton understand the impact of Amazon.com when it went live in 1995?
The answer to all of the above is "no." These innovations are rare and hugely disruptive. They are far from doomsday scenarios except for firms so entrenched in their models that they can't adapt. And there is the problem for Big Data. As with Li's risk analysis algorithm, when the world changed, the model did not. Feedback loops are important as a way of maintaining relevance through incremental improvement, but what happens when the world changes so much that current assumptions become irrelevant and the clock must be started again. Not only must we remember that all data is historical, but we must also remember that at some point historical data becomes irrelevant when the context changes.
Motives: In a recent BusinessWeek article (Palantir, "The War on Terror's Secret Weapon"), Peter Theil is quoted as saying: "We cannot afford to have another 9/11 event in the U.S. or anything bigger than that. That day opened the doors to all sorts of crazy abuses and draconian policies." Theil, characterized in the article as a libertarian, sees the analysis of data as a positive for civil liberties. That position is debatable as groups like NO2ID.net form to campaign against not only the use of Big Data, but its creation. Given the complexity of the data and associated models, along with various intended of unintended biases, organizations have to go out of their way to discern the motives of those developing analytics models, lest they allow programs to manipulate data in a way that may precipitate negative social, legal, or fiduciary outcomes.
Acting on the Model: Consider crime analysis. George Mohler of Santa Clara University in California has applied equations that predict earthquake aftershocks to crime. By using location and data and times of recent crimes, the system predicts "aftercrimes." This kind of anticipatory data may result in bastions of police flooding a neighborhood following one burglary. With no police presence, the anticipated crimes may well take place. If the burglars, however, see an increase in surveillance and police activity, they may abandon planned targets and seek new ones, thus invalidating the models' predictions, potentially in terms of time and location. The proponents of Big Data need to ensure that the users of their models understand the intricacies of trend analysis, what a trend really is, and the implications of acting on a model's recommendations.
Where Big Data Will Work
Some of the emerging Big Data stories don't test the existential limits of technology, nor do they threaten global catastrophe. The worst outcome in dinosaur fossil hunting is not finding dinosaur bones where you expect them, and the worst outcome of a crime prediction is a burglary that doesn't take place.
Big Data will no doubt be used to target advertising, reduce fraud, fight crime, find tax evaders, collect child support payments, create better health outcomes, and myrid other activities from the mundane to the ridiculous.
And along the way, the software companies and those who invested in Big Data will share their stories.
Companies like monumental con
structor Arup use Big Data as a way to better model the use of the buildings they build. The Arup software arm, Oasys, recently acquired MassMotion to help them understand the flow of people through buildings. MassMotion can model the intimacy of a coffee shop or the flow of hundreds of thousands through a terminal or metro system using its agent technology. His models are informed from data about traffic patterns, arrival and departure times for various forms of transportation and the environment, such as shopping locations and information desks where people might stop or congregate. The result is a model, sometimes with thousands of avatars, pushing and shoving, congregating and separating--all based on MassMotion's Erin Morrow and how he perceives the world.
Another movement oriented application of Big Data, Jyotish (Sanskrit for astrology), comes from Boeing's research cener at the University of Illinois in Urbana-Champaign. This application predicts the movement of work crews within Boeing's factories. It will ultimately help them figure out how to save costs and increase satisfaction by ensuring that services, like Wi-Fi, are available where and when they are needed.
Palantir, the Palo Alto-based startup focused on solving the intelligence problem of 9/11, discovers correlations in the data that informs military and intelligence agencies who, what, and when a potential threat turns into an imminent threat. With access to data models and data across government silos, Palantir may well make its hero case more often than not in individual cases. They have to be cautious about applying their ideas to different domains where underlying rules might not be so clear or data so well-defined.
For some fields, like biology, placing large data sets into open source areas may bring a kind of convergence as collaboration ensues. But as Michael Nielsen points out in Reinventing Discovery, scientists have very little motivation to collaborate given the nature of publication, reputation, and tenure.
I seriously doubt that we have the intellectual infrastructure to support the collaborative capabilities of the Internet. We may well be able to connect all sorts of data and run all kinds of analyzes, but in the end, we may not be equipped to apply the technology in a meaningful and safe way at scales that outstrip our ability to represent, understand, and validate the models and their data.
A Telling Story
In the Scientific American article, Helbing relates an old story. A drunk man is looking for his keys under a street lamp. When asked why, he responds, "that's where the light is." To Helbing, it appears that he wants to use Big Data to create a brighter light so that scientists can peer beneath if for insight.
Scenario planners tell that story too, but we do so to make the point that while one is looking for keys in the light, they aren't paying any attention to the darkness. The future of Big Data lies not in the stories of anecdotal triumph that report sophisticated, but limited accomplishments--no, the future of Big Data rather lies in the darkness of context change, complexity, and overconfidence.
I will end, as Thaler did in his New York Times article, by quoting Mark Twain: "It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so."
Kessler in the WSJ: The Button That Made Facebook Billions
The power of 'Like' as an emotional sensor is driving the company's exorbitant valuation.
Andy Kessler, The Wall Street Journal, February 2, 2012
As bizarre as this sounds, one of the most valuable innovations in technology over the last several decades is Facebook's "Like" button. That's what has propelled the company to a galaxy-orbit valuation for its forthcoming initial public offering, filed Wednesday.
This is not only because the word "like" is, like, the identifying word of an entire generation. It's because computing has evolved beyond just taking directions from humans—and instead is cozying up to us and sniffing out our emotions and intent.
Advertisers are drooling. Procter & Gamble CEO Robert McDonald, while announcing a layoff of 1,600 people, said this week that the company is trimming its $10 billion annual ad budget as well because, he told analysts, "things like Facebook and Google" can be "much more efficient."
Computing's business model has evolved from selling hardware and software to enticing millions of users to one spot and then selling them something else. The industry has moved from the cool business of directly selling innovation to the age-old media model of selling ads—a $500 billion business.
And behind all the glitz, that's what Facebook is. Its tons of cheap servers sitting in dark rooms store our thoughts, photos and videos, to be delivered and shared to a select group of friends, our social network.
Advertisers rightly took note of the potential. But running ads next to pictures of your buddy Johnny funneling beers at a lacrosse game is not exactly what they had in mind. Then, in mid-2010, Facebook rolled out its Like button, which transformed the company from a somewhat interesting social network into a major media player. The power of Like as an emotional sensor is what's driving Facebook's exorbitant valuation.
Note that no one dominates this next wave of computing—not yet anyway. Instead, lots of companies own a piece of the market, and all are trying to eat the others' lunch.
Apple, worth $425 billion with $125 billion in annual sales, is a huge consumer platform. It keeps hardware and software tightly coupled, selling more than 100 million phones or tablets yearly. This lets it sell other stuff, like apps, and content like music and videos—and some ads. With the ability to track our clicks and location, it knows what all of us are doing minute by minute, sort of—but so far it leverages that by selling ever-better devices which we constantly upgrade.
Apple has already tried and failed to take on Facebook with a homegrown social network, Ping. But Apple is a threat to Facebook in many ways, especially since more than half of Facebook users, I'd guess, access the site via mobile phones.
Amazon, worth $80 billion with $48 billion in annual sales, sells us stuff directly. These sales are relatively low-margin, compared to software anyway. But Amazon leverages its infrastructure and 150 million users by selling excess capacity and Web services to smaller players, by selling ads, and to kick-start its own Kindle, which is now a lot more than an e-book reader. The Kindle Fire, with six million sold in the last three months, is a full-fledged shopping platform.
Google, worth $190 billion with $38 billion in annual sales, is the closest real competitor to Facebook. Google lures you to its site via its search engine and sells ads against results, paid per click. Yes, yes, Google has lots of other services, like email and maps, but basically it runs an ad platform. It's a great business with operating profits of 35%, similar to Facebook's.
Google+ is the next-largest social network to Facebook, and Google is scrambling to integrate your friends' opinions into search results. Google's Android phones are another gateway that Facebook and its users must traverse.
There are lots of other potential competitors trying to leverage our emotions. Twitter with its 140-character messaging. Netflix with streaming video. OpenTable with restaurant reservations. Microsoft with Skype video conferencing and Xbox videogames. Groupon and LivingSocial with social coupons. LinkedIn with business resumes and job search. Each provides some piece of productivity that users or businesses or advertisers will pay for. Each company that fails to dominate will end up being acquired or subsumed by the eventual winners.
Which brings us back to Facebook, which listed 2011 earnings of $3.71 billion in its IPO filing and is believed to have a potential worth of $75 billion to $100 billion—though I hear that some shares traded privately before the filing at an assumed valuation of $150 billion. Facebook doesn't sell phones or tablets, or ship physical products or even do searches. Instead, it has a vibrant, pulsating community of 845 million people willing to share their personal lives with others. Facebook is a giant emotional locker.
That's where the Like button comes in. The adage about advertising is that only half of ads are effective, but no one knows which half. So companies will drop $3.5 million to NBC for a 30-second Super Bowl ad. Or run Keystone Light ads on Comedy Central. They may work, but advertising—ask P&G—is an industry ripe for productive innovation. With the Like button, Facebook is like Bob Eubanks on "The Newlywed Game," who promised contestants "a prize chosen especially for you." Advertising's nirvana is an ad chosen especially for you. Of all the players, Facebook is the closest to delivering.
Others will catch up. Facebook CEO Mark Zuckerberg would do well to take the billions raised from the IPO and go on the offensive. Already—via a new subscribe feature with which you can "follow" someone—it is trying to neutralize Twitter. A Facebook phone? I doubt it, but lucrative deals with other hardware players are almost a must.
My best advice to Facebook is to keep playing poker and betting big while Apple, Amazon and Google move around their chess pieces playing defense. With a potential enterprise value at 25 times sales, a lot of things have to go right for Facebook over the next five years.
Mr. Kessler, a former hedge-fund manager, is the author most recently of "Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs" (Portfolio, 2011).
--------------------------------------------------
P Please consider the environment before printing this e-mail.
Andy Kessler, The Wall Street Journal, February 2, 2012
As bizarre as this sounds, one of the most valuable innovations in technology over the last several decades is Facebook's "Like" button. That's what has propelled the company to a galaxy-orbit valuation for its forthcoming initial public offering, filed Wednesday.
This is not only because the word "like" is, like, the identifying word of an entire generation. It's because computing has evolved beyond just taking directions from humans—and instead is cozying up to us and sniffing out our emotions and intent.
Advertisers are drooling. Procter & Gamble CEO Robert McDonald, while announcing a layoff of 1,600 people, said this week that the company is trimming its $10 billion annual ad budget as well because, he told analysts, "things like Facebook and Google" can be "much more efficient."
Computing's business model has evolved from selling hardware and software to enticing millions of users to one spot and then selling them something else. The industry has moved from the cool business of directly selling innovation to the age-old media model of selling ads—a $500 billion business.
And behind all the glitz, that's what Facebook is. Its tons of cheap servers sitting in dark rooms store our thoughts, photos and videos, to be delivered and shared to a select group of friends, our social network.
Advertisers rightly took note of the potential. But running ads next to pictures of your buddy Johnny funneling beers at a lacrosse game is not exactly what they had in mind. Then, in mid-2010, Facebook rolled out its Like button, which transformed the company from a somewhat interesting social network into a major media player. The power of Like as an emotional sensor is what's driving Facebook's exorbitant valuation.
Note that no one dominates this next wave of computing—not yet anyway. Instead, lots of companies own a piece of the market, and all are trying to eat the others' lunch.
Apple, worth $425 billion with $125 billion in annual sales, is a huge consumer platform. It keeps hardware and software tightly coupled, selling more than 100 million phones or tablets yearly. This lets it sell other stuff, like apps, and content like music and videos—and some ads. With the ability to track our clicks and location, it knows what all of us are doing minute by minute, sort of—but so far it leverages that by selling ever-better devices which we constantly upgrade.
Apple has already tried and failed to take on Facebook with a homegrown social network, Ping. But Apple is a threat to Facebook in many ways, especially since more than half of Facebook users, I'd guess, access the site via mobile phones.
Amazon, worth $80 billion with $48 billion in annual sales, sells us stuff directly. These sales are relatively low-margin, compared to software anyway. But Amazon leverages its infrastructure and 150 million users by selling excess capacity and Web services to smaller players, by selling ads, and to kick-start its own Kindle, which is now a lot more than an e-book reader. The Kindle Fire, with six million sold in the last three months, is a full-fledged shopping platform.
Google, worth $190 billion with $38 billion in annual sales, is the closest real competitor to Facebook. Google lures you to its site via its search engine and sells ads against results, paid per click. Yes, yes, Google has lots of other services, like email and maps, but basically it runs an ad platform. It's a great business with operating profits of 35%, similar to Facebook's.
Google+ is the next-largest social network to Facebook, and Google is scrambling to integrate your friends' opinions into search results. Google's Android phones are another gateway that Facebook and its users must traverse.
There are lots of other potential competitors trying to leverage our emotions. Twitter with its 140-character messaging. Netflix with streaming video. OpenTable with restaurant reservations. Microsoft with Skype video conferencing and Xbox videogames. Groupon and LivingSocial with social coupons. LinkedIn with business resumes and job search. Each provides some piece of productivity that users or businesses or advertisers will pay for. Each company that fails to dominate will end up being acquired or subsumed by the eventual winners.
Which brings us back to Facebook, which listed 2011 earnings of $3.71 billion in its IPO filing and is believed to have a potential worth of $75 billion to $100 billion—though I hear that some shares traded privately before the filing at an assumed valuation of $150 billion. Facebook doesn't sell phones or tablets, or ship physical products or even do searches. Instead, it has a vibrant, pulsating community of 845 million people willing to share their personal lives with others. Facebook is a giant emotional locker.
That's where the Like button comes in. The adage about advertising is that only half of ads are effective, but no one knows which half. So companies will drop $3.5 million to NBC for a 30-second Super Bowl ad. Or run Keystone Light ads on Comedy Central. They may work, but advertising—ask P&G—is an industry ripe for productive innovation. With the Like button, Facebook is like Bob Eubanks on "The Newlywed Game," who promised contestants "a prize chosen especially for you." Advertising's nirvana is an ad chosen especially for you. Of all the players, Facebook is the closest to delivering.
Others will catch up. Facebook CEO Mark Zuckerberg would do well to take the billions raised from the IPO and go on the offensive. Already—via a new subscribe feature with which you can "follow" someone—it is trying to neutralize Twitter. A Facebook phone? I doubt it, but lucrative deals with other hardware players are almost a must.
My best advice to Facebook is to keep playing poker and betting big while Apple, Amazon and Google move around their chess pieces playing defense. With a potential enterprise value at 25 times sales, a lot of things have to go right for Facebook over the next five years.
Mr. Kessler, a former hedge-fund manager, is the author most recently of "Eat People: And Other Unapologetic Rules for Game-Changing Entrepreneurs" (Portfolio, 2011).
--------------------------------------------------
P Please consider the environment before printing this e-mail.
Wednesday, February 1, 2012
FT: Big innovation needs 'Big Data'
John Hamm, The Financial Times, February 1, 2012
‘Big data’ is all the rage, and for good reason. Companies need to sort through tons of it to understand what products their customers really want, and what they will buy. After all, their future depends on it. Yet, in spite of vast amounts of market intelligence and virtually unlimited information, how well do companies really know their customer?
In a world where, according to the Association of Product Management and product Marketing, more than 50 per cent of new technology products that enter the market fail, and roughly 75 per cent of consumer packaged goods and retail products fail to earn even $7.5m during their first year, we are clearly not doing a good job putting this data to work.
While marketing and manufacturing organisations have begun to take advantage of big data, the same cannot be said of product and service planning, a function most companies still regard as an art. The fact is that reliably creating high-demand products and services requires companies to balance that art with science. That’s where IT comes in.
For senior IT executives, though, the challenges are two-fold: First, to recognize the value of big data in mining customer needs and desires, and second, to devise a data management strategy that integrates big data into the front end of the innovation pipeline.
The third wave
Market research has evolved from the days of discovering customer needs principally on the basis of periodic surveys, focus groups, behavior observation and ethnographic studies. While still important sources of customer reaction, these techniques do not scale to the requirements of global markets or mass customisation.
Next, some companies began to incorporate collaboration technologies, which let customers directly influence product and service features. IBM and Siemens, for example, make use of crowdsourcing. Open innovation initiatives at Proctor & Gamble, Airbus, and Sara Lee are having a notable influence on product development.
Interestingly, a third wave now requires companies to accumulate data from sources residing at close proximity to the customer. In particular, online and brick-and-mortar retail, insurance, transportation and telecommunication companies, healthcare providers, utilities and banks collect large volumes of customer data.
There is transactional data from web searches, purchases, customer service calls and visits to the bank or doctor’s office; social media data from LinkedIn, Facebook, Twitter and blogs, online customer reviews and sentiment analysis, and data produced by the so-called ‘Internet of Things’ which includes location-based services and data from sensors and monitors embedded in a growing array of products, from automobiles to medical devices to household appliances.
With the evolution of software as a service and open source technologies, capturing and managing this data has become more cost effective for companies than ever before. However, sorting through and getting at the right information, and then putting it to work for product development is the real challenge. This requires a fully integrated approach to managing product life cycles so that each data point can be used to paint a full picture of what the customer really wants and needs, which can then be reflected in the end product.
This third wave will account for the biggest growth in big data. A fact made all the more vivid by the recent prediction by Hans Vestberg, Ericsson’s chief executive, that there will be 50bn internet-connected devices by 2020.
IT’s crucial role
That IT has had a positive impact on productivity and efficiency is undeniable. Now, as the pace of innovation quickens and the pressure to remain competitive intensifies, IT can help company innovators build the right products and services. This time, the task is to collect, synthesise, analyse and integrate big data and then merge those results with other sources of market research. There are three areas where IT can contribute.
The first is building the necessary infrastructure. This includes defining the data and computing architectures for extracting and analysing the data, and working with third-party sources of big data. It also means making the data available to existing innovation management and product lifecycle management systems so that those that are best able to act upon it, have easy access. A critical part of this infrastructure also includes developing a system that allows for end-to-end automation of information to avoid siloing and other classic roadblocks in the development cycle.
The second is identifying operational processes. This includes social media analytics and applications for data visualisation, collaboration and business intelligence. And it includes helping to define new business processes and workflows for product and service development organisations.
Third, is advocating for proper policies and governance. The availability of sensitive data to more organisations carries with it new attendant risks. These risks must be assessed in the context of new usage models and existing security, privacy and data access policies.
Eyes on the prize
The barriers to incorporating big data into the innovation process can be significant, ranging from cultural to organisational, technological and budgetary. However, the opportunities are also clear: quicker to market with new products and services; identifying weak product and service candidates earlier in the development cycle; eliminating features that customers don’t want while adding features they are will to pay a premium for; and identifying and prioritising requirements for specific markets.
The bottom line is that the more you know about your customers, and the more you incorporate that knowledge into products and service, the more positive the impact on revenues, margins and market share.
John Hamm is the CEO of Accept Software and the author of Unusually Excellent .
‘Big data’ is all the rage, and for good reason. Companies need to sort through tons of it to understand what products their customers really want, and what they will buy. After all, their future depends on it. Yet, in spite of vast amounts of market intelligence and virtually unlimited information, how well do companies really know their customer?
In a world where, according to the Association of Product Management and product Marketing, more than 50 per cent of new technology products that enter the market fail, and roughly 75 per cent of consumer packaged goods and retail products fail to earn even $7.5m during their first year, we are clearly not doing a good job putting this data to work.
While marketing and manufacturing organisations have begun to take advantage of big data, the same cannot be said of product and service planning, a function most companies still regard as an art. The fact is that reliably creating high-demand products and services requires companies to balance that art with science. That’s where IT comes in.
For senior IT executives, though, the challenges are two-fold: First, to recognize the value of big data in mining customer needs and desires, and second, to devise a data management strategy that integrates big data into the front end of the innovation pipeline.
The third wave
Market research has evolved from the days of discovering customer needs principally on the basis of periodic surveys, focus groups, behavior observation and ethnographic studies. While still important sources of customer reaction, these techniques do not scale to the requirements of global markets or mass customisation.
Next, some companies began to incorporate collaboration technologies, which let customers directly influence product and service features. IBM and Siemens, for example, make use of crowdsourcing. Open innovation initiatives at Proctor & Gamble, Airbus, and Sara Lee are having a notable influence on product development.
Interestingly, a third wave now requires companies to accumulate data from sources residing at close proximity to the customer. In particular, online and brick-and-mortar retail, insurance, transportation and telecommunication companies, healthcare providers, utilities and banks collect large volumes of customer data.
There is transactional data from web searches, purchases, customer service calls and visits to the bank or doctor’s office; social media data from LinkedIn, Facebook, Twitter and blogs, online customer reviews and sentiment analysis, and data produced by the so-called ‘Internet of Things’ which includes location-based services and data from sensors and monitors embedded in a growing array of products, from automobiles to medical devices to household appliances.
With the evolution of software as a service and open source technologies, capturing and managing this data has become more cost effective for companies than ever before. However, sorting through and getting at the right information, and then putting it to work for product development is the real challenge. This requires a fully integrated approach to managing product life cycles so that each data point can be used to paint a full picture of what the customer really wants and needs, which can then be reflected in the end product.
This third wave will account for the biggest growth in big data. A fact made all the more vivid by the recent prediction by Hans Vestberg, Ericsson’s chief executive, that there will be 50bn internet-connected devices by 2020.
IT’s crucial role
That IT has had a positive impact on productivity and efficiency is undeniable. Now, as the pace of innovation quickens and the pressure to remain competitive intensifies, IT can help company innovators build the right products and services. This time, the task is to collect, synthesise, analyse and integrate big data and then merge those results with other sources of market research. There are three areas where IT can contribute.
The first is building the necessary infrastructure. This includes defining the data and computing architectures for extracting and analysing the data, and working with third-party sources of big data. It also means making the data available to existing innovation management and product lifecycle management systems so that those that are best able to act upon it, have easy access. A critical part of this infrastructure also includes developing a system that allows for end-to-end automation of information to avoid siloing and other classic roadblocks in the development cycle.
The second is identifying operational processes. This includes social media analytics and applications for data visualisation, collaboration and business intelligence. And it includes helping to define new business processes and workflows for product and service development organisations.
Third, is advocating for proper policies and governance. The availability of sensitive data to more organisations carries with it new attendant risks. These risks must be assessed in the context of new usage models and existing security, privacy and data access policies.
Eyes on the prize
The barriers to incorporating big data into the innovation process can be significant, ranging from cultural to organisational, technological and budgetary. However, the opportunities are also clear: quicker to market with new products and services; identifying weak product and service candidates earlier in the development cycle; eliminating features that customers don’t want while adding features they are will to pay a premium for; and identifying and prioritising requirements for specific markets.
The bottom line is that the more you know about your customers, and the more you incorporate that knowledge into products and service, the more positive the impact on revenues, margins and market share.
John Hamm is the CEO of Accept Software and the author of Unusually Excellent .
MIT Sloan course to tackle 'Big Data'
Chris Reidy, The Boston Globe, February 01, 2012,
Companies are now so overwhelmed by information overload that the MIT Sloan School of Management said it is gearing up to teach a two-day course that aims to help executives cope with massive data dumps and extract meaningful insights from real-time digital data.
Set for March 27 and March 28, the course is titled, “Big Data: Making Complex Things Simpler.” In an MIT Sloan press release, professor Erik Brynjolfsson, director of MIT’s Center for Digital Business, commented on the phenomenon of “Big Data.” According to Brynjolfsson, companies adept at using “Big Data” are “around 5 percent more productive and profitable than their competitors.”
Professor Alex ‘Sandy’ Pentland, director of MIT’s Media Lab Entrepreneurship Program, also teaches in the Big Data program.
“Many managers undervalue the worth of data, but really it is like money in your bank account and you should be getting a return on it,” Pentland said in a statement. “This program will show managers how to capture the benefits of data such as creating better customer analytics or capturing real-time consumer preferences.”
Mobile phones are one rich source of Big Data, Pentland noted. On Black Friday, the big shopping day on the Friday after Thanksgiving, projections on sales could be based on mobile phone activity. Even more intriguing perhaps, mobile phone activity has the potential to help saloon-keepers to track the movements of potential patrons.
“We could predict Black Friday sales before certain stores make their own announcements based on how many mobile phones were in their parking lots that day,” Pentland said. “We also can find out which bars are popular at 9:00 p.m. versus 11:00 p.m. with this type of mobile phone data.”
Companies are now so overwhelmed by information overload that the MIT Sloan School of Management said it is gearing up to teach a two-day course that aims to help executives cope with massive data dumps and extract meaningful insights from real-time digital data.
Set for March 27 and March 28, the course is titled, “Big Data: Making Complex Things Simpler.” In an MIT Sloan press release, professor Erik Brynjolfsson, director of MIT’s Center for Digital Business, commented on the phenomenon of “Big Data.” According to Brynjolfsson, companies adept at using “Big Data” are “around 5 percent more productive and profitable than their competitors.”
Professor Alex ‘Sandy’ Pentland, director of MIT’s Media Lab Entrepreneurship Program, also teaches in the Big Data program.
“Many managers undervalue the worth of data, but really it is like money in your bank account and you should be getting a return on it,” Pentland said in a statement. “This program will show managers how to capture the benefits of data such as creating better customer analytics or capturing real-time consumer preferences.”
Mobile phones are one rich source of Big Data, Pentland noted. On Black Friday, the big shopping day on the Friday after Thanksgiving, projections on sales could be based on mobile phone activity. Even more intriguing perhaps, mobile phone activity has the potential to help saloon-keepers to track the movements of potential patrons.
“We could predict Black Friday sales before certain stores make their own announcements based on how many mobile phones were in their parking lots that day,” Pentland said. “We also can find out which bars are popular at 9:00 p.m. versus 11:00 p.m. with this type of mobile phone data.”
Monday, January 30, 2012
10 ways big data is remaking energy
Katie Fehrenbacher, Gigaom, Jan. 29, 2012, 9:30pm
One of the most obvious trends from the big smart grid conference DistribuTECH last week was how much analytics and big data tools will be used to try to remake energy in 2012, from curbing energy consumption, to reducing energy loss, to adding in more clean power to the grid. Here’s 10 ways that analytics and big data will start to shape the production and consumption of energy in the world:
1). Weather data: Having a finger on the pulse of constantly changing weather data on a micro and macro level can help utilities, building owners and consumers optimize their energy consumption habits and promote energy efficiency. Startup EnergyHub recently partnered with sensor network player Earth Networks to use weather data to make a more efficient form of demand response (utilities controlling power consumption). Other startups like EcoFactor, Opower and Tendril also use weather data as part of their energy behavioral analytics.
IBM has long sold a weather prediction service called Deep Thunder to municipalities, organizations and utilities, which use it to do things like tailor their services, change routes, or generate more or less power. I think weather data could some day provide a platform for some very important next generation services and applications for energy efficiency, much in the way that location data is used as a platform for a variety of services.
2). Cell phone data: Cell phones in our pockets are essentially palm-sized sensors and computers sending a constant stream of information to the cloud where companies could one day use that data to create energy efficiency and better energy products. And yes, a lot of that data is private information, but after that data is anonymized it can be used for the greater good of the community — particularly via the billions of cell phones in developing countries. A startup called Jana does research projects around cell phone data in developing countries, and looks to work with NGOs on programs to create better infrastructure, energy infrastructure and resources.
3). Connected thermostat data: One of the biggest trends from DistribuTECH this year was the overwhelming amount of smart thermostats that are now being sold and marketed. Companies can incorporate that thermostat data into data bases that can be used to promote energy efficiency. EcoFactor’s service remembers every time a home owner overrides the automated smart thermostat system and changes the personalized service to accommodate that manual override. Using 100,000 connected thermostats (which produce 5 billion data points each month) EnergyHub found some interesting statistics like folks in cold climates have a lower average heating temperature set point than households in warmer states.
4). Hadoop & energy databases: The open source data base tool Hadoop is well known — and oft used — in the computing worlds. But in the energy and utility worlds it’s quite rare. However, as the amount of energy data has started to rapidly grow from the smart grid, some companies are embracing Hadoop as a key way to manage energy info. Opower tells me it’s using Hadoop (and the company commercializing Hadoop, Cloudera) as an important way to manage its massive energy data streams. Likewise PJM has turned to Hadoop as a way to organize the energy data coming off of a synchophaser sensor project.
5). Clean power data: One of the main goals for the smart grid is to enable the addition of more variable clean power, which is far more unreliable than fossil fuels (the sun doesn’t shine and the wind doesn’t blow 24/7). Analytics crunching the data from a utilities’ energy supply and demand can help make clean power a little less variable, by being able to more accurately predict the environmental conditions, as well as more accurately assess demand from energy users.
6). Electric car data: Electric cars will by their nature be connected cars, using information technology to manage the vehicle charge and location. Utilities will be closely tracking the charging habits of electric car owners in order to make sure that the grid isn’t overloaded in some early adopter neighborhoods.
7). Power line sensors: One of the areas of low hanging fruit for the power grid is the simple task of helping utilities find blackouts more easily and be able to monitor and manage grid outages. That’s partly where sensor systems called synchophasers come in, which can in real time monitor the health of power lines, collecting multiple data streams per second. Expect all major networks to have synchophaser systems installed over the coming years.
8). Real estate data: Startups like First Fuel Software can use big data to make super accurate assessments about buildings and ways to reduce the energy consumption of buildings — without any extra hardware or monitoring software being installed at the building. Things like weather around the building, demographics of the people in the building, and the building’s historical energy consumption can be used to create an accurate projection. The best way to make a building more energy efficient is by getting as much data about the building;s energy use as possible.
9). Variable pricing: Some day when electricity is sold throughout the world at different prices dependent on supply and demand, massive data bases will be needed. This type of variable pricing is offered in some places in the world, but if it ever becomes ubiquitous it will help curb consumption, by offering high prices when energy is being over used.
10). Using behavioral analytics to curb energy consumption: Getting into the brains of energy users is the job of startups like Opower and Tendril (after it acquired Gr0unded Power.) Essentially these companies have collected data on consumers and demographics and they are using it to try to guess the best way to influence the consumer to do things like upgrade their home appliances and lights to more efficient ones.
One of the most obvious trends from the big smart grid conference DistribuTECH last week was how much analytics and big data tools will be used to try to remake energy in 2012, from curbing energy consumption, to reducing energy loss, to adding in more clean power to the grid. Here’s 10 ways that analytics and big data will start to shape the production and consumption of energy in the world:
1). Weather data: Having a finger on the pulse of constantly changing weather data on a micro and macro level can help utilities, building owners and consumers optimize their energy consumption habits and promote energy efficiency. Startup EnergyHub recently partnered with sensor network player Earth Networks to use weather data to make a more efficient form of demand response (utilities controlling power consumption). Other startups like EcoFactor, Opower and Tendril also use weather data as part of their energy behavioral analytics.
IBM has long sold a weather prediction service called Deep Thunder to municipalities, organizations and utilities, which use it to do things like tailor their services, change routes, or generate more or less power. I think weather data could some day provide a platform for some very important next generation services and applications for energy efficiency, much in the way that location data is used as a platform for a variety of services.
2). Cell phone data: Cell phones in our pockets are essentially palm-sized sensors and computers sending a constant stream of information to the cloud where companies could one day use that data to create energy efficiency and better energy products. And yes, a lot of that data is private information, but after that data is anonymized it can be used for the greater good of the community — particularly via the billions of cell phones in developing countries. A startup called Jana does research projects around cell phone data in developing countries, and looks to work with NGOs on programs to create better infrastructure, energy infrastructure and resources.
3). Connected thermostat data: One of the biggest trends from DistribuTECH this year was the overwhelming amount of smart thermostats that are now being sold and marketed. Companies can incorporate that thermostat data into data bases that can be used to promote energy efficiency. EcoFactor’s service remembers every time a home owner overrides the automated smart thermostat system and changes the personalized service to accommodate that manual override. Using 100,000 connected thermostats (which produce 5 billion data points each month) EnergyHub found some interesting statistics like folks in cold climates have a lower average heating temperature set point than households in warmer states.
4). Hadoop & energy databases: The open source data base tool Hadoop is well known — and oft used — in the computing worlds. But in the energy and utility worlds it’s quite rare. However, as the amount of energy data has started to rapidly grow from the smart grid, some companies are embracing Hadoop as a key way to manage energy info. Opower tells me it’s using Hadoop (and the company commercializing Hadoop, Cloudera) as an important way to manage its massive energy data streams. Likewise PJM has turned to Hadoop as a way to organize the energy data coming off of a synchophaser sensor project.
5). Clean power data: One of the main goals for the smart grid is to enable the addition of more variable clean power, which is far more unreliable than fossil fuels (the sun doesn’t shine and the wind doesn’t blow 24/7). Analytics crunching the data from a utilities’ energy supply and demand can help make clean power a little less variable, by being able to more accurately predict the environmental conditions, as well as more accurately assess demand from energy users.
6). Electric car data: Electric cars will by their nature be connected cars, using information technology to manage the vehicle charge and location. Utilities will be closely tracking the charging habits of electric car owners in order to make sure that the grid isn’t overloaded in some early adopter neighborhoods.
7). Power line sensors: One of the areas of low hanging fruit for the power grid is the simple task of helping utilities find blackouts more easily and be able to monitor and manage grid outages. That’s partly where sensor systems called synchophasers come in, which can in real time monitor the health of power lines, collecting multiple data streams per second. Expect all major networks to have synchophaser systems installed over the coming years.
8). Real estate data: Startups like First Fuel Software can use big data to make super accurate assessments about buildings and ways to reduce the energy consumption of buildings — without any extra hardware or monitoring software being installed at the building. Things like weather around the building, demographics of the people in the building, and the building’s historical energy consumption can be used to create an accurate projection. The best way to make a building more energy efficient is by getting as much data about the building;s energy use as possible.
9). Variable pricing: Some day when electricity is sold throughout the world at different prices dependent on supply and demand, massive data bases will be needed. This type of variable pricing is offered in some places in the world, but if it ever becomes ubiquitous it will help curb consumption, by offering high prices when energy is being over used.
10). Using behavioral analytics to curb energy consumption: Getting into the brains of energy users is the job of startups like Opower and Tendril (after it acquired Gr0unded Power.) Essentially these companies have collected data on consumers and demographics and they are using it to try to guess the best way to influence the consumer to do things like upgrade their home appliances and lights to more efficient ones.
WSJ Opinion: The Coming Tech-led Boom
Three breakthroughs are poised to transform this century as much as telephony and electricity did the last.
Mark P. Mills & Julio M. Ottino, The Wall Street Journal, January 30, 2012
In January 1912, the United States emerged from a two-year recession. Nineteen more followed—along with a century of phenomenal economic growth. Americans in real terms are 700% wealthier today.
In hindsight it seems obvious that emerging technologies circa 1912—electrification, telephony, the dawn of the automobile age, the invention of stainless steel and the radio amplifier—would foster such growth. Yet even knowledgeable contemporary observers failed to grasp their transformational power.
In January 2012, we sit again on the cusp of three grand technological transformations with the potential to rival that of the past century. All find their epicenters in America: big data, smart manufacturing and the wireless revolution.
Information technology has entered a big-data era. Processing power and data storage are virtually free. A hand-held device, the iPhone, has computing power that shames the 1970s-era IBM mainframe. The Internet is evolving into the "cloud"—a network of thousands of data centers any one of which makes a 1990 supercomputer look antediluvian. From social media to medical revolutions anchored in metadata analyses, wherein astronomical feats of data crunching enable heretofore unimaginable services and businesses, we are on the cusp of unimaginable new markets.
The second transformation? Smart manufacturing. This is the first structural shift since Henry Ford launched the economic power of "mass production." While we see evidence already in automation and information systems applied to supply-chain management, we are just entering an era where the very fabrication of physical things is revolutionized by emerging materials science. Engineers will soon design and build from the molecular level, optimizing features and even creating new materials, radically improving quality and reducing waste.
Devices and products are already appearing based on computationally engineered materials that literally did not exist a few years ago: novel metal alloys, graphene instead of silicon transistors (graphene and carbon enable a radically new class of electronic and structural materials), and meta-materials that possess properties not possible in nature; e.g., rendering an object invisible—speculation about which received understandable recent publicity.
This era of new materials will be economically explosive when combined with 3-D printing, also known as direct-digital manufacturing—literally "printing" parts and devices using computational power, lasers and basic powdered metals and plastics.
Already emerging are printed parts for high-value applications like patient-specific implants for hip joints or teeth, or lighter and stronger aircraft parts. Then one day, the Holy Grail: "desktop" printing of entire final products from wheels to even washing machines.
The era of near-perfect computational design and production will unleash as big a change in how we make things as the agricultural revolution did in how we grew things. And it will be defined by high talent not cheap labor.
Finally, there is the unfolding communications revolution where soon most humans on the planet will be connected wirelessly. Never before have a billion people—soon billions more—been able to communicate, socialize and trade in real time.
The implications of the radical collapse in the cost of wireless connectivity are as big as those following the dawn of telegraphy/telephony. Coupled with the cloud, the wireless world provides cheap connectivity, information and processing power to nearly everyone, everywhere. This introduces both rapid change—e.g., the Arab Spring—and great opportunity. Again, both the launch and epicenter of this technology reside in America.
Few deny that technology fuels economic growth as well as both social and lifestyle progress, the latter largely seen in health and environmental metrics. But consider three features that most define America, and that are essential for unleashing the promises of technological change: our youthful demographics, dynamic culture and diverse educational system.
First, demographics. By 2020, America will be younger than both China and the euro zone, if the latter still exists. Youth brings more than a base of workers and taxpayers; it brings the ineluctable energy that propels everything. Amplified and leavened by the experience of their elders, youth and economic scale (the U.S. is still the world's largest economy) are not to be underestimated, especially in the context of the other two great forces: our culture and educational system.
The American culture is particularly suited to times of tumult and challenge. Culture cannot be changed or copied overnight; it is a feature of a people that has, to use a physics term, high inertia. Ours is distinguished by incontrovertibly powerful features, namely open-mindedness, risk-taking, hard work, playfulness, and, critical for nascent new ideas, a healthy dose of anti-establishment thinking. Where else could an Apple or a Steve Jobs have emerged?
Then there's our educational system, often criticized as inadequate to global challenges. But American higher education eludes simple statistical measures since its most salient features are flexibility and diversity of educational philosophies, curricula and the professoriate. There is a dizzying range of approaches in American universities and colleges. Good. One size definitely does not fit all for students or the future.
We should also remember that more than half of the world's top 100 universities remain in America, a fact underscored by soaring foreign enrollments. Yes, other nations have fine universities, and many more will emerge over time. But again the epicenter remains here.
What should our politicians do to help usher in this new era of entrepreneurial growth? Liquid financial markets, sensible tax and immigration policy, and balanced regulations will allow the next boom to flourish. But the essential fuel is innovation. The promise resides in the tectonic technological shifts under way.
America's success isn't preordained. But the technological innovations circa 2012 are profound. They will engender sweeping changes to our society and our economy. All the forces are in place. It's just a matter of when.
Mr. Mills, a physicist and founder of the Digital Power Group, writes the Forbes Energy Intelligence column. Mr. Ottino is dean of the McCormick School of Engineering and Applied Sciences at Northwestern University.
Mark P. Mills & Julio M. Ottino, The Wall Street Journal, January 30, 2012
In January 1912, the United States emerged from a two-year recession. Nineteen more followed—along with a century of phenomenal economic growth. Americans in real terms are 700% wealthier today.
In hindsight it seems obvious that emerging technologies circa 1912—electrification, telephony, the dawn of the automobile age, the invention of stainless steel and the radio amplifier—would foster such growth. Yet even knowledgeable contemporary observers failed to grasp their transformational power.
In January 2012, we sit again on the cusp of three grand technological transformations with the potential to rival that of the past century. All find their epicenters in America: big data, smart manufacturing and the wireless revolution.
Information technology has entered a big-data era. Processing power and data storage are virtually free. A hand-held device, the iPhone, has computing power that shames the 1970s-era IBM mainframe. The Internet is evolving into the "cloud"—a network of thousands of data centers any one of which makes a 1990 supercomputer look antediluvian. From social media to medical revolutions anchored in metadata analyses, wherein astronomical feats of data crunching enable heretofore unimaginable services and businesses, we are on the cusp of unimaginable new markets.
The second transformation? Smart manufacturing. This is the first structural shift since Henry Ford launched the economic power of "mass production." While we see evidence already in automation and information systems applied to supply-chain management, we are just entering an era where the very fabrication of physical things is revolutionized by emerging materials science. Engineers will soon design and build from the molecular level, optimizing features and even creating new materials, radically improving quality and reducing waste.
Devices and products are already appearing based on computationally engineered materials that literally did not exist a few years ago: novel metal alloys, graphene instead of silicon transistors (graphene and carbon enable a radically new class of electronic and structural materials), and meta-materials that possess properties not possible in nature; e.g., rendering an object invisible—speculation about which received understandable recent publicity.
This era of new materials will be economically explosive when combined with 3-D printing, also known as direct-digital manufacturing—literally "printing" parts and devices using computational power, lasers and basic powdered metals and plastics.
Already emerging are printed parts for high-value applications like patient-specific implants for hip joints or teeth, or lighter and stronger aircraft parts. Then one day, the Holy Grail: "desktop" printing of entire final products from wheels to even washing machines.
The era of near-perfect computational design and production will unleash as big a change in how we make things as the agricultural revolution did in how we grew things. And it will be defined by high talent not cheap labor.
Finally, there is the unfolding communications revolution where soon most humans on the planet will be connected wirelessly. Never before have a billion people—soon billions more—been able to communicate, socialize and trade in real time.
The implications of the radical collapse in the cost of wireless connectivity are as big as those following the dawn of telegraphy/telephony. Coupled with the cloud, the wireless world provides cheap connectivity, information and processing power to nearly everyone, everywhere. This introduces both rapid change—e.g., the Arab Spring—and great opportunity. Again, both the launch and epicenter of this technology reside in America.
Few deny that technology fuels economic growth as well as both social and lifestyle progress, the latter largely seen in health and environmental metrics. But consider three features that most define America, and that are essential for unleashing the promises of technological change: our youthful demographics, dynamic culture and diverse educational system.
First, demographics. By 2020, America will be younger than both China and the euro zone, if the latter still exists. Youth brings more than a base of workers and taxpayers; it brings the ineluctable energy that propels everything. Amplified and leavened by the experience of their elders, youth and economic scale (the U.S. is still the world's largest economy) are not to be underestimated, especially in the context of the other two great forces: our culture and educational system.
The American culture is particularly suited to times of tumult and challenge. Culture cannot be changed or copied overnight; it is a feature of a people that has, to use a physics term, high inertia. Ours is distinguished by incontrovertibly powerful features, namely open-mindedness, risk-taking, hard work, playfulness, and, critical for nascent new ideas, a healthy dose of anti-establishment thinking. Where else could an Apple or a Steve Jobs have emerged?
Then there's our educational system, often criticized as inadequate to global challenges. But American higher education eludes simple statistical measures since its most salient features are flexibility and diversity of educational philosophies, curricula and the professoriate. There is a dizzying range of approaches in American universities and colleges. Good. One size definitely does not fit all for students or the future.
We should also remember that more than half of the world's top 100 universities remain in America, a fact underscored by soaring foreign enrollments. Yes, other nations have fine universities, and many more will emerge over time. But again the epicenter remains here.
What should our politicians do to help usher in this new era of entrepreneurial growth? Liquid financial markets, sensible tax and immigration policy, and balanced regulations will allow the next boom to flourish. But the essential fuel is innovation. The promise resides in the tectonic technological shifts under way.
America's success isn't preordained. But the technological innovations circa 2012 are profound. They will engender sweeping changes to our society and our economy. All the forces are in place. It's just a matter of when.
Mr. Mills, a physicist and founder of the Digital Power Group, writes the Forbes Energy Intelligence column. Mr. Ottino is dean of the McCormick School of Engineering and Applied Sciences at Northwestern University.
Thursday, January 26, 2012
Big Data at Davos
At Davos, Discussions of a Global Data Deluge
Nick Bilton, The New York Times, January 25, 2012
DAVOS, Switzerland — Each year, a new theme reverberates through technology conferences around the world. A few years ago it was predictions of the coming wave of social media, location-based services and mobile, long before they became mainstream.
Today’s topic de jour: data. Lots of data.
What makes the data discussion different than in previous years is that it is being discussed in high-profile nontechnology meetings too. This is clearly evident at this year’s annual World Economic Forum.
Meetings here this week include: “From data to decisions: How are new approaches to data intelligence transforming decision-making?” “Data deluge and citizen science.”
“Incidents from digital crime to massive incidents of data theft are increasing significantly, with major political, social and economic implications.” “How is big data being used to uncover individual and collective human dynamics?”
The discussions are not confined to technology attendees either. Chancellors, bankers and educators meeting at the conference are being asked to discuss what the forum calls a growing data deluge and how to manage it.
A 2012 report released by the World Economic Forum, titled, “Big Data, Big Impact: New Possibilities for International Development,” outlines some of the possibilities data can bring around the globe to business and education. It also warns of its potential privacy implications.
The report says data is a new economic asset class, which touches all aspects of society, regardless of income or location.
“Big data represents one of these seismic economic shifts that happens every 10 years,” said Zach Bogue, co-founder of a stealth data investment fund called Data Collective, who was attending Davos. “In some sense, this data has always existed, but until now the bandwidth, storage capability and compute power haven’t existed to harness it.”
Yet as there is talk of data, the discussion of privacy is not far behind.
Earlier this week in Munich, Viviane Reding, the European justice commissioner, repeatedly talked about data in respect to privacy. Ms. Reding said there were 27 laws that apply to data in Europe, most of which date back more than a decade and don’t properly protect consumers today.
Ms. Reding outlined new regulations that were presented in Brussels on Wednesday and were designed to implement one sweeping data protection regulation that would apply to all of Europe.
The new regulations are part of the discussion at Davos as these new rules would drastically affect the way companies operate and collect data. For example, one component of this legislation will require companies to communicate to users why they are collecting this data and how long it is being stored on company servers.
Although the proposed data law being presented by Ms. Reding will apply to companies in Europe, it will clearly affect technology companies around the globe too, including Facebook, Twitter and Google in the United States.
At the World Economic Forum, the discussion is also focusing on one of the toughest regulation challenges in regards to data collection: How to manage a balancing act between governments overseeing data collection and its actions stifling innovation. More importantly, regulators hope to figure out a global solution to data collection.
As the World Economic Forum reports says: ”Concerted action is needed by governments, development organizations and companies to ensure that this data helps the individuals and communities who create it.”
Nick Bilton, The New York Times, January 25, 2012
DAVOS, Switzerland — Each year, a new theme reverberates through technology conferences around the world. A few years ago it was predictions of the coming wave of social media, location-based services and mobile, long before they became mainstream.
Today’s topic de jour: data. Lots of data.
What makes the data discussion different than in previous years is that it is being discussed in high-profile nontechnology meetings too. This is clearly evident at this year’s annual World Economic Forum.
Meetings here this week include: “From data to decisions: How are new approaches to data intelligence transforming decision-making?” “Data deluge and citizen science.”
“Incidents from digital crime to massive incidents of data theft are increasing significantly, with major political, social and economic implications.” “How is big data being used to uncover individual and collective human dynamics?”
The discussions are not confined to technology attendees either. Chancellors, bankers and educators meeting at the conference are being asked to discuss what the forum calls a growing data deluge and how to manage it.
A 2012 report released by the World Economic Forum, titled, “Big Data, Big Impact: New Possibilities for International Development,” outlines some of the possibilities data can bring around the globe to business and education. It also warns of its potential privacy implications.
The report says data is a new economic asset class, which touches all aspects of society, regardless of income or location.
“Big data represents one of these seismic economic shifts that happens every 10 years,” said Zach Bogue, co-founder of a stealth data investment fund called Data Collective, who was attending Davos. “In some sense, this data has always existed, but until now the bandwidth, storage capability and compute power haven’t existed to harness it.”
Yet as there is talk of data, the discussion of privacy is not far behind.
Earlier this week in Munich, Viviane Reding, the European justice commissioner, repeatedly talked about data in respect to privacy. Ms. Reding said there were 27 laws that apply to data in Europe, most of which date back more than a decade and don’t properly protect consumers today.
Ms. Reding outlined new regulations that were presented in Brussels on Wednesday and were designed to implement one sweeping data protection regulation that would apply to all of Europe.
The new regulations are part of the discussion at Davos as these new rules would drastically affect the way companies operate and collect data. For example, one component of this legislation will require companies to communicate to users why they are collecting this data and how long it is being stored on company servers.
Although the proposed data law being presented by Ms. Reding will apply to companies in Europe, it will clearly affect technology companies around the globe too, including Facebook, Twitter and Google in the United States.
At the World Economic Forum, the discussion is also focusing on one of the toughest regulation challenges in regards to data collection: How to manage a balancing act between governments overseeing data collection and its actions stifling innovation. More importantly, regulators hope to figure out a global solution to data collection.
As the World Economic Forum reports says: ”Concerted action is needed by governments, development organizations and companies to ensure that this data helps the individuals and communities who create it.”
Why 3-D Printing Will Go the Way of Virtual Reality
Extruding, printing, and sintering are not the same as manufacturing.
Christopher Mims, Technology Review, January 25, 2012
There is a species of magical thinking practiced by geeks whose experience is computers and electronics—realms of infinite possibility that are purposely constrained from the messiness of the physical world—that is typical of Singularitarianism, mid-90s missives about the promise of virtual reality, and now, 3-D printing.
As 3-D printers come within reach of the hobbyist—$1,100 for MakerBot's Thing-O-Matic—and The Pirate Bay declares "physibles" the next frontier of piracy, I'm seeing usually level-headed thinkers like Clive Thompson and Tim Maly declare that the end of shipping is here and we should all start boning up on Cory Doctorow's science fiction fantasies of a world in which any object can be rapidly synthesized with a little bit of energy and raw materials.
This isn't just premature, it's absurd. 3-D printing, like VR before it, is one of those technologies that suggest a trend of long and steep adoption driven by rapid advances on the systems we have now. And granted, some of what's going on at present is pretty cool—whether it's in rapid prototyping, solid-fuel rockets, bio-assembly or just giant plastic showpieces.
But the notion that 3-D printing will on any reasonable time scale become a "mature" technology that can reproduce all the goods on which we rely is to engage in a complete denial of the complexities of modern manufacturing, and, more to the point, the challenges of working with matter.
Let's start with the mechanism. Most 3-D printers lay down thin layers of extruded plastic. That's great for creating cheap plastic toys with a limited spatial resolution. But printing your Mii or customizing an iPhone case isn't the same thing as firing ceramics in a kiln or smelting metal or mixing lime with sand at high temperatures to produce glass—unless you'd like everything that's currently made from those substances to be replaced with plastic, and there are countless environmental, health, and durability reasons you don't.
Advocates of 3-D printing also neglect entirely the fact that so much of what we use continues to be made out of natural substances, and for good reason. By any number of measures, wood is pound-for-pound stronger than steel, and the move toward natural products for packaging suggests that the strength and affordability of paper, bamboo and even mushrooms mean that in the future there will be more and not less of all of these.
The desire for 3-D printing to take over from traditional manufacturing needs to be recognized for what it is: an ideology. Getting all of our goods from a box in the corner of our home has attractive implications, from mass customization to "the end of consumerism." With stakes like those, who wouldn't want to be a true believer?
Hype is inevitably followed by some level of backlash, or at least disinterest, and it would be a shame for 3-D printing to head into a too-deep trough of the Gartner hype cycle. There will be plenty of interesting applications for 3-D printing, but I'll bet the ones that will have the biggest impact will be within traditional factories, where rapid prototyping is already having a huge impact.
Khosla: Do We Need Doctors or Algorithms?
Vinod Khosla, TechCrunch, January, 2012
Editor’s note: This is Part II of a guest series written by legendary Silicon Valley investor Vinod Khosla, the founder of Khosla Ventures. In Part I, he laid the groundwork by describing how artificial intelligence is a combination of human and computer capabilities. In Part III, he will talk about how technology will sweep through education.
Later, I got to cogitating about what I had said and why, and how embarrassingly wrong that might be. But the more I think about it the more I feel my gut reaction was probably right. The beginnings of “Doctor Algorithm” or Dr. A for short, most likely (and that does not mean “certainly” or “maybe”) will be much criticized. We’ll see all sorts of press wisdom decrying “they don’t work” or “look at all the silly things they come up with.” But Dr A. will get better and better and will go from providing “bionic assistance” to second opinions to assisting doctors to providing first opinions and as referral computers (with complete and accurate synopses and all possible hypotheses of the hardest cases) to the best 20% of the human breed doctors. And who knows what will happen beyond that?
Assessing Current Healthcare
Let’s start with healthcare (or sickcare, as many knowledgeable people call it). Think about what happens when you visit a doctor. You have to physically go to the hospital or some office, where you wait (with no real predictability for how long), and then the nurse probably takes you in and checks your vitals. Only after all this does the doctor show up and, after some friendly banter, asks you to describe your own symptoms.
The doctor assesses them and hunts around (probably in your throat or lungs) for clues as to their source, provides the diagnosis, writes a prescription, and sends you off.
The entire encounter should take no more than 15 minutes and usually takes probably less than that. Sometimes a test or two may be ordered, if you can afford it. And, as we all know, most of the time, it turns out to be some routine diagnosis with a standard treatment . . . something a computer algorithm could do if the treatment involved no harm, or at least do as well as the median doctor (I am not talking about the top 20% of doctors here—80% of doctors are below the “top 20%” but that is hard for people to intuit!).
So what’s wrong with this situation? This is by no means an exhaustive list, but it sets up a nice springboard:
Physically having to go to your doctor’s office makes sense for the most part, except that a lot of the basic tests are either visual (tongue and throat check) or auditory (listening to the breath and vibrations in the abdomen). Time plus cost will often discourage people from taking that first step to visit a doctor. Most of the time a Dr. A could at least advise you when it is worth visiting based on your normal body functions, your current indications, and your locality’s current infections and other symptom trends.
- A lot of the vitals being tested for (e.g. blood pressure, pulse) can now be routinely done at home or even with the help of an iPhone and an explosion of additional possibilities will emerge in the next decade.
- You are the one telling the doctor your symptoms.
- The doctor has to inquire (probably every time) into any possible history of each symptom, test results, and illnesses, except when he does not have time for you in that village in India.
- The prescriptions are still done on paper, requiring you to, again, physically go to a pharmacy and pick up what you need there. So compliance is an issue.
Going down the list, we find a pretty negative assessment. The vital signs could all be determined with the help of mobile devices, the operation of which do not require years of training and a certification. You will be able to do this by yourself—Philips already is using the iPhone camera to try to measure vital indicators, others will be even more innovative and as an insurance company it would be cost-effective to give them to every insured person for free. Skin Scan is measuring your risk of skin cancer from a photograph of a skin lesion. Telemedicine is accelerating and a Qualcomm company is measuring heart rates using an iPhone. Cell phones that display your vital signs and take ultrasound images of your heart or abdomen are in the offing as well as genetic scans of malignant cells that match your cancer to the most effective treatment. Ear infection and skin rash pictures and more will all be mobile phone based, often supplemented by the kind of (fractal) analysis that Skin Scan does, and more than what the doctors naked eye could usually see.
The history of symptoms, illnesses, and test results could be accessed, processed, and assessed by a computer to see any correlation or trends with the patient’s past. You are the one providing the doctor with the symptoms anyway after all!
Any follow-up hunts for clues could again be done with mobile devices. The prescriptions—along with the medical records—could relocate to electronic and digital methods, saving paper, reducing bureaucracy, and easing the healing process. If 90% of the time the doctor knows exactly the right kind of diagnosis from these very few and superficial inputs (we haven’t even considered genetics yet!), does it really require 10+ years of intense education for every diagnostician?
The fault is not entirely with the doctors, though. Most of us don’t know what set of symptoms warrant the full-scale attention of medical personnel, so we either go all the time or we do not go at all (save for emergencies). We also cannot realistically expect any (even our family) doctor to remember every single symptom and test result over the years, definitely not in a government hospital in China. Similarly, we cannot expect our doctor to be able to remember everything from medical school twenty years ago or memorize the whole Physicians Desk Reference (PDR) and to know everything from the latest research, and so on and so forth. This is why, every time I visit the doctor, I like to get a second opinion. I do my Internet research and feel much better.
Identifying Emerging Trends In Healthcare
But I always wonder why I cannot input my specific test numbers and have a system offer me a “second opinion” on the diagnosis since it has all the data that the doctor has and can use all my current and historical data effectively. In fact, it is not hard to imagine it having more data than the doctor has since my full patient record would be at the tip of its digital brain, unlike the average doctor who probably doesn’t remember my blood glucose levels or my ferritin from two years ago. He does not remember all the complex correlations from med school in which ferritin matters—there are three thousand or more metabolic pathways, I was once told, in the human body and they impact each other in very complex ways. These tasks are perfect for a computer to model as “systems biology” researchers are trying to do.
Add to it my baseline numbers from when I was not sick, which most doctors don’t have and if they did 80% of physicians would be too lazy to use or not know how to use. Applied Proteomics can extract tens of gigabytes of proteomics—what my genes are actually doing instead of what they can do—baseline data from one drop of blood.
Oh, by the way I have my 23andMe data to add my genetic propensities (howsoever imprecise today, but improving rapidly with time and more data). The doctor uses a lot of imprecise judgments too as most good doctors will readily admit. My very good doctor did not check that I have relative insensitivity, genetically, to Metformin, a diabetes drug. It is easy to input the PDR (the Physicians Desk Reference), the massively thick, small-font book that all physicians are supposed to know backwards and forwards. They often don’t remember everything they read, in med school but it is a piece of cake for computers. The book on your typical doctor’s desk is probably not current on the leading-edge science either. Confirmed science and emerging science are different things and each has a role. Doctors mostly use confirmed science, the average doctor not understanding and pros and cons of each or the expected value of a treatment (benefit and harm). And our 18th century tradition of “first do no harm” dictates that if a treatment hurts ten patients a year but saves a thousand lives we reject it.
With enough examples, today’s techniques for language translation (or newer techniques) can translate from human lingo for symptoms (“I feel itchy” or “buzzy” or “reddish bubbly rash with pimples” or “less energy in the morning” or “sort of a stretch in my tendon” and the myriad of imprecise ways symptoms are described and results interpreted — these are highly amenable to big data analysis) into medical lingo matching the PDR. With easy input of real medical results into a computer and long-standing historical data per patient and per population, which a human cannot possibly handle, and patient and population genetics, I suspect getting a second opinion of my diagnosis from Dr. A is a reasonable expectation, and it should certainly be better than a middling physician’s (especially in less developed countries like India, where there is a dire shortage of trained physicians).
I may still need a surgeon (though robotic surgeons like those from Intuitive Surgical are on the way too) or other specialists for some tasks for a little while and the software may move from “second opinion” (in three years? Or seven?) to “bionic software” for the physicians (in five or ten years, with enough patient data?). Bionic software, again, defined here as software which augments and amplifies human understanding.
But I doubt very much if within 10-15 years (given continued investment and innovation and keeping the AMA from quashing such efforts politically) I won’t be able to ask Siri’s great great grandchild (Version 9.0?) for an opinion far more accurate than the one I get today from the average physician. Instead of asking Siri 9.0, “I feel like sushi” or “where can I dispose a body” (try it…it’s fairly accurate!) and with your iPhone X or Android Y with all the power of IBM’s current Watson computer in the mobile phone and an even more powerful “Nvidia times 10-100” server which will cost far less than med school with terabytes or petabytes of data on hundreds of millions (billions?) of patients, including their complete genomics and proteomics (each sample costing about the same as a typical blood test).
IBM’s Watson computer, I understand, is now being applied to medical diagnosis after handling imprecise and vague tasks like winning at Jeopardy, which experts a few years ago would have said could not be done. “Computers cannot match the judgment of humans on these kinds of tasks!” And with enough data, medical diagnosis or 90% of it is an easier task than Jeopardy.
Already Kaiser Permanent already has 10 million real-time medical records with details of 30,000,000 e-visits last year with caregivers and computer modeling of key diseases per individual that data scientists would love to get their hand on. Already, according to IDC 14% of the US population is using their phones for medical help and 200 million health and fitness related mobile applications have been downloaded according to pyramid research. Fun stuff, though early. They are probably two generations away from systems that are actually useful.
A more elaborate vision, one that is not very useful today because of lack of enough data and enough science, is defined in Experimental Man and websites like Quantified Self. Though they feel like toys today, they are much further along than the mobile phone was pre-iPhone in January of 2007. And data, the key ingredient to useful analysis, and diagnosis, is starting to explode exponentially—be it genetic data, proteomic data or physical data about my steps, my exercises, my stress levels or my normal heart and respiration rates.
My UP wristband or something like it (disclosure: I am an investor in Jawbone)) will know all my sleep patterns when I am healthy and how many steps I take each day and may have more data on my mobility if I ever get depressed than any psychiatrist ever will know what to do with. Within a few years, my band will know my heart rate at all times, my respiration rate, my galvanic skin resistance (one parameter among multiple ones used to measure my stress level), my metabolic rate (should cost about $10 to add to the band by measuring my CO2 in my breath and may detect changes in my body chemistry too like when I get a certain type of cancer and traces of it show up in my breath).
All my “health data” as well as my “sick data” and my “activity data” will be accessible to Dr. A (and location when I was stressed or breathing hard or getting the allergic reaction and what chemicals were nearby or in the air—did toluene exposure cause me to break out in a rash from that new carpet or trigger a systemic reaction from my body?). I doubt I will be prescribed an arthritis medicine without Dr. A knowing my genetics and the genetics of my autoimmune disease. Or a cancer medicine without the genetics of my cancer when the genetic sequence (once per life) costs far less than a single dose of medicine. In fact all my infectious disease treatments may be based on analysis of my full genome and my history of exposure to viruses, bacteria and toxic chemicals.
Constant everyday health data from non-medical devices will swamp the “sickness tests” used in most medical diagnosis and be supplemented by detailed genetic, proteomic and sick data with bionic software and machine learning systems. Siri might even remind me one day that my heart rate while sleeping has gone up abnormally over the last year, so I should go run some heart sickness cardiograms or imaging tests.
Obviously, Siri’s children and its server friends will be able to keep up with the latest research and decide on optimal strategies based on patient preference (“I prefer to live longer even if it means all the fancy treatments” or “I want to live a normal life and die. I prefer to spend more of my time with my children than at the hospital” or “I like taking risky treatments”). They will take into account known research, early pioneering approaches, very complex interrelationships and much more.
My best guess is that today a physician’s bias makes all these personal decisions for patients in a majority of the cases without the patient (or sometimes even the physician) realizing what “preferences “ are being incorporated into their recommendations. The situation gets worse the less educated or economically less well-off the patient is, such as in developing countries, in my estimation.
Envisioning Future Healthcare
Eventually, we won’t need the average doctor and will have much better and cheaper care for 90-99% of our medical needs. We will still need to leverage the top 10 or 20% of doctors (at least for the next two decades) to help that bionic software get better at diagnosis. So a world mostly without doctors (at least average ones) is not only not reasonable, but also more likely than not. There will be exceptions, and plenty of stories around these exceptions, but what I am talking about will most likely be the rule and doctors may be the exception rather than the other way around.
However fictionalized, we will be aiming to produce doctors like Gregory House who solve biomedical puzzles beyond our best input ability. And India, China and other countries may not have to worry about the investment in massive healthcare or massive inequalities in the type of physicians they might have access to. And hopefully our bionic software (or independent software someday) will be free of the influence of heavily marketed but only minimally effective drugs or treatment regimes or branding campaigns against generics or lower-cost and equally effective, more affordable drugs and treatments. Dr. A will be able to do a cost optimization too both at the patient level and at the policy level (but we may choose, at least for a decade or two, to reject its recommendations—we will still be free to be stupid or political).
What is important to realize is how medical education and the medical profession will change toward the better as a result of these trends. The vision I am proposing here, though, is one in which those decades of learning and experience are used where they actually matter. We consider doctors some of the most learned people in our society. We should aim to use their time and knowledge in the most efficient manner possible.
And everybody should have access to the skills of the very best ones instead of only having access to the average doctor. And the not so “Dr. House’ doctors will help us with better patient skills, bedside manners, empathy, advice and caring, and they will have more time for that too. If computers can drive cars and deal with all the knowledge in jeopardy, surely their next to next to next…generation can do diagnosis, treatment and teaching in these far less uncertain domains and with a lot more data. Further the equalizing impact of both electronic doctors and teaching environments has hugely positive social implications. Besides, who wants to be treated by an “average” doctor? And who does not want to be an empowered patient?
The best way to predict this future is not to extrapolate the past and what has or has not worked, but to invent the future we want, the one we believe possible!
Image credit: Shutterstock/koya979
P Please consider the environment before printing this e-mail.
Subscribe to:
Posts (Atom)