80 Million Links A Day Don't Lie: How Bitly Reveals The Web And The World
Erin Schulte Fast Company October 24, 2011
You might think of Bitly simply as a service that shortens links for your Twitter feed.
But to Hilary Mason, the company's chief scientist, Bitly is building a fresh new way to know what's going on in the world.
"We're able to learn things from the Bitly data set that people have never been able to share before," says Mason. "What is the half life of a link? How long will people be able to pay attention to it? What percent of links being shared on the web right now tend to be about the weather?"
The company--which shortens 80 million URLs every day--is working on a trove of new ways to mine that data. With its new search technology, for instance, it's studying and classifying every URL it shortens to index the most viral content on the web. The company recently rolled out a reputation monitoring service based on this data, which the company says acts as an "early warning system, designed to alert you in realtime to swings in volume and sentiment related to specific keywords," essentially allowing customers to monitor what people are and will be saying about products, brands, or any topic on social media.
Instead of a backward-looking view like you might get from a clipping service, Bitly is attempting to project where the conversation is headed. And unlike search--which might return, say, an official company website or a Wikipedia page--the Bitly service uncovers sites and conversations that haven't yet achieved a high pagerank (or any pagerank, for that matter).
"It's been fascinating to watch Bitly grow from a purely engineering orginzation into something where we're able to say, 'OK, there's still a lot of work that goes into that, but we're pretty good at that core piece of infrustructure--what can we build on top of it?'" says Mason. "Our products are starting to grow up and become adults."
Click play on the video below to learn more about Bitly's mission and what inspires Mason to keep innovating.
Tuesday, October 25, 2011
Monday, October 24, 2011
More Jobs Predicted for Machines, Not People
Steve Lohr The New York Times October 23, 2011
A faltering economy explains much of the job shortage in America, but advancing technology has sharply magnified the effect, more so than is generally understood, according to two researchers at the Massachusetts Institute of Technology.
The automation of more and more work once done by humans is the central theme of “Race Against the Machine,” an e-book to be published on Monday.
“Many workers, in short, are losing the race against the machine,” the authors write.
Erik Brynjolfsson, an economist and director of the M.I.T. Center for Digital Business, and Andrew P. McAfee, associate director and principal research scientist at the center, are two of the nation’s leading experts on technology and productivity. The tone of alarm in their book is a departure for the pair, whose previous research has focused mainly on the benefits of advancing technology.
Indeed, they were originally going to write a book titled, “The Digital Frontier,” about the “cornucopia of innovation that is going on,” Mr. McAfee said. Yet as the employment picture failed to brighten in the last two years, the two changed course to examine technology’s role in the jobless recovery.
The authors are not the only ones recently to point to the job fallout from technology. In the current issue of the McKinsey Quarterly, W. Brian Arthur, an external professor at the Santa Fe Institute, warns that technology is quickly taking over service jobs, following the waves of automation of farm and factory work. “This last repository of jobs is shrinking — fewer of us in the future may have white-collar business process jobs — and we have a problem,” Mr. Arthur writes.
The M.I.T. authors’ claim that automation is accelerating is not shared by some economists. Prominent among them are Robert J. Gordon of Northwestern and Tyler Cowen of George Mason University, who contend that productivity improvement owing to technological innovation rose from 1995 to 2004, but has trailed off since. Mr. Cowen emphasized that point in an e-book, “The Great Stagnation,” published this year.
Technology has always displaced some work and jobs. Over the years, many experts have warned — mistakenly — that machines were gaining the upper hand. In 1930, the economist John Maynard Keynes warned of a “new disease” that he termed “technological unemployment,” the inability of the economy to create new jobs faster than jobs were lost to automation.
But Mr. Brynjolfsson and Mr. McAfee argue that the pace of automation has picked up in recent years because of a combination of technologies including robotics, numerically controlled machines, computerized inventory control, voice recognition and online commerce.
Faster, cheaper computers and increasingly clever software, the authors say, are giving machines capabilities that were once thought to be distinctively human, like understanding speech, translating from one language to another and recognizing patterns. So automation is rapidly moving beyond factories to jobs in call centers, marketing and sales — parts of the services sector, which provides most jobs in the economy.
During the last recession, the authors write, one in 12 people in sales lost their jobs, for example. And the downturn prompted many businesses to look harder at substituting technology for people, if possible. Since the end of the recession in June 2009, they note, corporate spending on equipment and software has increased by 26 percent, while payrolls have been flat.
Corporations are doing fine. The companies in the Standard & Poor’s 500-stock index are expected to report record profits this year, a total $927 billion, estimates FactSet Research. And the authors point out that corporate profit as a share of the economy is at a 50-year high.
Productivity growth in the last decade, at more than 2.5 percent, they observe, is higher than the 1970s, 1980s and even edges out the 1990s. Still the economy, they write, did not add to its total job count, the first time that has happened over a decade since the Depression.
The skills of machines, the authors write, will only improve. In 2004, two leading economists, Frank Levy and Richard J. Murnane, published “The New Division of Labor,” which analyzed the capabilities of computers and human workers. Truck driving was cited as an example of the kind of work computers could not handle, recognizing and reacting to moving objects in real time.
But last fall, Google announced that its robot-driven cars had logged thousands of miles on American roads with only an occasional assist from human back-seat drivers. The Google cars, Mr. Brynjolfsson said, are but one sign of the times.
As others have, he pointed to I.B.M.’s “Jeopardy”-playing computer, Watson, which in February beat a pair of human “Jeopardy” champions; and Apple’s new personal assistant software, Siri, which responds to voice commands.
“This technology can do things now that only a few years ago were thought to be beyond the reach of computers,” Mr. Brynjolfsson said.
Yet computers, the authors say, tend to be narrow and literal-minded, good at assigned tasks but at a loss when a solution requires intuition and creativity — human traits. A partnership, they assert, is the path to job creation in the future.
“In medicine, law, finance, retailing, manufacturing and even scientific discovery,” they write, “the key to winning the race is not to competeagainst machines but to compete with machines.”
A faltering economy explains much of the job shortage in America, but advancing technology has sharply magnified the effect, more so than is generally understood, according to two researchers at the Massachusetts Institute of Technology.
The automation of more and more work once done by humans is the central theme of “Race Against the Machine,” an e-book to be published on Monday.
“Many workers, in short, are losing the race against the machine,” the authors write.
Erik Brynjolfsson, an economist and director of the M.I.T. Center for Digital Business, and Andrew P. McAfee, associate director and principal research scientist at the center, are two of the nation’s leading experts on technology and productivity. The tone of alarm in their book is a departure for the pair, whose previous research has focused mainly on the benefits of advancing technology.
Indeed, they were originally going to write a book titled, “The Digital Frontier,” about the “cornucopia of innovation that is going on,” Mr. McAfee said. Yet as the employment picture failed to brighten in the last two years, the two changed course to examine technology’s role in the jobless recovery.
The authors are not the only ones recently to point to the job fallout from technology. In the current issue of the McKinsey Quarterly, W. Brian Arthur, an external professor at the Santa Fe Institute, warns that technology is quickly taking over service jobs, following the waves of automation of farm and factory work. “This last repository of jobs is shrinking — fewer of us in the future may have white-collar business process jobs — and we have a problem,” Mr. Arthur writes.
The M.I.T. authors’ claim that automation is accelerating is not shared by some economists. Prominent among them are Robert J. Gordon of Northwestern and Tyler Cowen of George Mason University, who contend that productivity improvement owing to technological innovation rose from 1995 to 2004, but has trailed off since. Mr. Cowen emphasized that point in an e-book, “The Great Stagnation,” published this year.
Technology has always displaced some work and jobs. Over the years, many experts have warned — mistakenly — that machines were gaining the upper hand. In 1930, the economist John Maynard Keynes warned of a “new disease” that he termed “technological unemployment,” the inability of the economy to create new jobs faster than jobs were lost to automation.
But Mr. Brynjolfsson and Mr. McAfee argue that the pace of automation has picked up in recent years because of a combination of technologies including robotics, numerically controlled machines, computerized inventory control, voice recognition and online commerce.
Faster, cheaper computers and increasingly clever software, the authors say, are giving machines capabilities that were once thought to be distinctively human, like understanding speech, translating from one language to another and recognizing patterns. So automation is rapidly moving beyond factories to jobs in call centers, marketing and sales — parts of the services sector, which provides most jobs in the economy.
During the last recession, the authors write, one in 12 people in sales lost their jobs, for example. And the downturn prompted many businesses to look harder at substituting technology for people, if possible. Since the end of the recession in June 2009, they note, corporate spending on equipment and software has increased by 26 percent, while payrolls have been flat.
Corporations are doing fine. The companies in the Standard & Poor’s 500-stock index are expected to report record profits this year, a total $927 billion, estimates FactSet Research. And the authors point out that corporate profit as a share of the economy is at a 50-year high.
Productivity growth in the last decade, at more than 2.5 percent, they observe, is higher than the 1970s, 1980s and even edges out the 1990s. Still the economy, they write, did not add to its total job count, the first time that has happened over a decade since the Depression.
The skills of machines, the authors write, will only improve. In 2004, two leading economists, Frank Levy and Richard J. Murnane, published “The New Division of Labor,” which analyzed the capabilities of computers and human workers. Truck driving was cited as an example of the kind of work computers could not handle, recognizing and reacting to moving objects in real time.
But last fall, Google announced that its robot-driven cars had logged thousands of miles on American roads with only an occasional assist from human back-seat drivers. The Google cars, Mr. Brynjolfsson said, are but one sign of the times.
As others have, he pointed to I.B.M.’s “Jeopardy”-playing computer, Watson, which in February beat a pair of human “Jeopardy” champions; and Apple’s new personal assistant software, Siri, which responds to voice commands.
“This technology can do things now that only a few years ago were thought to be beyond the reach of computers,” Mr. Brynjolfsson said.
Yet computers, the authors say, tend to be narrow and literal-minded, good at assigned tasks but at a loss when a solution requires intuition and creativity — human traits. A partnership, they assert, is the path to job creation in the future.
“In medicine, law, finance, retailing, manufacturing and even scientific discovery,” they write, “the key to winning the race is not to competeagainst machines but to compete with machines.”
Monday, October 17, 2011
Jeff Jonas et al on Big Data
Big Data: Advancing the Art of Analytics
Federal NewsRadio Monday - 10/17/2011, 12:39pm ET
October 20th, 2010 at 11 AM
The application of knowledge discovery within the cloud is immensely powerful, but not inbuilt. We are collectively moving past the question of "what is cloud computing", and swiftly moving towards "how does the cloud enable advanced analysis against massive volumes of data?" With industry and government leveraging multiple clouds, how do we successfully share and search large collections of data across systems, departments, and geographies? Organizations will continue to discuss and better understand the analytic power and economies of cloud computing, in the sense of data storage, sharing, and management; but we are quickly discovering that creating knowledge from data is more than just a discussion of technology.
It's a discussion of what can be accomplished when massive data and cloud computing efficiencies combine to make advanced analysis and innovation possible.
Listen at http://www.federalnewsradio.com/?nid=240&sid=2080808
Panelists:
Michael Byrne- Geographic Information Officer, Federal Communications CommissionJeff Jonas- Chief Scientist, IBM Entity Analytics GroupDavid Mihelcic- Chief Technology Officer, Defense Information Systems AgencyChris Nissen- National Security Analysis Group, MITREMike Olson- Chief Executive Officer, Cloudera
Moderator: Chris Kelly - Senior Vice President, Booz Allen Hamilton
Federal NewsRadio Monday - 10/17/2011, 12:39pm ET
October 20th, 2010 at 11 AM
The application of knowledge discovery within the cloud is immensely powerful, but not inbuilt. We are collectively moving past the question of "what is cloud computing", and swiftly moving towards "how does the cloud enable advanced analysis against massive volumes of data?" With industry and government leveraging multiple clouds, how do we successfully share and search large collections of data across systems, departments, and geographies? Organizations will continue to discuss and better understand the analytic power and economies of cloud computing, in the sense of data storage, sharing, and management; but we are quickly discovering that creating knowledge from data is more than just a discussion of technology.
It's a discussion of what can be accomplished when massive data and cloud computing efficiencies combine to make advanced analysis and innovation possible.
Listen at http://www.federalnewsradio.com/?nid=240&sid=2080808
Panelists:
Michael Byrne- Geographic Information Officer, Federal Communications CommissionJeff Jonas- Chief Scientist, IBM Entity Analytics GroupDavid Mihelcic- Chief Technology Officer, Defense Information Systems AgencyChris Nissen- National Security Analysis Group, MITREMike Olson- Chief Executive Officer, Cloudera
Moderator: Chris Kelly - Senior Vice President, Booz Allen Hamilton
Tuesday, October 11, 2011
Big Data in the Dirt (and the Cloud)
Quentin Hardy The New York Times October 11, 2011, 12:01 am
Big data, the term for scanning loads of information for possibly profitable patterns, is a growing sector of corporate technology. Mostly people think in terms of online behavior, like mouse clicks, LinkedIn affiliations and Amazon shopping choices. But other big databases in the real world, lying around for years, are there to exploit.
A company called the Climate Corporation was formed in 2006 by two former Google employees who wanted to make use of the vast amount of free data published by the National Weather Service on heat and precipitation patterns around the country. At first they called the company WeatherBill, and used the data to sell insurance to businesses that depended heavily on the weather, from ski resorts and miniature golf courses to house painters and farmers.
It did pretty well, raising more than $50 million from the likes of Google Ventures, Khosla Ventures, and Allen & Company. The problem was, it was hard to sell insurance policies to so many little businesses, even using an online shopping model. People like having their insurance explained. The answer was to get even more data, and focus on the agriculture market through the same sales force that sells federal crop insurance.
“We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture,” says David Friedberg, chief executive of the Climate Corporation, a name WeatherBill started using Tuesday. “We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat.”
The product, insurance against things like drought, too much rain at the planting or the harvest, or an early freeze, is sold through 10,000 agents nationwide. The Climate Corporation, which also added Byron Dorgan, the former senator from North Dakota, to its board on Tuesday, will very likely get into insurance for specialty crops like tomatoes and grapes, which do not have federal insurance.
Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product. Mr. Friedberg was an early member of the corporate development team at Google. The co-founder, Siraj Khaliq, worked in distributed computing, which involves apportioning big data computing problems across multiple machines. He works as the Climate Corporation’s chief technical officer. Out of the staff of 60 in the company’s San Francisco office (another 30 work in the field) about 12 have doctorates, in areas like environmental science and applied mathematics.
“They like that this is a real-world problem, not just clicks on a Web site,” Mr. Friedberg says.
He figures that the Climate Corporation is one of the world’s largest users of MapReduce, an increasingly popular software technique for making sense of very large data systems. The number crunching is performed on Amazon.com’s Amazon Web Services computers.
The Climate Corporation is working with data intended to judge how different crops will react to certain soils, water and heat. It might be valuable to commodities traders as well, but Mr. Friedberg figures the better business is to expand in farming. Besides the other crops, he is looking at offering the service in Canada and Brazil, or anywhere else that he can get decent long-term data. It’s unlikely he’ll get the quality he got from the federal government, for a price anywhere near “free.”
Big data, the term for scanning loads of information for possibly profitable patterns, is a growing sector of corporate technology. Mostly people think in terms of online behavior, like mouse clicks, LinkedIn affiliations and Amazon shopping choices. But other big databases in the real world, lying around for years, are there to exploit.
A company called the Climate Corporation was formed in 2006 by two former Google employees who wanted to make use of the vast amount of free data published by the National Weather Service on heat and precipitation patterns around the country. At first they called the company WeatherBill, and used the data to sell insurance to businesses that depended heavily on the weather, from ski resorts and miniature golf courses to house painters and farmers.
It did pretty well, raising more than $50 million from the likes of Google Ventures, Khosla Ventures, and Allen & Company. The problem was, it was hard to sell insurance policies to so many little businesses, even using an online shopping model. People like having their insurance explained. The answer was to get even more data, and focus on the agriculture market through the same sales force that sells federal crop insurance.
“We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture,” says David Friedberg, chief executive of the Climate Corporation, a name WeatherBill started using Tuesday. “We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat.”
The product, insurance against things like drought, too much rain at the planting or the harvest, or an early freeze, is sold through 10,000 agents nationwide. The Climate Corporation, which also added Byron Dorgan, the former senator from North Dakota, to its board on Tuesday, will very likely get into insurance for specialty crops like tomatoes and grapes, which do not have federal insurance.
Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product. Mr. Friedberg was an early member of the corporate development team at Google. The co-founder, Siraj Khaliq, worked in distributed computing, which involves apportioning big data computing problems across multiple machines. He works as the Climate Corporation’s chief technical officer. Out of the staff of 60 in the company’s San Francisco office (another 30 work in the field) about 12 have doctorates, in areas like environmental science and applied mathematics.
“They like that this is a real-world problem, not just clicks on a Web site,” Mr. Friedberg says.
He figures that the Climate Corporation is one of the world’s largest users of MapReduce, an increasingly popular software technique for making sense of very large data systems. The number crunching is performed on Amazon.com’s Amazon Web Services computers.
The Climate Corporation is working with data intended to judge how different crops will react to certain soils, water and heat. It might be valuable to commodities traders as well, but Mr. Friedberg figures the better business is to expand in farming. Besides the other crops, he is looking at offering the service in Canada and Brazil, or anywhere else that he can get decent long-term data. It’s unlikely he’ll get the quality he got from the federal government, for a price anywhere near “free.”
Monday, October 10, 2011
NYT on Big Data: Government Aims to Build a 'Data Eye in the Sky'
John Markoff The New York Times October 10, 2011
More than 60 years ago, in his “Foundation” series, the science fiction novelist Isaac Asimov invented a new science — psychohistory — that combined mathematics and psychology to predict the future.
Now social scientists are trying to mine the vast resources of the Internet — Web searches and Twitter messages, Facebook and blog posts, the digital location trails generated by billions of cellphones — to do the same thing.
The most optimistic researchers believe that these storehouses of “big data” will for the first time reveal sociological laws of human behavior — enabling them to predict political crises, revolutions and other forms of social and economic instability, just as physicists and chemists can predict natural phenomena.
“This is a significant step forward,” said Thomas Malone, the director of the Center for Collective Intelligence at the Massachusetts Institute of Technology. “We have vastly more detailed and richer kinds of data available as well as predictive algorithms to use, and that makes possible a kind of prediction that would have never been possible before.”
The government is showing interest in the idea. This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for “big data,” according to a research proposal being circulated by the agency. The three-year experiment, to begin in April, is being financed by the Intelligence Advanced Research Projects Activity, or Iarpa (pronounced eye-AR-puh), part of the office of the director of national intelligence.
The automated data collection system is to focus on patterns of communication, consumption and movement of populations. It will use publicly accessible data, including Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries.
It is intended to be an entirely automated system, a “data eye in the sky” without human intervention, according to the program proposal. The research would not be limited to political and economic events, but would also explore the ability to predict pandemics and other types of widespread contagion, something that has been pursued independently by civilian researchers and by companies like Google.
Some social scientists and advocates of privacy rights are deeply skeptical of the project, saying it evokes queasy memories of Total Information Awareness, a post-9/11 Pentagon program that proposed hunting for potential attackers by identifying patterns in vast collections of public and private data: telephone calling records, e-mail, travel data, visa and passport information, and credit card transactions.
“I have Total Information Awareness flashbacks when things like this happen,” said David Price, an anthropologist at St. Martin’s University in Lacey, Wash., who has written about cooperation between social scientists and intelligence agencies. “On the one hand it’s understandable for a nation-state to want to track things like the outbreak of a pandemic, but I have to wonder about the total automation of this and what productive will come of it.”
Iarpa officials declined to discuss the research program, saying they are prohibited from giving interviews until contract awards are made later this year.
A similar project by their military sister organization, the Defense Advanced Research Projects Agency, or Darpa, aims to automatically identify insurgent social networks in Afghanistan.
In its most recent budget proposal, the defense agency argues that its analysis can expose terrorist cells and other stateless groups by tracking their meetings, rehearsals and sharing of material and money transfers.
So far there have been only scattered examples of the potential of mining social media. Last year HP Labs researchers used Twitter data to accurately predict box office revenues of Hollywood movies. In August, the National Science Foundation approved funds for research in using social media like Twitter and Facebook to assess earthquake damage in real time.
The accessibility and computerization of huge databases has already begun to spur the development of new statistical techniques and new software to manage data sets with trillions of entries or more.
“Big data allows one to move beyond inference and statistical significance and move toward meaningful and accurate analyses,” said Norman Nie, a political scientist who was a pioneering developer of statistical tools for social scientists and who recently formed a new company, Revolution Analytics, to develop software for the analysis of immense data sets.
Some scientists are skeptical. They cite the Pentagon’s ill-fated Project Camelot in the 1960s, which also explored the possibility that social science could predict political and economic events, but was canceled in the face of widespread criticism by scholars.
The project focused on Chile, with the goal of developing methods for anticipating “violent changes” and offering ways of averting possible rebellions. It led to an uproar among social scientists, who argued that the study would compromise their professional ethics.
In recent years, however, academic opposition to military financing of research has faded. Since 2008, a Pentagon project called the Minerva Initiative has paid for an array of studies, including research at Arizona State University into political opponents of radical Muslims and a University of Texas study on the effects of climate change on African political stability.
Social scientists who cooperate with the research agencies contend that, on balance, the new technologies will have a positive effect.
“The result will be much better understanding of what is going on in the world, and how well local governments are handling the situation,” said Sandy Pentland, a computer scientist at the M.I.T. Media Laboratory. “I find this all very hopeful rather than scary, because this is perhaps the first real opportunity for all of humanity to have transparency in government.”
But advocates of privacy rights worry that public data and the related techniques developed in the new Iarpa project will be adapted for clandestine “total information” operations.
“These techniques are double-edged,” said Marc Rotenberg, president of the Electronic Privacy Information Center, a privacy rights group based in Washington. “They can be used as easily against political opponents in the United States as they can against threats from foreign countries.”
And some computer scientists expressed skepticism about efforts to predict political instability with indicators like Web searches.
“I’m hard pressed to say that we are witnessing a revolution,” said Prabhakar Raghavan, the director of Yahoo Labs, who is an information retrieval specialist. He noted that much had been written about predicting flu epidemics by looking at Web searches for “flu,” but noted that the predictions did not improve significantly on what could already be found in data from the Centers for Disease Control and Prevention.
“You can look at search queries and divine that flu is about to break out,” he said, “but what our research has highlighted is that many of these new methods don’t add a huge lift.”
Other researchers are far more optimistic. “There is a huge amount of predictive power in this data,” said Albert-Laszlo Barabasi, a physicist at Notre Dame who specializes in network science. “If I have hourly information about your location, with about 93 percent accuracy I can predict where you are going to be an hour or a day later.”
Still, the ease of acquiring and manipulating huge data sets charting Internet behavior causes many researchers to warn that the data mining technologies may be quickly outrunning the ability of scientists to think through questions of privacy and ethics.
There is also the deeper question of whether it will be possible to discern behavioral laws that match the laws of physical sciences. For Isaac Asimov, the predictive powers of psychohistory worked only when it was possible to measure the human population of an entire galaxy.
More than 60 years ago, in his “Foundation” series, the science fiction novelist Isaac Asimov invented a new science — psychohistory — that combined mathematics and psychology to predict the future.
Now social scientists are trying to mine the vast resources of the Internet — Web searches and Twitter messages, Facebook and blog posts, the digital location trails generated by billions of cellphones — to do the same thing.
The most optimistic researchers believe that these storehouses of “big data” will for the first time reveal sociological laws of human behavior — enabling them to predict political crises, revolutions and other forms of social and economic instability, just as physicists and chemists can predict natural phenomena.
“This is a significant step forward,” said Thomas Malone, the director of the Center for Collective Intelligence at the Massachusetts Institute of Technology. “We have vastly more detailed and richer kinds of data available as well as predictive algorithms to use, and that makes possible a kind of prediction that would have never been possible before.”
The government is showing interest in the idea. This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for “big data,” according to a research proposal being circulated by the agency. The three-year experiment, to begin in April, is being financed by the Intelligence Advanced Research Projects Activity, or Iarpa (pronounced eye-AR-puh), part of the office of the director of national intelligence.
The automated data collection system is to focus on patterns of communication, consumption and movement of populations. It will use publicly accessible data, including Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries.
It is intended to be an entirely automated system, a “data eye in the sky” without human intervention, according to the program proposal. The research would not be limited to political and economic events, but would also explore the ability to predict pandemics and other types of widespread contagion, something that has been pursued independently by civilian researchers and by companies like Google.
Some social scientists and advocates of privacy rights are deeply skeptical of the project, saying it evokes queasy memories of Total Information Awareness, a post-9/11 Pentagon program that proposed hunting for potential attackers by identifying patterns in vast collections of public and private data: telephone calling records, e-mail, travel data, visa and passport information, and credit card transactions.
“I have Total Information Awareness flashbacks when things like this happen,” said David Price, an anthropologist at St. Martin’s University in Lacey, Wash., who has written about cooperation between social scientists and intelligence agencies. “On the one hand it’s understandable for a nation-state to want to track things like the outbreak of a pandemic, but I have to wonder about the total automation of this and what productive will come of it.”
Iarpa officials declined to discuss the research program, saying they are prohibited from giving interviews until contract awards are made later this year.
A similar project by their military sister organization, the Defense Advanced Research Projects Agency, or Darpa, aims to automatically identify insurgent social networks in Afghanistan.
In its most recent budget proposal, the defense agency argues that its analysis can expose terrorist cells and other stateless groups by tracking their meetings, rehearsals and sharing of material and money transfers.
So far there have been only scattered examples of the potential of mining social media. Last year HP Labs researchers used Twitter data to accurately predict box office revenues of Hollywood movies. In August, the National Science Foundation approved funds for research in using social media like Twitter and Facebook to assess earthquake damage in real time.
The accessibility and computerization of huge databases has already begun to spur the development of new statistical techniques and new software to manage data sets with trillions of entries or more.
“Big data allows one to move beyond inference and statistical significance and move toward meaningful and accurate analyses,” said Norman Nie, a political scientist who was a pioneering developer of statistical tools for social scientists and who recently formed a new company, Revolution Analytics, to develop software for the analysis of immense data sets.
Some scientists are skeptical. They cite the Pentagon’s ill-fated Project Camelot in the 1960s, which also explored the possibility that social science could predict political and economic events, but was canceled in the face of widespread criticism by scholars.
The project focused on Chile, with the goal of developing methods for anticipating “violent changes” and offering ways of averting possible rebellions. It led to an uproar among social scientists, who argued that the study would compromise their professional ethics.
In recent years, however, academic opposition to military financing of research has faded. Since 2008, a Pentagon project called the Minerva Initiative has paid for an array of studies, including research at Arizona State University into political opponents of radical Muslims and a University of Texas study on the effects of climate change on African political stability.
Social scientists who cooperate with the research agencies contend that, on balance, the new technologies will have a positive effect.
“The result will be much better understanding of what is going on in the world, and how well local governments are handling the situation,” said Sandy Pentland, a computer scientist at the M.I.T. Media Laboratory. “I find this all very hopeful rather than scary, because this is perhaps the first real opportunity for all of humanity to have transparency in government.”
But advocates of privacy rights worry that public data and the related techniques developed in the new Iarpa project will be adapted for clandestine “total information” operations.
“These techniques are double-edged,” said Marc Rotenberg, president of the Electronic Privacy Information Center, a privacy rights group based in Washington. “They can be used as easily against political opponents in the United States as they can against threats from foreign countries.”
And some computer scientists expressed skepticism about efforts to predict political instability with indicators like Web searches.
“I’m hard pressed to say that we are witnessing a revolution,” said Prabhakar Raghavan, the director of Yahoo Labs, who is an information retrieval specialist. He noted that much had been written about predicting flu epidemics by looking at Web searches for “flu,” but noted that the predictions did not improve significantly on what could already be found in data from the Centers for Disease Control and Prevention.
“You can look at search queries and divine that flu is about to break out,” he said, “but what our research has highlighted is that many of these new methods don’t add a huge lift.”
Other researchers are far more optimistic. “There is a huge amount of predictive power in this data,” said Albert-Laszlo Barabasi, a physicist at Notre Dame who specializes in network science. “If I have hourly information about your location, with about 93 percent accuracy I can predict where you are going to be an hour or a day later.”
Still, the ease of acquiring and manipulating huge data sets charting Internet behavior causes many researchers to warn that the data mining technologies may be quickly outrunning the ability of scientists to think through questions of privacy and ethics.
There is also the deeper question of whether it will be possible to discern behavioral laws that match the laws of physical sciences. For Isaac Asimov, the predictive powers of psychohistory worked only when it was possible to measure the human population of an entire galaxy.
Sunday, October 9, 2011
New Report: The open Internet - Platform for Growth
Access to internet-based content, applications and services has risen up the policy agenda because such applications are recognised to be of growing importance to citizens, consumers, public services, the economy and society; and because some network access providers have engaged in harmful blocking and discrimination against such applications.
It has also been argued by some European network operators that demand and associated traffic growth is a problem rather than an opportunity and that policy makers should support the introduction of new charges in relation to content and application providers.
The question of how the open internet is governed is one of central economic and social importance.
For this reason a group of content, application and service providers commissioned Plum to consider the question of how value is created and distributed along the value chain and what form of governance should apply in order to sustain innovation and investment along the value chain, to the benefit of all.
Our aim in producing this report has been to bring clarity to the issues - to make them as simple as possible but no simpler - and to provide a clear sense of direction for those charged with policy making and policy implementation, without being overly prescriptive given the pace of change and innovation which characterises the market.
It has also been argued by some European network operators that demand and associated traffic growth is a problem rather than an opportunity and that policy makers should support the introduction of new charges in relation to content and application providers.
The question of how the open internet is governed is one of central economic and social importance.
For this reason a group of content, application and service providers commissioned Plum to consider the question of how value is created and distributed along the value chain and what form of governance should apply in order to sustain innovation and investment along the value chain, to the benefit of all.
Our aim in producing this report has been to bring clarity to the issues - to make them as simple as possible but no simpler - and to provide a clear sense of direction for those charged with policy making and policy implementation, without being overly prescriptive given the pace of change and innovation which characterises the market.
WashPost: Innovators, data will set you free
Dominic Basulto The Washington Post October 7, 2011
The pace of data creation on the Internet is growing at an exponential rate. In any 48-hour period in 2010, more data was created than had been created by all of humanity in the past 30,000 years, according to a presentation by entrepreneur Yuri Milner. By the year 2020, that same amount of data will be created in a single hour.
“Information wants to be free,” declared writer Stewart Brand,
Data will set you free.
Digital Libertarianism seems like a radical concept until you consider the fact that - across many industries - regulators are no longer able to keep up with all the data that is being created every day. The most obvious example is the financial markets, where even traders and investors talk about the creation of a vast new "shadow market" where financial instruments are just too complex to regulate and algorithmic traders, using powerful supercomputers to analyze all of the data, can yank the market down in a matter of seconds. Viewed in this way, is it actually possible that it would be in a financial institution’s competitive best-interest to release as much data as possible into the marketplace, in exchange for freedom from regulatory shackles?
How can this be the case? Isn’t proprietary data the key to competitive success in the markets? Not always — consider all of the open innovation initiatives formed to solve the world’s most difficult problems in fields like biology and chemistry and physics. Large corporations have found that full data disclosure, offered in good faith and made available to researchers, can be a way to accelerate growth — especially if the crowd gets involved. It can also be a brake on business practices that harm society. Take taxes, for example. Full data disclosure can also solve for problems like the propensity of individuals and corporations to play fast and loose with their taxes. Governments have always struggled to collect the full tax burden from individuals and corporations.
Consider your own experience on a site like Facebook. The more information and data that you make available to others, the more that you are held accountable to others. On the flip side, all this data leads to new types of experiences, like the ability for corporations to perfectly customize their products to your tastes and needs. Now that Facebook has created the Timeline, people will know exactly what you read, where you were last night, what music you listened to and what Web site you think has the best “LOL kittens.” Your personal life just became public. Imagine if corporations had to disclose as much data about everything they did.
As Supreme Court Justice Louis Brandeis once said: “Sunlight is the best disinfectant“? Perhaps that could be the case here as well.
The idea of making data completely open and transparent is gaining traction in tech circles. Consider New York City's recent overture to the tech crowd, opening up all of its data about the city to developers and entrepreneurs. Government officials could easily hide information about crime reports, public health issues, building complaints and traffic reports. But what if this information could become the stimulus for new innovation by the crowd to solve problems that the government could not?
An increase in the amount of data made freely available to all could lead to governments and businesses doing more good for society. That may sound hopelessly idealistic, but so was the original premise of the Web. As participants in a democracy, we have rights as well as responsibilities. In an age of Wikileaks and forced data transparency, one new responsibility might be to take more than a passing interest in all of the data being created every 48 hours, and instead, use it to hold our society’s institutions fully accountable.
and so does all of this data. In fact, the big data movement — organizations scrambling to make sense of all this data — may be the single most profound trend that the Web has created in the past 12 months. It has also given rise to a group of Digital Libertarians who see the beginning of a new era in which total data transparency makes it possible to regulate the activity of corporations, governments and individuals.
The pace of data creation on the Internet is growing at an exponential rate. In any 48-hour period in 2010, more data was created than had been created by all of humanity in the past 30,000 years, according to a presentation by entrepreneur Yuri Milner. By the year 2020, that same amount of data will be created in a single hour.
“Information wants to be free,” declared writer Stewart Brand,
Data will set you free.
Digital Libertarianism seems like a radical concept until you consider the fact that - across many industries - regulators are no longer able to keep up with all the data that is being created every day. The most obvious example is the financial markets, where even traders and investors talk about the creation of a vast new "shadow market" where financial instruments are just too complex to regulate and algorithmic traders, using powerful supercomputers to analyze all of the data, can yank the market down in a matter of seconds. Viewed in this way, is it actually possible that it would be in a financial institution’s competitive best-interest to release as much data as possible into the marketplace, in exchange for freedom from regulatory shackles?
How can this be the case? Isn’t proprietary data the key to competitive success in the markets? Not always — consider all of the open innovation initiatives formed to solve the world’s most difficult problems in fields like biology and chemistry and physics. Large corporations have found that full data disclosure, offered in good faith and made available to researchers, can be a way to accelerate growth — especially if the crowd gets involved. It can also be a brake on business practices that harm society. Take taxes, for example. Full data disclosure can also solve for problems like the propensity of individuals and corporations to play fast and loose with their taxes. Governments have always struggled to collect the full tax burden from individuals and corporations.
Consider your own experience on a site like Facebook. The more information and data that you make available to others, the more that you are held accountable to others. On the flip side, all this data leads to new types of experiences, like the ability for corporations to perfectly customize their products to your tastes and needs. Now that Facebook has created the Timeline, people will know exactly what you read, where you were last night, what music you listened to and what Web site you think has the best “LOL kittens.” Your personal life just became public. Imagine if corporations had to disclose as much data about everything they did.
As Supreme Court Justice Louis Brandeis once said: “Sunlight is the best disinfectant“? Perhaps that could be the case here as well.
The idea of making data completely open and transparent is gaining traction in tech circles. Consider New York City's recent overture to the tech crowd, opening up all of its data about the city to developers and entrepreneurs. Government officials could easily hide information about crime reports, public health issues, building complaints and traffic reports. But what if this information could become the stimulus for new innovation by the crowd to solve problems that the government could not?
An increase in the amount of data made freely available to all could lead to governments and businesses doing more good for society. That may sound hopelessly idealistic, but so was the original premise of the Web. As participants in a democracy, we have rights as well as responsibilities. In an age of Wikileaks and forced data transparency, one new responsibility might be to take more than a passing interest in all of the data being created every 48 hours, and instead, use it to hold our society’s institutions fully accountable.
and so does all of this data. In fact, the big data movement — organizations scrambling to make sense of all this data — may be the single most profound trend that the Web has created in the past 12 months. It has also given rise to a group of Digital Libertarians who see the beginning of a new era in which total data transparency makes it possible to regulate the activity of corporations, governments and individuals.
Thursday, October 6, 2011
Brian Arthur: Contemplate an economy run entirely by machines
The second economy
Digitization is creating a second economy that’s vast, automatic, and invisible—thereby bringing the biggest change since the Industrial Revolution.
W. Brian Arthur McKinsey Quarterly October 2011
In This Article
· Sidebar: How fast is the second economy growing?
· About the author
In 1850, a decade before the Civil War, the United States’ economy was small—it wasn’t much bigger than Italy’s. Forty years later, it was the largest economy in the world. What happened in-between was the railroads. They linked the east of the country to the west, and the interior to both. They gave access to the east’s industrial goods; they made possible economies of scale; they stimulated steel and manufacturing—and the economy was never the same.
Deep changes like this are not unusual. Every so often—every 60 years or so—a body of technology comes along and over several decades, quietly, almost unnoticeably, transforms the economy: it brings new social classes to the fore and creates a different world for business. Can such a transformation—deep and slow and silent—be happening today
We could look for one in the genetic technologies, or in nanotech, but their time hasn’t fully come. But I want to argue that something deep is going on with information technology, something that goes well beyond the use of computers, social media, and commerce on the Internet. Business processes that once took place among human beings are now being executed electronically. They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential—it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.
Let me begin with two examples. Twenty years ago, if you went into an airport you would walk up to a counter and present paper tickets to a human being. That person would register you on a computer, notify the flight you’d arrived, and check your luggage in. All this was done by humans. Today, you walk into an airport and look for a machine. You put in a frequent-flier card or credit card, and it takes just three or four seconds to get back a boarding pass, receipt, and luggage tag. What interests me is what happens in those three or four seconds. The moment the card goes in, you are starting a huge conversation conducted entirely among machines. Once your name is recognized, computers are checking your flight status with the airlines, your past travel history, your name with the TSA1 (and possibly also with the National Security Agency). They are checking your seat choice, your frequent-flier status, and your access to lounges. This unseen, underground conversation is happening among multiple servers talking to other servers, talking to satellites that are talking to computers (possibly in London, where you’re going), and checking with passport control, with foreign immigration, with ongoing connecting flights. And to make sure the aircraft’s weight distribution is fine, the machines are also starting to adjust the passenger count and seating according to whether the fuselage is loaded more heavily at the front or back.
These large and fairly complicated conversations that you’ve triggered occur entirely among things remotely talking to other things: servers, switches, routers, and other Internet and telecommunications devices, updating and shuttling information back and forth. All of this occurs in the few seconds it takes to get your boarding pass back. And even after that happens, if you could see these conversations as flashing lights, they’d still be flashing all over the country for some time, perhaps talking to the flight controllers—starting to say that the flight’s getting ready for departure and to prepare for that.
Now consider a second example, from supply chain management. Twenty years ago, if you were shipping freight through Rotterdam into the center of Europe, people with clipboards would be registering arrival, checking manifests, filling out paperwork, and telephoning forward destinations to let other people know. Now such shipments go through an RFID2 portal where they are scanned, digitally captured, and automatically dispatched. The RFID portal is in conversation digitally with the originating shipper, other depots, other suppliers, and destinations along the route, all keeping track, keeping control, and reconfiguring routing if necessary to optimize things along the way. What used to be done by humans is now executed as a series of conversations among remotely located servers.
In both these examples, and all across economies in the developed world, processes in the physical economy are being entered into the digital economy, where they are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.
So we can say that another economy—a second economy—of all of these digitized business processes conversing, executing, and triggering further actions is silently forming alongside the physical economy.
Aspen root systems
If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent—a great computer expression—which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.
These last descriptors sound biological—and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, there’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.
The metaphor isn’t perfect: this emerging second-economy root system is more complicated than any aspen system, since it’s also making new connections and new configurations on the fly. But the aspen metaphor is useful for capturing the reality that the observable physical world of aspen trees hides an unseen underground root system just as large or even larger.
How large is the unseen second economy? By a rough back-of-the-envelope calculation (see sidebar, “How fast is the second economy growing?”), in about two decades the digital economy will reach the same size as the physical economy. It’s as if there will be another American economy anchored off San Francisco (or, more in keeping with my metaphor, slipped in underneath the original economy) and growing all the while.
Now this second, digital economy isn’t producing anything tangible. It’s not making my bed in a hotel, or bringing me orange juice in the morning. But it is running an awful lot of the economy. It’s helping architects design buildings, it’s tracking sales and inventory, getting goods from here to there, executing trades and banking operations, controlling manufacturing equipment, making design calculations, billing clients, navigating aircraft, helping diagnose patients, and guiding laparoscopic surgeries. Such operations grow slowly and take time to form. In any deep transformation, industries do not so much adopt the new body of technology as encounter it, and as they do so they create new ways to profit from its possibilities.
The deep transformation I am describing is happening not just in the United States but in all advanced economies, especially in Europe and Japan. And its revolutionary scale can only be grasped if we go beyond my aspen metaphor to another analogy.
A neural system for the economy
Recall that in the digital conversations I was describing, something that occurs in the physical economy is sensed by the second economy—which then gives back an appropriate response. A truck passes its load through an RFID sensor or you check in at the airport, a lot of recomputation takes place, and appropriate physical actions are triggered.
There’s a parallel in this with how biologists think of intelligence. I’m not talking about human intelligence or anything that would qualify as conscious intelligence. Biologists tell us that an organism is intelligent if it senses something, changes its internal state, and reacts appropriately. If you put an E. coli bacterium into an uneven concentration of glucose, it does the appropriate thing by swimming toward where the glucose is more concentrated. Biologists would call this intelligent behavior. The bacterium senses something, “computes” something (although we may not know exactly how), and returns an appropriate response.
No brain need be involved. A primitive jellyfish doesn’t have a central nervous system or brain. What it has is a kind of neural layer or nerve net that lets it sense and react appropriately. I’m arguing that all these aspen roots—this vast global digital network that is sensing, “computing,” and reacting appropriately—is starting to constitute a neural layer for the economy. The second economy constitutes a neural layer for the physical economy. Just what sort of change is this qualitatively?
Think of it this way. With the coming of the Industrial Revolution—roughly from the 1760s, when Watt’s steam engine appeared, through around 1850 and beyond—the economy developed a muscular system in the form of machine power. Now it is developing a neural system. This may sound grandiose, but actually I think the metaphor is valid. Around 1990, computers started seriously to talk to each other, and all these connections started to happen. The individual machines—servers—are like neurons, and the axons and synapses are the communication pathways and linkages that enable them to be in conversation with each other and to take appropriate action.
Is this the biggest change since the Industrial Revolution? Well, without sticking my neck out too much, I believe so. In fact, I think it may well be the biggest change ever in the economy. It is a deep qualitative change that is bringing intelligent, automatic response to the economy. There’s no upper limit to this, no place where it has to end. Now, I’m not interested in science fiction, or predicting the singularity, or talking about cyborgs. None of that interests me. What I am saying is that it would be easy to underestimate the degree to which this is going to make a difference.
I think that for the rest of this century, barring wars and pestilence, a lot of the story will be the building out of this second economy, an unseen underground economy that basically is giving us intelligent reactions to what we do above the ground. For example, if I’m driving in Los Angeles in 15 years’ time, likely it’ll be a driverless car in a flow of traffic where my car’s in a conversation with the cars around it that are in conversation with general traffic and with my car. The second economy is creating for us—slowly, quietly, and steadily—a different world.
A downside
Of course, as with most changes, there is a downside. I am concerned that there is an adverse impact on jobs. Productivity increasing, say, at 2.4 percent in a given year means either that the same number of people can produce 2.4 percent more output or that we can get the same output with 2.4 percent fewer people. Both of these are happening. We are getting more output for each person in the economy, but overall output, nationally, requires fewer people to produce it. Nowadays, fewer people are required behind the desk of an airline. Much of the work is still physical—someone still has to take your luggage and put it on the belt—but much has vanished into the digital world of sensing, digital communication, and intelligent response.
Physical jobs are disappearing into the second economy, and I believe this effect is dwarfing the much more publicized effect of jobs disappearing to places like India and China.
There are parallels with what has happened before. In the early 20th century, farm jobs became mechanized and there was less need for farm labor, and some decades later manufacturing jobs became mechanized and there was less need for factory labor. Now business processes—many in the service sector—are becoming “mechanized” and fewer people are needed, and this is exerting systematic downward pressure on jobs. We don’t have paralegals in the numbers we used to. Or draftsmen, telephone operators, typists, or bookkeeping people. A lot of that work is now done digitally. We do have police and teachers and doctors; where there’s a need for human judgment and human interaction, we still have that. But the primary cause of all of the downsizing we’ve had since the mid-1990s is that a lot of human jobs are disappearing into the second economy. Not to reappear.
Seeing things this way, it’s not surprising we are still working our way out of the bad 2008–09 recession with a great deal of joblessness.
There’s a larger lesson to be drawn from this. The second economy will certainly be the engine of growth and the provider of prosperity for the rest of this century and beyond, but it may not provide jobs, so there may be prosperity without full access for many. This suggests to me that the main challenge of the economy is shifting from producing prosperity to distributing prosperity. The second economy will produce wealth no matter what we do; distributing that wealth has become the main problem. For centuries, wealth has traditionally been apportioned in the West through jobs, and jobs have always been forthcoming. When farm jobs disappeared, we still had manufacturing jobs, and when these disappeared we migrated to service jobs. With this digital transformation, this last repository of jobs is shrinking—fewer of us in the future may have white-collar business process jobs—and we face a problem.
The system will adjust of course, though I can’t yet say exactly how. Perhaps some new part of the economy will come forward and generate a whole new set of jobs. Perhaps we will have short workweeks and long vacations so there will be more jobs to go around. Perhaps we will have to subsidize job creation. Perhaps the very idea of a job and of being productive will change over the next two or three decades. The problem is by no means insoluble. The good news is that if we do solve it we may at last have the freedom to invest our energies in creative acts.
Economic possibilities for our grandchildren
In 1930, Keynes wrote a famous essay, “Economic possibilities for our grandchildren.” Reading it now, in the era of those grandchildren, I am surprised just how accurate it is. Keynes predicts that “the standard of life in progressive countries one hundred years hence will be between four and eight times as high as it is to-day.” He rightly warns of “technological unemployment,” but dares to surmise that “the economic problem [of producing enough goods] may be solved.” If we had asked him and his contemporaries how all this might come about, they might have imagined lots of factories with lots of machines, possibly even with robots, with the workers in these factories gradually being replaced by machines and by individual robots.
That is not quite how things have developed. We do have sophisticated machines, but in the place of personal automation (robots) we have a collective automation. Underneath the physical economy, with its physical people and physical tasks, lies a second economy that is automatic and neurally intelligent, with no upper limit to its buildout. The prosperity we enjoy and the difficulties with jobs would not have surprised Keynes, but the means of achieving that prosperity would have.
Digitization is creating a second economy that’s vast, automatic, and invisible—thereby bringing the biggest change since the Industrial Revolution.
W. Brian Arthur McKinsey Quarterly October 2011
In This Article
· Sidebar: How fast is the second economy growing?
· About the author
In 1850, a decade before the Civil War, the United States’ economy was small—it wasn’t much bigger than Italy’s. Forty years later, it was the largest economy in the world. What happened in-between was the railroads. They linked the east of the country to the west, and the interior to both. They gave access to the east’s industrial goods; they made possible economies of scale; they stimulated steel and manufacturing—and the economy was never the same.
Deep changes like this are not unusual. Every so often—every 60 years or so—a body of technology comes along and over several decades, quietly, almost unnoticeably, transforms the economy: it brings new social classes to the fore and creates a different world for business. Can such a transformation—deep and slow and silent—be happening today
We could look for one in the genetic technologies, or in nanotech, but their time hasn’t fully come. But I want to argue that something deep is going on with information technology, something that goes well beyond the use of computers, social media, and commerce on the Internet. Business processes that once took place among human beings are now being executed electronically. They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential—it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.
Let me begin with two examples. Twenty years ago, if you went into an airport you would walk up to a counter and present paper tickets to a human being. That person would register you on a computer, notify the flight you’d arrived, and check your luggage in. All this was done by humans. Today, you walk into an airport and look for a machine. You put in a frequent-flier card or credit card, and it takes just three or four seconds to get back a boarding pass, receipt, and luggage tag. What interests me is what happens in those three or four seconds. The moment the card goes in, you are starting a huge conversation conducted entirely among machines. Once your name is recognized, computers are checking your flight status with the airlines, your past travel history, your name with the TSA1 (and possibly also with the National Security Agency). They are checking your seat choice, your frequent-flier status, and your access to lounges. This unseen, underground conversation is happening among multiple servers talking to other servers, talking to satellites that are talking to computers (possibly in London, where you’re going), and checking with passport control, with foreign immigration, with ongoing connecting flights. And to make sure the aircraft’s weight distribution is fine, the machines are also starting to adjust the passenger count and seating according to whether the fuselage is loaded more heavily at the front or back.
These large and fairly complicated conversations that you’ve triggered occur entirely among things remotely talking to other things: servers, switches, routers, and other Internet and telecommunications devices, updating and shuttling information back and forth. All of this occurs in the few seconds it takes to get your boarding pass back. And even after that happens, if you could see these conversations as flashing lights, they’d still be flashing all over the country for some time, perhaps talking to the flight controllers—starting to say that the flight’s getting ready for departure and to prepare for that.
Now consider a second example, from supply chain management. Twenty years ago, if you were shipping freight through Rotterdam into the center of Europe, people with clipboards would be registering arrival, checking manifests, filling out paperwork, and telephoning forward destinations to let other people know. Now such shipments go through an RFID2 portal where they are scanned, digitally captured, and automatically dispatched. The RFID portal is in conversation digitally with the originating shipper, other depots, other suppliers, and destinations along the route, all keeping track, keeping control, and reconfiguring routing if necessary to optimize things along the way. What used to be done by humans is now executed as a series of conversations among remotely located servers.
In both these examples, and all across economies in the developed world, processes in the physical economy are being entered into the digital economy, where they are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.
So we can say that another economy—a second economy—of all of these digitized business processes conversing, executing, and triggering further actions is silently forming alongside the physical economy.
Aspen root systems
If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent—a great computer expression—which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.
These last descriptors sound biological—and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, there’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.
The metaphor isn’t perfect: this emerging second-economy root system is more complicated than any aspen system, since it’s also making new connections and new configurations on the fly. But the aspen metaphor is useful for capturing the reality that the observable physical world of aspen trees hides an unseen underground root system just as large or even larger.
How large is the unseen second economy? By a rough back-of-the-envelope calculation (see sidebar, “How fast is the second economy growing?”), in about two decades the digital economy will reach the same size as the physical economy. It’s as if there will be another American economy anchored off San Francisco (or, more in keeping with my metaphor, slipped in underneath the original economy) and growing all the while.
Now this second, digital economy isn’t producing anything tangible. It’s not making my bed in a hotel, or bringing me orange juice in the morning. But it is running an awful lot of the economy. It’s helping architects design buildings, it’s tracking sales and inventory, getting goods from here to there, executing trades and banking operations, controlling manufacturing equipment, making design calculations, billing clients, navigating aircraft, helping diagnose patients, and guiding laparoscopic surgeries. Such operations grow slowly and take time to form. In any deep transformation, industries do not so much adopt the new body of technology as encounter it, and as they do so they create new ways to profit from its possibilities.
The deep transformation I am describing is happening not just in the United States but in all advanced economies, especially in Europe and Japan. And its revolutionary scale can only be grasped if we go beyond my aspen metaphor to another analogy.
A neural system for the economy
Recall that in the digital conversations I was describing, something that occurs in the physical economy is sensed by the second economy—which then gives back an appropriate response. A truck passes its load through an RFID sensor or you check in at the airport, a lot of recomputation takes place, and appropriate physical actions are triggered.
There’s a parallel in this with how biologists think of intelligence. I’m not talking about human intelligence or anything that would qualify as conscious intelligence. Biologists tell us that an organism is intelligent if it senses something, changes its internal state, and reacts appropriately. If you put an E. coli bacterium into an uneven concentration of glucose, it does the appropriate thing by swimming toward where the glucose is more concentrated. Biologists would call this intelligent behavior. The bacterium senses something, “computes” something (although we may not know exactly how), and returns an appropriate response.
No brain need be involved. A primitive jellyfish doesn’t have a central nervous system or brain. What it has is a kind of neural layer or nerve net that lets it sense and react appropriately. I’m arguing that all these aspen roots—this vast global digital network that is sensing, “computing,” and reacting appropriately—is starting to constitute a neural layer for the economy. The second economy constitutes a neural layer for the physical economy. Just what sort of change is this qualitatively?
Think of it this way. With the coming of the Industrial Revolution—roughly from the 1760s, when Watt’s steam engine appeared, through around 1850 and beyond—the economy developed a muscular system in the form of machine power. Now it is developing a neural system. This may sound grandiose, but actually I think the metaphor is valid. Around 1990, computers started seriously to talk to each other, and all these connections started to happen. The individual machines—servers—are like neurons, and the axons and synapses are the communication pathways and linkages that enable them to be in conversation with each other and to take appropriate action.
Is this the biggest change since the Industrial Revolution? Well, without sticking my neck out too much, I believe so. In fact, I think it may well be the biggest change ever in the economy. It is a deep qualitative change that is bringing intelligent, automatic response to the economy. There’s no upper limit to this, no place where it has to end. Now, I’m not interested in science fiction, or predicting the singularity, or talking about cyborgs. None of that interests me. What I am saying is that it would be easy to underestimate the degree to which this is going to make a difference.
I think that for the rest of this century, barring wars and pestilence, a lot of the story will be the building out of this second economy, an unseen underground economy that basically is giving us intelligent reactions to what we do above the ground. For example, if I’m driving in Los Angeles in 15 years’ time, likely it’ll be a driverless car in a flow of traffic where my car’s in a conversation with the cars around it that are in conversation with general traffic and with my car. The second economy is creating for us—slowly, quietly, and steadily—a different world.
A downside
Of course, as with most changes, there is a downside. I am concerned that there is an adverse impact on jobs. Productivity increasing, say, at 2.4 percent in a given year means either that the same number of people can produce 2.4 percent more output or that we can get the same output with 2.4 percent fewer people. Both of these are happening. We are getting more output for each person in the economy, but overall output, nationally, requires fewer people to produce it. Nowadays, fewer people are required behind the desk of an airline. Much of the work is still physical—someone still has to take your luggage and put it on the belt—but much has vanished into the digital world of sensing, digital communication, and intelligent response.
Physical jobs are disappearing into the second economy, and I believe this effect is dwarfing the much more publicized effect of jobs disappearing to places like India and China.
There are parallels with what has happened before. In the early 20th century, farm jobs became mechanized and there was less need for farm labor, and some decades later manufacturing jobs became mechanized and there was less need for factory labor. Now business processes—many in the service sector—are becoming “mechanized” and fewer people are needed, and this is exerting systematic downward pressure on jobs. We don’t have paralegals in the numbers we used to. Or draftsmen, telephone operators, typists, or bookkeeping people. A lot of that work is now done digitally. We do have police and teachers and doctors; where there’s a need for human judgment and human interaction, we still have that. But the primary cause of all of the downsizing we’ve had since the mid-1990s is that a lot of human jobs are disappearing into the second economy. Not to reappear.
Seeing things this way, it’s not surprising we are still working our way out of the bad 2008–09 recession with a great deal of joblessness.
There’s a larger lesson to be drawn from this. The second economy will certainly be the engine of growth and the provider of prosperity for the rest of this century and beyond, but it may not provide jobs, so there may be prosperity without full access for many. This suggests to me that the main challenge of the economy is shifting from producing prosperity to distributing prosperity. The second economy will produce wealth no matter what we do; distributing that wealth has become the main problem. For centuries, wealth has traditionally been apportioned in the West through jobs, and jobs have always been forthcoming. When farm jobs disappeared, we still had manufacturing jobs, and when these disappeared we migrated to service jobs. With this digital transformation, this last repository of jobs is shrinking—fewer of us in the future may have white-collar business process jobs—and we face a problem.
The system will adjust of course, though I can’t yet say exactly how. Perhaps some new part of the economy will come forward and generate a whole new set of jobs. Perhaps we will have short workweeks and long vacations so there will be more jobs to go around. Perhaps we will have to subsidize job creation. Perhaps the very idea of a job and of being productive will change over the next two or three decades. The problem is by no means insoluble. The good news is that if we do solve it we may at last have the freedom to invest our energies in creative acts.
Economic possibilities for our grandchildren
In 1930, Keynes wrote a famous essay, “Economic possibilities for our grandchildren.” Reading it now, in the era of those grandchildren, I am surprised just how accurate it is. Keynes predicts that “the standard of life in progressive countries one hundred years hence will be between four and eight times as high as it is to-day.” He rightly warns of “technological unemployment,” but dares to surmise that “the economic problem [of producing enough goods] may be solved.” If we had asked him and his contemporaries how all this might come about, they might have imagined lots of factories with lots of machines, possibly even with robots, with the workers in these factories gradually being replaced by machines and by individual robots.
That is not quite how things have developed. We do have sophisticated machines, but in the place of personal automation (robots) we have a collective automation. Underneath the physical economy, with its physical people and physical tasks, lies a second economy that is automatic and neurally intelligent, with no upper limit to its buildout. The prosperity we enjoy and the difficulties with jobs would not have surprised Keynes, but the means of achieving that prosperity would have.
his second economy that is silently forming—vast, interconnected, and extraordinarily productive—is creating for us a new economic world. How we will fare in this world, how we will adapt to it, how we will profit from it and share its benefits, is very much up to us.
W. Brian Arthur is a visting researcher with the Intelligent System Lab at the Palo Alto Research Center (PARC) and an external professor at the Santa Fe Institute. He is an economist and technology thinker and a pioneer in the science of complexity. His 1994 book, Increasing Returns and Path Dependence in the Economy (University of Michigan Press, December 1994), contains several of his seminal papers. More recently, Arthur was the author of The Nature of Technology: What it is and How it Evolves (Free Press, August 2009).
Tuesday, October 4, 2011
Rob Atkinson: Government Opportunities to Harness “Big Data”
Posted by Rob Atkinson to Innovation Policy Blog
 |
Recently more attention has been drawn to the emergence of "Big Data"—large scale data sets that businesses and government are using to unlock new value using today's computing and communications power. As a McKinsey Global Institute (MGI) study recently showed, Big Data offers a wide range of commercial opportunities in virtually every sector of the economy for the United States. To take one example, the MGI estimates that better use of big data in health care could generate an additional $300 billion, with approximately two-thirds of that values coming from more efficient delivery of health care.
The use of Big Data should not be confined to just the private sector; data offers incredible new opportunities to the public sector as well.Policymakers have the opportunity to use Big Data to improve government in areas such as public safety, public health, public utilities and public transportation.
ITIF has discussed many of these opportunities before.
ITIF has discussed many of these opportunities before.
Consider the following:
- Electric power utilities can use data analytics and smart meters to better manage resources and avoid blackouts,
- Food inspectors can use data to better track meat and produce safety from farm to fork ,
- Public health officials can use health data to detect infectious disease outbreaks,
- Regulators can track pharmaceutical and medical device safety and effectiveness through better data analytics,
- Police departments can use data analytics to target crime hotspots and prevent crime waves,
- Public utilities can use sensors to collect data on water and sewer usage to detect leaks and reduce water consumption,
- First responders can use sensors, GPS, cameras and better communication systems to let police and fire fighters better protect citizens when responding to emergencies,
- State departments of transportation can use data to reduce traffic, more efficiently deploy resources, and implement congestion pricing systems.
Better use of data can help government agencies, from city agencies to federal bureaucracies, operate more efficiently, create more transparency, and make more informed decisions. And government can use cloud computing to more efficiently develop online systems that provide anytime, anywhere access to information. However, government officials should do more to spur uses of data. Taking advantage of these opportunities will require federal government leadership, such as the Department of Commerce creating a data policy office to spur data innovation and overcome obstacles to adoption, all the while protecting privacy. And going forward, government agencies will increasingly have to deal with issues such as data security and identity management, so these issues do not become impediments to successful utilization of data analytics. Local governments can help pioneer the use of data as well. For example, the city of Boston city sponsored the development of a mobile app "Street Bump" to automatically determine where potholes are based on data collected using citizen's smart phones equipped with GPS and accelerometers. Tools like these are helping create "smart cities" and build a world that is alive with information.
Although there have been many successes in this area, much more can be done. For example, in homeland security, law enforcement must deal with a changing threat landscape. While corporations and individuals can increasingly use better technology to communicate and store data security, criminals can also use these same tools. As a result, law enforcement is increasingly confronting the "Going Dark" problem where they have less access to investigative data, not because of a lack of legal authority, but because of technological hurdles. Yet while law enforcement may have a reduced ability to intercept some types of communication, they now have many more sources of data, such as transactional data, to use to detect threats. As ITIF discussed at an event in 2010 following the Christmas Day terrorist attempt, the intelligence community still needs to develop better analytical tools to "connect the dots" and allow intelligence officers to do a better job. Similarly in many other sectors, Big Data offers government opportunities to reinvent how to operate effectively.
Overall, more investment in data infrastructure and analytics will enable government to better provide and efficiently deliver values and services to its citizens.
Silicon Valley reborn as Smartphone Valley (FT)
By Chris Nuttall in Palo Alto October 3, 2011 4:55 pm Finncial Times
Apple’s unveiling of the iPhone 5 on Tuesday at its Cupertino headquarters is just the latest sign that Silicon Valley is taking on a fresh mantle of Smartphone Valley, with its growing reputation making it a magnet for mobile operators around the world.
AT&T, Verizon and Vodafone have all just opened research, testing and incubation centres in San Francisco and the Valley only weeks apart.
“The reason we are here is this is the centre of the Earth right now when it comes to innovation,” said Fay Arjomandi, head of research and development for Vodafone in the US, at the opening of its Xone Lab in Redwood City last month.
Apple’s iPhone and Google’s Android operating system have set new standards for hardware and software in the mobile industry.
The app culture that both have engendered means there is a race on among operators to feature the newest trends and advances in software first, amid fierce competition for the attention of Silicon Valley developers.
Vodafone’s lab allows developers to incubate their ideas and test how apps and services would perform on the worldwide networks it replicates at its facility.
“We want to bring to market these ideas and accelerate the process – the hope is that six to nine months after we prove something here, it can be in the hands of a user in one of our operating countries,” says Ms Arjomandi.
A week after the Vodafone opening, AT&T launched its Foundry innovation lab in an old French steam laundry in Palo Alto.
It too emphasised the need to move “at web developer rather than carrier speed” – to the extent that much of the office furniture is on wheels and is easily reconfigurable.
“This is part of our transformation – to move from being a telecoms company to a technology company,” said John Donovan, AT&T’s chief technology officer.
The operators have felt the need to change as technology companies such as Apple and Google, which is buying Motorola, have become more influential and handset makers have aligned themselves behind different operating systems.
“They have to show they are a part of the ecosystem, there’s been a big shift in the last year or so, when there has been the threat that carriers just become a dumb pipe [for data],” says Chris Jones, a telecoms analyst based in the Valley for the Canalys research firm.
Trevor Healy, chief executive of the mobile advertising platform Amobee, whose previous Valley company Jajah was bought by Spain’s Telefónica, has also noticed a change in attitude forced upon the operators.
“It’s not long ago that these companies were myopic and closed, but now there’s a carrot and stick for them in the shape of new revenue streams when their core revenues are stagnant and [they face] the stick of other big players like Google and Apple making ground,” he says.
The operators are arriving late to the Valley compared with infrastructure and handset players such as Ericsson and Nokia, who have had research centres here for years and base their chief technology officers in the Bay Area.
“The operators may have abdicated their positions with the operating systems and handsets, but now they’re putting a stake in the ground and focusing on app development, advertising and payments and billing,” says Mr Healy.
The consumer smartphone market may appear to have been carved up between Apple’s App Store and the Android Market, but operators feel they can still offer premium apps and services based on their network’s ability to know location, process payments, serve advertising, communicate with a range of devices and combine disparate services.
Mr Donovan boasted that the Foundry had 106 projects under way and that AT&T was now measuring itself on the openness of its network: 82 APIs (or access points into the network) had been launched over the past year.
Start-up demonstrations at both the AT&T and Vodafone opening events hinted at the “mash-ups” that were possible through mobile networks – for example, how an app recording running performances could link up with ones relaying sensor information on a person’s health.
Augmented reality applications, improved graphics for gaming and the ability for machines to talk to other machines – “the internet of things” – could all benefit from being hard-wired into the operators’ networks, the demonstrations proved.
It remains to be seen whether developers will be attracted in significant numbers to working with the operators, but Ms Arjomandi argues all parties can benefit.
“We would like to have the advantage of [perhaps one-year exclusivity] lead time to the market – to have a differentiation for us, we need to get that lead time,” she said.
“In exchange, we give them access to 340m customers around the world in 30-plus countries – so it’s definitely a win-win situation.”
Apple’s unveiling of the iPhone 5 on Tuesday at its Cupertino headquarters is just the latest sign that Silicon Valley is taking on a fresh mantle of Smartphone Valley, with its growing reputation making it a magnet for mobile operators around the world.
AT&T, Verizon and Vodafone have all just opened research, testing and incubation centres in San Francisco and the Valley only weeks apart.
“The reason we are here is this is the centre of the Earth right now when it comes to innovation,” said Fay Arjomandi, head of research and development for Vodafone in the US, at the opening of its Xone Lab in Redwood City last month.
Apple’s iPhone and Google’s Android operating system have set new standards for hardware and software in the mobile industry.
The app culture that both have engendered means there is a race on among operators to feature the newest trends and advances in software first, amid fierce competition for the attention of Silicon Valley developers.
Vodafone’s lab allows developers to incubate their ideas and test how apps and services would perform on the worldwide networks it replicates at its facility.
“We want to bring to market these ideas and accelerate the process – the hope is that six to nine months after we prove something here, it can be in the hands of a user in one of our operating countries,” says Ms Arjomandi.
A week after the Vodafone opening, AT&T launched its Foundry innovation lab in an old French steam laundry in Palo Alto.
It too emphasised the need to move “at web developer rather than carrier speed” – to the extent that much of the office furniture is on wheels and is easily reconfigurable.
“This is part of our transformation – to move from being a telecoms company to a technology company,” said John Donovan, AT&T’s chief technology officer.
The operators have felt the need to change as technology companies such as Apple and Google, which is buying Motorola, have become more influential and handset makers have aligned themselves behind different operating systems.
“They have to show they are a part of the ecosystem, there’s been a big shift in the last year or so, when there has been the threat that carriers just become a dumb pipe [for data],” says Chris Jones, a telecoms analyst based in the Valley for the Canalys research firm.
Trevor Healy, chief executive of the mobile advertising platform Amobee, whose previous Valley company Jajah was bought by Spain’s Telefónica, has also noticed a change in attitude forced upon the operators.
“It’s not long ago that these companies were myopic and closed, but now there’s a carrot and stick for them in the shape of new revenue streams when their core revenues are stagnant and [they face] the stick of other big players like Google and Apple making ground,” he says.
The operators are arriving late to the Valley compared with infrastructure and handset players such as Ericsson and Nokia, who have had research centres here for years and base their chief technology officers in the Bay Area.
“The operators may have abdicated their positions with the operating systems and handsets, but now they’re putting a stake in the ground and focusing on app development, advertising and payments and billing,” says Mr Healy.
The consumer smartphone market may appear to have been carved up between Apple’s App Store and the Android Market, but operators feel they can still offer premium apps and services based on their network’s ability to know location, process payments, serve advertising, communicate with a range of devices and combine disparate services.
Mr Donovan boasted that the Foundry had 106 projects under way and that AT&T was now measuring itself on the openness of its network: 82 APIs (or access points into the network) had been launched over the past year.
Start-up demonstrations at both the AT&T and Vodafone opening events hinted at the “mash-ups” that were possible through mobile networks – for example, how an app recording running performances could link up with ones relaying sensor information on a person’s health.
Augmented reality applications, improved graphics for gaming and the ability for machines to talk to other machines – “the internet of things” – could all benefit from being hard-wired into the operators’ networks, the demonstrations proved.
It remains to be seen whether developers will be attracted in significant numbers to working with the operators, but Ms Arjomandi argues all parties can benefit.
“We would like to have the advantage of [perhaps one-year exclusivity] lead time to the market – to have a differentiation for us, we need to get that lead time,” she said.
“In exchange, we give them access to 340m customers around the world in 30-plus countries – so it’s definitely a win-win situation.”
Thursday, September 29, 2011
Big Data Take Center Stage at Health 2.0 Conference
George Lauer, iHealthBeat Contributing Editor September 29, 2011
SAN FRANCISCO -- If the Health 2.0 movement is perceived as an ongoing conversation, the first four years could be seen as determining who was going to be talking and what the means of communication was going to be. Now, in year five, the focus is turning to what, exactly, everybody's going to be talking about.
"Data is everything," said Health 2.0 cofounder Mathew Holt at the Fifth Annual Health 2.0 Conference this week.
Raw information -- the gathering of it, collating, crunching and ultimately putting it to work -- was a major theme of this year's event.
"If we see data in health care as this huge glacier, we really are just now coming to see the tip of the iceberg," said Indu Subaiya, cofounder with Holt. Together, in 2007, they launched the first Health 2.0 event aimed at sparking innovative ways to get consumers involved in actively managing their own health care. Between 400 and 500 people ("mostly techies and nerds," according to Holt) showed up for the first conference. This week, more than 1,500 people from several countries attended.
"We're getting to the stage that is kind of the holy grail of Health 2.0, where the rubber meets the road," Subaiya said. "This year at Health 2.0 we're really beginning to see the intelligent mining of big data that can truly change how the system works. It's all around us, this data, but we have so much more work to do to link it and make use of it."
The kind of data examined during the conference ranged from tiny sensors and devices to monitor heart rate and glucose levels to "Big Data" gathered by the government and huge corporations to analyze big-picture situations ranging from flu epidemics to payer and provider databases.
"Data at every level is going to be what drives things forward," Holt said.
"Almost everything we talk about here is being generated and displayed on the Internet, and we're only talking about a small, tiny fraction of what's out there," Holt added.
Government Embracing Health 2.0
What started as a grassroots effort to get patients involved with health IT has landed the federal government as a partner. Two HHS representatives participated in the Health 2.0 event this week -- Farzad Mostashari, national coordinator for health IT, and Lygeia Ricciardi, senior adviser for consumer eHealth at the Office of the National Coordinator for Health IT.
"We are embracing this with the passion of a recent convert," Mostashari said during a patient-centered panel presentation. "We come to this community with humility. This is not something that I've been at the forefront fighting for, but I really do believe we're on the right track."
Ricciardi urged attendees to check out the government's new offerings at HealthIT.gov, including a new educational campaign, "Putting the I in Health IT."
"We had this big formal launch in Washington two weeks ago as part of National Health IT Week," Ricciardi said. "One side (of the site's home page) is set up for providers and professionals and the other for patients and families. There's a whole body of information on health IT for the general public to understand it better," Ricciardi added.
Health 2.0 and ONC announced the launch of another in its series of challenges to health care technology innovators -- this one an invitation to multidisciplinary teams of developers to create a user-friendly way for patients to manage cardiovascular health using health IT. The project is part of the larger Health 2.0 Developer Challenge, a partnership between ONC and Health 2.0 designed to spur innovations in the use of technology to improve health care outcomes.
Earlier this month, ONC and Health 2.0 launched two competitions to encourage development of health IT applications to improve outcomes in patients transitioning from hospital to home and to facilitate reporting of adverse events related to medical devices.
'About To Change the World'
Although health care faces many obstacles in trying to bring the industry into the 21st century, the people and ideas involved in Health 2.0 are "on the verge of doing all the things that are about to change the world," according to Mark Smith, president and CEO of the California HealthCare Foundation and the conference's keynote speaker. CHCF publishes California Healthline.
Compared to other parts of modern life, such as banking and research, "in the trenches of American medical practice, we are still about 20 years behind," Smith said.
The good news, Smith said, is that technology is maturing and policy is evolving. The bad news is that time is running out in the race to fix an "enormously complex" health care system with "perverse incentives."
Smith pointed to Bear Stearns and Lehman Brothers, investment banks that folded in the credit crisis of 2008, as examples of "what happens when you run out of time."
"The fact of the matter is we're seeing lots of niceties in health policy swamped by fiscal realities," Smith said. "The amount of time we have is dwindling."
In November 2010, CHCF launched a $10 million fund to support health care innovation.
"We are interested in helping to stimulate innovations that can get some kind of market traction," Smith told a room full of entrepreneurs.
To illustrate the kinds of ideas that don't get traction, Smith showed a slide of the cartoon South Park's Underpants Business Model:
· Phase One -- Collect underpants
· Phase Two -- ?????
· Phase Three -- Profits!!!
"For a lot of health care ideas, that's really what the business model amounts to," Smith said. CHCF wants "to fund successful companies because we're tired of seeing good ideas dying on the vine."
The CHCF Health Innovation Fund invests in for-profit and not-for-profit companies and organizations, with a focus on start-ups. Investments range from seed grants of $50,000 to as much as $3 million over the life of a company or service.
Sampling of Launches, Rollouts
Dozens of new products, projects, partnerships and campaigns were announced during the two-day conference, including:
· Health eVillages, a partnership between the Robert F. Kennedy Center for Justice and Human Rights and Massachusetts-based Physicians Interactive Holdings. The project aims to get mobile phones and handheld devices to medical professionals in poor, remote and underserved parts of the world. So far, the organization has mounted pilot projects in Haiti, Uganda, Kenya and the U.S. Gulf Coast;
· Society for Participatory Medicine, which encourages patients to take an active role in their health care and encourages physicians to make that happen, announced a campaign to get 10,000 physicians enrolled in a program to recieve the Society Seal, proclaiming their compliance with three basic tenets of the organization -- providing patients with all medical data, convening a patient advisory group, soliciting input from patients and providing resources to patients; and
· Clarimed, billed as a "first-of-its-kind" health care rating organization, launched its online rating service. Calling itself the first independent rating agency for medical devices, diseases and health care products and services "in the gap that exists between scientific portals and medical websites for consumers," Clarimed was formed in April, launched a beta site in June and went live this week.
MORE ON THE WEB
· Health 2.0 San Francisco 2011
· HealthIT.gov
Read more: http://www.ihealthbeat.org/features/2011/data-take-center-stage-at-health-2-0-conference.aspx#ixzz1ZMzOTLWZ
SAN FRANCISCO -- If the Health 2.0 movement is perceived as an ongoing conversation, the first four years could be seen as determining who was going to be talking and what the means of communication was going to be. Now, in year five, the focus is turning to what, exactly, everybody's going to be talking about.
"Data is everything," said Health 2.0 cofounder Mathew Holt at the Fifth Annual Health 2.0 Conference this week.
Raw information -- the gathering of it, collating, crunching and ultimately putting it to work -- was a major theme of this year's event.
"If we see data in health care as this huge glacier, we really are just now coming to see the tip of the iceberg," said Indu Subaiya, cofounder with Holt. Together, in 2007, they launched the first Health 2.0 event aimed at sparking innovative ways to get consumers involved in actively managing their own health care. Between 400 and 500 people ("mostly techies and nerds," according to Holt) showed up for the first conference. This week, more than 1,500 people from several countries attended.
"We're getting to the stage that is kind of the holy grail of Health 2.0, where the rubber meets the road," Subaiya said. "This year at Health 2.0 we're really beginning to see the intelligent mining of big data that can truly change how the system works. It's all around us, this data, but we have so much more work to do to link it and make use of it."
The kind of data examined during the conference ranged from tiny sensors and devices to monitor heart rate and glucose levels to "Big Data" gathered by the government and huge corporations to analyze big-picture situations ranging from flu epidemics to payer and provider databases.
"Data at every level is going to be what drives things forward," Holt said.
"Almost everything we talk about here is being generated and displayed on the Internet, and we're only talking about a small, tiny fraction of what's out there," Holt added.
Government Embracing Health 2.0
What started as a grassroots effort to get patients involved with health IT has landed the federal government as a partner. Two HHS representatives participated in the Health 2.0 event this week -- Farzad Mostashari, national coordinator for health IT, and Lygeia Ricciardi, senior adviser for consumer eHealth at the Office of the National Coordinator for Health IT.
"We are embracing this with the passion of a recent convert," Mostashari said during a patient-centered panel presentation. "We come to this community with humility. This is not something that I've been at the forefront fighting for, but I really do believe we're on the right track."
Ricciardi urged attendees to check out the government's new offerings at HealthIT.gov, including a new educational campaign, "Putting the I in Health IT."
"We had this big formal launch in Washington two weeks ago as part of National Health IT Week," Ricciardi said. "One side (of the site's home page) is set up for providers and professionals and the other for patients and families. There's a whole body of information on health IT for the general public to understand it better," Ricciardi added.
Health 2.0 and ONC announced the launch of another in its series of challenges to health care technology innovators -- this one an invitation to multidisciplinary teams of developers to create a user-friendly way for patients to manage cardiovascular health using health IT. The project is part of the larger Health 2.0 Developer Challenge, a partnership between ONC and Health 2.0 designed to spur innovations in the use of technology to improve health care outcomes.
Earlier this month, ONC and Health 2.0 launched two competitions to encourage development of health IT applications to improve outcomes in patients transitioning from hospital to home and to facilitate reporting of adverse events related to medical devices.
'About To Change the World'
Although health care faces many obstacles in trying to bring the industry into the 21st century, the people and ideas involved in Health 2.0 are "on the verge of doing all the things that are about to change the world," according to Mark Smith, president and CEO of the California HealthCare Foundation and the conference's keynote speaker. CHCF publishes California Healthline.
Compared to other parts of modern life, such as banking and research, "in the trenches of American medical practice, we are still about 20 years behind," Smith said.
The good news, Smith said, is that technology is maturing and policy is evolving. The bad news is that time is running out in the race to fix an "enormously complex" health care system with "perverse incentives."
Smith pointed to Bear Stearns and Lehman Brothers, investment banks that folded in the credit crisis of 2008, as examples of "what happens when you run out of time."
"The fact of the matter is we're seeing lots of niceties in health policy swamped by fiscal realities," Smith said. "The amount of time we have is dwindling."
In November 2010, CHCF launched a $10 million fund to support health care innovation.
"We are interested in helping to stimulate innovations that can get some kind of market traction," Smith told a room full of entrepreneurs.
To illustrate the kinds of ideas that don't get traction, Smith showed a slide of the cartoon South Park's Underpants Business Model:
· Phase One -- Collect underpants
· Phase Two -- ?????
· Phase Three -- Profits!!!
"For a lot of health care ideas, that's really what the business model amounts to," Smith said. CHCF wants "to fund successful companies because we're tired of seeing good ideas dying on the vine."
The CHCF Health Innovation Fund invests in for-profit and not-for-profit companies and organizations, with a focus on start-ups. Investments range from seed grants of $50,000 to as much as $3 million over the life of a company or service.
Sampling of Launches, Rollouts
Dozens of new products, projects, partnerships and campaigns were announced during the two-day conference, including:
· Health eVillages, a partnership between the Robert F. Kennedy Center for Justice and Human Rights and Massachusetts-based Physicians Interactive Holdings. The project aims to get mobile phones and handheld devices to medical professionals in poor, remote and underserved parts of the world. So far, the organization has mounted pilot projects in Haiti, Uganda, Kenya and the U.S. Gulf Coast;
· Society for Participatory Medicine, which encourages patients to take an active role in their health care and encourages physicians to make that happen, announced a campaign to get 10,000 physicians enrolled in a program to recieve the Society Seal, proclaiming their compliance with three basic tenets of the organization -- providing patients with all medical data, convening a patient advisory group, soliciting input from patients and providing resources to patients; and
· Clarimed, billed as a "first-of-its-kind" health care rating organization, launched its online rating service. Calling itself the first independent rating agency for medical devices, diseases and health care products and services "in the gap that exists between scientific portals and medical websites for consumers," Clarimed was formed in April, launched a beta site in June and went live this week.
MORE ON THE WEB
· Health 2.0 San Francisco 2011
· HealthIT.gov
Read more: http://www.ihealthbeat.org/features/2011/data-take-center-stage-at-health-2-0-conference.aspx#ixzz1ZMzOTLWZ
Subscribe to:
Posts (Atom)