Steve Lohr The New York Times October 31, 2011, 11:24 am
Big data is, yes, about more data — the rising flood from corporate databases, Web browsing trails, sensors and social network communications. But it is just as much about speed.
If “big data” is more than a marketing term, it has to be the raw material for making smarter decisions, faster. And that means, as the big-data industry evolves, the need for groundbreaking new approaches to computing, both in hardware and software.
A simple example: the Watson question-answering computer that beat two human “Jeopardy!” champions earlier this year had to pore through vast quantities of data and come back with an answer in less than three seconds.
The speed requirement meant I.B.M.’s Watson had to do its near-instant data digging in memory instead of finding data on hard disks. Traditionally, memory chips surrounding the computer processor held the small amounts of data that had to be on hand for immediate tasks.
But getting answers quickly in the world of big data necessitates this new approach, called in-memory processing. “It’s a model for the future,” John E. Kelly, the head of I.B.M. research, said during an interview at Watson Labs.
Early signs of the move toward this new architecture can be seen in recent announcements of new computer appliances designed for high-speed data applications. Earlier this month, Oracle introduced its Exalytics “business intelligence machine,” an in-memory hardware and software system for faster data analysis.
In late September, both EMC’s Greenplum division and Teradata brought out computer appliances tailored for big data applications. Last year, I.B.M. paid $1.7 billion for Netezza, maker of an appliance for high-speed data analysis.
SAS Institute, the big-data software pioneer, announced last week that its software has been fine-tuned to work on the EMC Greenplum and Teradata high-speed appliances. The big private company had been working with Netezza as well, until I.B.M. bought the company. “We can’t share all our secrets with them,” Jim Goodnight, founder and chief executive of SAS, said in an interview.
Mr. Goodnight, 68, has been doing some of the research himself on software designed for in-memory processing and is a co-inventor on a few patents in that field.
Not today or tomorrow perhaps, but big data portends striking changes for the computer industry, predicted Mr. Kelly of I.B.M. “It’s going to break all of the technology we have and it’s where the next big arena of value is going to be in this industry,” he said.
In discussing the path over the next decade, Mr. Kelly talks of seeking inspiration, if not a blueprint, from biology, and a coming switch in chip-making from silicon to other materials, most likely carbon-based devices
Just what the computing framework may be is uncertain, but Mr. Kelly said, “It won’t be a Von Neumann architecture.”
The Von Neumann model, named after the mathematican John von Neumman has been the template for computing for six decades. It has a processor at the center, surrounded by memory and storage, and it is the basis of computing today. But the “Von Neumann bottleneck” describes the slowdown in performance that comes from that approach, with the processor tightly managing data input and output.
“We’re going to have to move from processor-centric computing to data and memory-centric computing with processors sprinkled in it,” Mr. Kelly said.
Monday, October 31, 2011
Tuesday, October 25, 2011
How big data will help manage a world of 7 billion people
Katie Fehrenbacher Gigom October 24, 2011,
By this time next week, the world will have 7 billion people in it, according to the United Nations, and by 2050, there are supposed to be 9 billion people in the world.
This rapid population growth will fundamentally change the way populations use resources like energy, water and food, and corporations, governments and NGOs will increasingly turn to analytics, software and big data tools to manage how to deliver these resources to the populations that need them.
Here are eight ways big data and analytics are already helping manage resources for a booming population:
1. SAP’s Population Demographics. In time for the 7 billion mark next week, SAP and the United Nations Population Fund created interactive maps that show the demographics of both an aging and a youthful population of 7 billion people. It only takes 13 years to add another 1 billion more people to the planet, according to the report. You can check out world stats, like birth rates, deaths, percentage of population by age, as well as gender.
2. Space-Time Insight. A startup called Space-Time Insight creates software that merges real-time geospatial data with Google Maps, and sells the software to utilities and gas and oil companies to manage their resources in real time. California’s Independent System Operator Corporation (Cal ISO) — which manages 85 percent of the state’s power load — has installed an 80-foot by 6.5-foot screen in its control room to display real-time power-grid data from thousands of endpoints. Cal ISO used to get the data in four-second intervals, but given the growth in resources, population and data feeds, it now gets updates by the millisecond.
3. The Climate Corporation. Formerly called WeatherBill, the now renamed Climate Corporation uses big data tools to offer analytics and reports to the agriculture industry, and also sells a weather insurance product to farmers, to help protect them from losses from extreme weather events. The world has seen a rise in extreme weather events, partly do to a change in climate, and farmers can expect more of this unpredictability going forward. Combined with more unpredictable weather, food prices will likely rise as the population grows and usable land becomes constrained, particularly in developing countries.
4. Google Earth Engine. Google launched Google Earth Engine, a year ago at COP 16, and the product combines an open API; a computing platform; and 25 years of satellite imagery available to researchers, scientists, organizations and government agencies. Google Earth Engine is interesting because it offers both tools and parallel processing computing power to groups to be able to use satellite imagery to analyze environmental conditions in order to make sustainability decisions. For example, the government of Mexico created the first comprehensive, high-resolution map of Mexico’s forests, using Google Earth Engine, incorporating 53,000 Landsat images, to produce a 6-gigabyte product. The Mexican government and NGOs can now use the map to make decisions about land use, sustainable agriculture, and species protection in combination with a growing population.
5. Google Oceans. Climate change and population growth will also affect how the world protects, uses and manages the oceans. In Google’s 5.0 version of its Google Earth tool, it included detailed ocean data (the ability for the user to dive beneath the surface) and “historical imagery” that features a time slider of satellite data for a location over time. (Below, Jimmy Buffet looking up at Google Oceans).
6. Building energy management. As more and more buildings are built to accommodate the growing population, the energy consumption of those buildings will need to be managed. A startup called FirstFuel Software use analytics and a set of data to remotely determine energy information about a commercial building, like consumption habits, and give recommendations for how to make the building run more energy efficiently. The company says it uses no on-site hardware or onsite energy audits, to get this data and yet says its information is as accurate as information received through an on-site energy audit, which is far more costly. The data sets needed to produce an accurate set of energy information includes, utility-based electric and gas data for a year, the location’s weather and climate data, as well as GIS mapped building data.
7. Curbing home energy. Other companies like Opower are focused on using analytics and behavioral tools to get consumers to change their energy consumption habits in homes. Opower recently announced plans to launch a Facebook application it says could one day be the world’s largest social network around energy.
8. Mobile phone data from settlements: A group of Harvard researchers is looking to use cell phone data combined with mathematical models and statistics, to better understand the needs of the 1 billion people who live in informal housing, called settlements or slums in developing countries. Other researchers in the group are looking to use big data to predict food shortages in developing countries and crime sprees due to causal events, like climate change.
Image courtesy of Smagdali.
By this time next week, the world will have 7 billion people in it, according to the United Nations, and by 2050, there are supposed to be 9 billion people in the world.
This rapid population growth will fundamentally change the way populations use resources like energy, water and food, and corporations, governments and NGOs will increasingly turn to analytics, software and big data tools to manage how to deliver these resources to the populations that need them.
Here are eight ways big data and analytics are already helping manage resources for a booming population:
1. SAP’s Population Demographics. In time for the 7 billion mark next week, SAP and the United Nations Population Fund created interactive maps that show the demographics of both an aging and a youthful population of 7 billion people. It only takes 13 years to add another 1 billion more people to the planet, according to the report. You can check out world stats, like birth rates, deaths, percentage of population by age, as well as gender.
2. Space-Time Insight. A startup called Space-Time Insight creates software that merges real-time geospatial data with Google Maps, and sells the software to utilities and gas and oil companies to manage their resources in real time. California’s Independent System Operator Corporation (Cal ISO) — which manages 85 percent of the state’s power load — has installed an 80-foot by 6.5-foot screen in its control room to display real-time power-grid data from thousands of endpoints. Cal ISO used to get the data in four-second intervals, but given the growth in resources, population and data feeds, it now gets updates by the millisecond.
3. The Climate Corporation. Formerly called WeatherBill, the now renamed Climate Corporation uses big data tools to offer analytics and reports to the agriculture industry, and also sells a weather insurance product to farmers, to help protect them from losses from extreme weather events. The world has seen a rise in extreme weather events, partly do to a change in climate, and farmers can expect more of this unpredictability going forward. Combined with more unpredictable weather, food prices will likely rise as the population grows and usable land becomes constrained, particularly in developing countries.
4. Google Earth Engine. Google launched Google Earth Engine, a year ago at COP 16, and the product combines an open API; a computing platform; and 25 years of satellite imagery available to researchers, scientists, organizations and government agencies. Google Earth Engine is interesting because it offers both tools and parallel processing computing power to groups to be able to use satellite imagery to analyze environmental conditions in order to make sustainability decisions. For example, the government of Mexico created the first comprehensive, high-resolution map of Mexico’s forests, using Google Earth Engine, incorporating 53,000 Landsat images, to produce a 6-gigabyte product. The Mexican government and NGOs can now use the map to make decisions about land use, sustainable agriculture, and species protection in combination with a growing population.
5. Google Oceans. Climate change and population growth will also affect how the world protects, uses and manages the oceans. In Google’s 5.0 version of its Google Earth tool, it included detailed ocean data (the ability for the user to dive beneath the surface) and “historical imagery” that features a time slider of satellite data for a location over time. (Below, Jimmy Buffet looking up at Google Oceans).
6. Building energy management. As more and more buildings are built to accommodate the growing population, the energy consumption of those buildings will need to be managed. A startup called FirstFuel Software use analytics and a set of data to remotely determine energy information about a commercial building, like consumption habits, and give recommendations for how to make the building run more energy efficiently. The company says it uses no on-site hardware or onsite energy audits, to get this data and yet says its information is as accurate as information received through an on-site energy audit, which is far more costly. The data sets needed to produce an accurate set of energy information includes, utility-based electric and gas data for a year, the location’s weather and climate data, as well as GIS mapped building data.
7. Curbing home energy. Other companies like Opower are focused on using analytics and behavioral tools to get consumers to change their energy consumption habits in homes. Opower recently announced plans to launch a Facebook application it says could one day be the world’s largest social network around energy.
8. Mobile phone data from settlements: A group of Harvard researchers is looking to use cell phone data combined with mathematical models and statistics, to better understand the needs of the 1 billion people who live in informal housing, called settlements or slums in developing countries. Other researchers in the group are looking to use big data to predict food shortages in developing countries and crime sprees due to causal events, like climate change.
Image courtesy of Smagdali.
80 Million Links A Day Don't Lie:
80 Million Links A Day Don't Lie: How Bitly Reveals The Web And The World
Erin Schulte Fast Company October 24, 2011
You might think of Bitly simply as a service that shortens links for your Twitter feed.
But to Hilary Mason, the company's chief scientist, Bitly is building a fresh new way to know what's going on in the world.
"We're able to learn things from the Bitly data set that people have never been able to share before," says Mason. "What is the half life of a link? How long will people be able to pay attention to it? What percent of links being shared on the web right now tend to be about the weather?"
The company--which shortens 80 million URLs every day--is working on a trove of new ways to mine that data. With its new search technology, for instance, it's studying and classifying every URL it shortens to index the most viral content on the web. The company recently rolled out a reputation monitoring service based on this data, which the company says acts as an "early warning system, designed to alert you in realtime to swings in volume and sentiment related to specific keywords," essentially allowing customers to monitor what people are and will be saying about products, brands, or any topic on social media.
Instead of a backward-looking view like you might get from a clipping service, Bitly is attempting to project where the conversation is headed. And unlike search--which might return, say, an official company website or a Wikipedia page--the Bitly service uncovers sites and conversations that haven't yet achieved a high pagerank (or any pagerank, for that matter).
"It's been fascinating to watch Bitly grow from a purely engineering orginzation into something where we're able to say, 'OK, there's still a lot of work that goes into that, but we're pretty good at that core piece of infrustructure--what can we build on top of it?'" says Mason. "Our products are starting to grow up and become adults."
Click play on the video below to learn more about Bitly's mission and what inspires Mason to keep innovating.
Erin Schulte Fast Company October 24, 2011
You might think of Bitly simply as a service that shortens links for your Twitter feed.
But to Hilary Mason, the company's chief scientist, Bitly is building a fresh new way to know what's going on in the world.
"We're able to learn things from the Bitly data set that people have never been able to share before," says Mason. "What is the half life of a link? How long will people be able to pay attention to it? What percent of links being shared on the web right now tend to be about the weather?"
The company--which shortens 80 million URLs every day--is working on a trove of new ways to mine that data. With its new search technology, for instance, it's studying and classifying every URL it shortens to index the most viral content on the web. The company recently rolled out a reputation monitoring service based on this data, which the company says acts as an "early warning system, designed to alert you in realtime to swings in volume and sentiment related to specific keywords," essentially allowing customers to monitor what people are and will be saying about products, brands, or any topic on social media.
Instead of a backward-looking view like you might get from a clipping service, Bitly is attempting to project where the conversation is headed. And unlike search--which might return, say, an official company website or a Wikipedia page--the Bitly service uncovers sites and conversations that haven't yet achieved a high pagerank (or any pagerank, for that matter).
"It's been fascinating to watch Bitly grow from a purely engineering orginzation into something where we're able to say, 'OK, there's still a lot of work that goes into that, but we're pretty good at that core piece of infrustructure--what can we build on top of it?'" says Mason. "Our products are starting to grow up and become adults."
Click play on the video below to learn more about Bitly's mission and what inspires Mason to keep innovating.
Monday, October 24, 2011
More Jobs Predicted for Machines, Not People
Steve Lohr The New York Times October 23, 2011
A faltering economy explains much of the job shortage in America, but advancing technology has sharply magnified the effect, more so than is generally understood, according to two researchers at the Massachusetts Institute of Technology.
The automation of more and more work once done by humans is the central theme of “Race Against the Machine,” an e-book to be published on Monday.
“Many workers, in short, are losing the race against the machine,” the authors write.
Erik Brynjolfsson, an economist and director of the M.I.T. Center for Digital Business, and Andrew P. McAfee, associate director and principal research scientist at the center, are two of the nation’s leading experts on technology and productivity. The tone of alarm in their book is a departure for the pair, whose previous research has focused mainly on the benefits of advancing technology.
Indeed, they were originally going to write a book titled, “The Digital Frontier,” about the “cornucopia of innovation that is going on,” Mr. McAfee said. Yet as the employment picture failed to brighten in the last two years, the two changed course to examine technology’s role in the jobless recovery.
The authors are not the only ones recently to point to the job fallout from technology. In the current issue of the McKinsey Quarterly, W. Brian Arthur, an external professor at the Santa Fe Institute, warns that technology is quickly taking over service jobs, following the waves of automation of farm and factory work. “This last repository of jobs is shrinking — fewer of us in the future may have white-collar business process jobs — and we have a problem,” Mr. Arthur writes.
The M.I.T. authors’ claim that automation is accelerating is not shared by some economists. Prominent among them are Robert J. Gordon of Northwestern and Tyler Cowen of George Mason University, who contend that productivity improvement owing to technological innovation rose from 1995 to 2004, but has trailed off since. Mr. Cowen emphasized that point in an e-book, “The Great Stagnation,” published this year.
Technology has always displaced some work and jobs. Over the years, many experts have warned — mistakenly — that machines were gaining the upper hand. In 1930, the economist John Maynard Keynes warned of a “new disease” that he termed “technological unemployment,” the inability of the economy to create new jobs faster than jobs were lost to automation.
But Mr. Brynjolfsson and Mr. McAfee argue that the pace of automation has picked up in recent years because of a combination of technologies including robotics, numerically controlled machines, computerized inventory control, voice recognition and online commerce.
Faster, cheaper computers and increasingly clever software, the authors say, are giving machines capabilities that were once thought to be distinctively human, like understanding speech, translating from one language to another and recognizing patterns. So automation is rapidly moving beyond factories to jobs in call centers, marketing and sales — parts of the services sector, which provides most jobs in the economy.
During the last recession, the authors write, one in 12 people in sales lost their jobs, for example. And the downturn prompted many businesses to look harder at substituting technology for people, if possible. Since the end of the recession in June 2009, they note, corporate spending on equipment and software has increased by 26 percent, while payrolls have been flat.
Corporations are doing fine. The companies in the Standard & Poor’s 500-stock index are expected to report record profits this year, a total $927 billion, estimates FactSet Research. And the authors point out that corporate profit as a share of the economy is at a 50-year high.
Productivity growth in the last decade, at more than 2.5 percent, they observe, is higher than the 1970s, 1980s and even edges out the 1990s. Still the economy, they write, did not add to its total job count, the first time that has happened over a decade since the Depression.
The skills of machines, the authors write, will only improve. In 2004, two leading economists, Frank Levy and Richard J. Murnane, published “The New Division of Labor,” which analyzed the capabilities of computers and human workers. Truck driving was cited as an example of the kind of work computers could not handle, recognizing and reacting to moving objects in real time.
But last fall, Google announced that its robot-driven cars had logged thousands of miles on American roads with only an occasional assist from human back-seat drivers. The Google cars, Mr. Brynjolfsson said, are but one sign of the times.
As others have, he pointed to I.B.M.’s “Jeopardy”-playing computer, Watson, which in February beat a pair of human “Jeopardy” champions; and Apple’s new personal assistant software, Siri, which responds to voice commands.
“This technology can do things now that only a few years ago were thought to be beyond the reach of computers,” Mr. Brynjolfsson said.
Yet computers, the authors say, tend to be narrow and literal-minded, good at assigned tasks but at a loss when a solution requires intuition and creativity — human traits. A partnership, they assert, is the path to job creation in the future.
“In medicine, law, finance, retailing, manufacturing and even scientific discovery,” they write, “the key to winning the race is not to competeagainst machines but to compete with machines.”
A faltering economy explains much of the job shortage in America, but advancing technology has sharply magnified the effect, more so than is generally understood, according to two researchers at the Massachusetts Institute of Technology.
The automation of more and more work once done by humans is the central theme of “Race Against the Machine,” an e-book to be published on Monday.
“Many workers, in short, are losing the race against the machine,” the authors write.
Erik Brynjolfsson, an economist and director of the M.I.T. Center for Digital Business, and Andrew P. McAfee, associate director and principal research scientist at the center, are two of the nation’s leading experts on technology and productivity. The tone of alarm in their book is a departure for the pair, whose previous research has focused mainly on the benefits of advancing technology.
Indeed, they were originally going to write a book titled, “The Digital Frontier,” about the “cornucopia of innovation that is going on,” Mr. McAfee said. Yet as the employment picture failed to brighten in the last two years, the two changed course to examine technology’s role in the jobless recovery.
The authors are not the only ones recently to point to the job fallout from technology. In the current issue of the McKinsey Quarterly, W. Brian Arthur, an external professor at the Santa Fe Institute, warns that technology is quickly taking over service jobs, following the waves of automation of farm and factory work. “This last repository of jobs is shrinking — fewer of us in the future may have white-collar business process jobs — and we have a problem,” Mr. Arthur writes.
The M.I.T. authors’ claim that automation is accelerating is not shared by some economists. Prominent among them are Robert J. Gordon of Northwestern and Tyler Cowen of George Mason University, who contend that productivity improvement owing to technological innovation rose from 1995 to 2004, but has trailed off since. Mr. Cowen emphasized that point in an e-book, “The Great Stagnation,” published this year.
Technology has always displaced some work and jobs. Over the years, many experts have warned — mistakenly — that machines were gaining the upper hand. In 1930, the economist John Maynard Keynes warned of a “new disease” that he termed “technological unemployment,” the inability of the economy to create new jobs faster than jobs were lost to automation.
But Mr. Brynjolfsson and Mr. McAfee argue that the pace of automation has picked up in recent years because of a combination of technologies including robotics, numerically controlled machines, computerized inventory control, voice recognition and online commerce.
Faster, cheaper computers and increasingly clever software, the authors say, are giving machines capabilities that were once thought to be distinctively human, like understanding speech, translating from one language to another and recognizing patterns. So automation is rapidly moving beyond factories to jobs in call centers, marketing and sales — parts of the services sector, which provides most jobs in the economy.
During the last recession, the authors write, one in 12 people in sales lost their jobs, for example. And the downturn prompted many businesses to look harder at substituting technology for people, if possible. Since the end of the recession in June 2009, they note, corporate spending on equipment and software has increased by 26 percent, while payrolls have been flat.
Corporations are doing fine. The companies in the Standard & Poor’s 500-stock index are expected to report record profits this year, a total $927 billion, estimates FactSet Research. And the authors point out that corporate profit as a share of the economy is at a 50-year high.
Productivity growth in the last decade, at more than 2.5 percent, they observe, is higher than the 1970s, 1980s and even edges out the 1990s. Still the economy, they write, did not add to its total job count, the first time that has happened over a decade since the Depression.
The skills of machines, the authors write, will only improve. In 2004, two leading economists, Frank Levy and Richard J. Murnane, published “The New Division of Labor,” which analyzed the capabilities of computers and human workers. Truck driving was cited as an example of the kind of work computers could not handle, recognizing and reacting to moving objects in real time.
But last fall, Google announced that its robot-driven cars had logged thousands of miles on American roads with only an occasional assist from human back-seat drivers. The Google cars, Mr. Brynjolfsson said, are but one sign of the times.
As others have, he pointed to I.B.M.’s “Jeopardy”-playing computer, Watson, which in February beat a pair of human “Jeopardy” champions; and Apple’s new personal assistant software, Siri, which responds to voice commands.
“This technology can do things now that only a few years ago were thought to be beyond the reach of computers,” Mr. Brynjolfsson said.
Yet computers, the authors say, tend to be narrow and literal-minded, good at assigned tasks but at a loss when a solution requires intuition and creativity — human traits. A partnership, they assert, is the path to job creation in the future.
“In medicine, law, finance, retailing, manufacturing and even scientific discovery,” they write, “the key to winning the race is not to competeagainst machines but to compete with machines.”
Monday, October 17, 2011
Jeff Jonas et al on Big Data
Big Data: Advancing the Art of Analytics
Federal NewsRadio Monday - 10/17/2011, 12:39pm ET
October 20th, 2010 at 11 AM
The application of knowledge discovery within the cloud is immensely powerful, but not inbuilt. We are collectively moving past the question of "what is cloud computing", and swiftly moving towards "how does the cloud enable advanced analysis against massive volumes of data?" With industry and government leveraging multiple clouds, how do we successfully share and search large collections of data across systems, departments, and geographies? Organizations will continue to discuss and better understand the analytic power and economies of cloud computing, in the sense of data storage, sharing, and management; but we are quickly discovering that creating knowledge from data is more than just a discussion of technology.
It's a discussion of what can be accomplished when massive data and cloud computing efficiencies combine to make advanced analysis and innovation possible.
Listen at http://www.federalnewsradio.com/?nid=240&sid=2080808
Panelists:
Michael Byrne- Geographic Information Officer, Federal Communications CommissionJeff Jonas- Chief Scientist, IBM Entity Analytics GroupDavid Mihelcic- Chief Technology Officer, Defense Information Systems AgencyChris Nissen- National Security Analysis Group, MITREMike Olson- Chief Executive Officer, Cloudera
Moderator: Chris Kelly - Senior Vice President, Booz Allen Hamilton
Federal NewsRadio Monday - 10/17/2011, 12:39pm ET
October 20th, 2010 at 11 AM
The application of knowledge discovery within the cloud is immensely powerful, but not inbuilt. We are collectively moving past the question of "what is cloud computing", and swiftly moving towards "how does the cloud enable advanced analysis against massive volumes of data?" With industry and government leveraging multiple clouds, how do we successfully share and search large collections of data across systems, departments, and geographies? Organizations will continue to discuss and better understand the analytic power and economies of cloud computing, in the sense of data storage, sharing, and management; but we are quickly discovering that creating knowledge from data is more than just a discussion of technology.
It's a discussion of what can be accomplished when massive data and cloud computing efficiencies combine to make advanced analysis and innovation possible.
Listen at http://www.federalnewsradio.com/?nid=240&sid=2080808
Panelists:
Michael Byrne- Geographic Information Officer, Federal Communications CommissionJeff Jonas- Chief Scientist, IBM Entity Analytics GroupDavid Mihelcic- Chief Technology Officer, Defense Information Systems AgencyChris Nissen- National Security Analysis Group, MITREMike Olson- Chief Executive Officer, Cloudera
Moderator: Chris Kelly - Senior Vice President, Booz Allen Hamilton
Tuesday, October 11, 2011
Big Data in the Dirt (and the Cloud)
Quentin Hardy The New York Times October 11, 2011, 12:01 am
Big data, the term for scanning loads of information for possibly profitable patterns, is a growing sector of corporate technology. Mostly people think in terms of online behavior, like mouse clicks, LinkedIn affiliations and Amazon shopping choices. But other big databases in the real world, lying around for years, are there to exploit.
A company called the Climate Corporation was formed in 2006 by two former Google employees who wanted to make use of the vast amount of free data published by the National Weather Service on heat and precipitation patterns around the country. At first they called the company WeatherBill, and used the data to sell insurance to businesses that depended heavily on the weather, from ski resorts and miniature golf courses to house painters and farmers.
It did pretty well, raising more than $50 million from the likes of Google Ventures, Khosla Ventures, and Allen & Company. The problem was, it was hard to sell insurance policies to so many little businesses, even using an online shopping model. People like having their insurance explained. The answer was to get even more data, and focus on the agriculture market through the same sales force that sells federal crop insurance.
“We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture,” says David Friedberg, chief executive of the Climate Corporation, a name WeatherBill started using Tuesday. “We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat.”
The product, insurance against things like drought, too much rain at the planting or the harvest, or an early freeze, is sold through 10,000 agents nationwide. The Climate Corporation, which also added Byron Dorgan, the former senator from North Dakota, to its board on Tuesday, will very likely get into insurance for specialty crops like tomatoes and grapes, which do not have federal insurance.
Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product. Mr. Friedberg was an early member of the corporate development team at Google. The co-founder, Siraj Khaliq, worked in distributed computing, which involves apportioning big data computing problems across multiple machines. He works as the Climate Corporation’s chief technical officer. Out of the staff of 60 in the company’s San Francisco office (another 30 work in the field) about 12 have doctorates, in areas like environmental science and applied mathematics.
“They like that this is a real-world problem, not just clicks on a Web site,” Mr. Friedberg says.
He figures that the Climate Corporation is one of the world’s largest users of MapReduce, an increasingly popular software technique for making sense of very large data systems. The number crunching is performed on Amazon.com’s Amazon Web Services computers.
The Climate Corporation is working with data intended to judge how different crops will react to certain soils, water and heat. It might be valuable to commodities traders as well, but Mr. Friedberg figures the better business is to expand in farming. Besides the other crops, he is looking at offering the service in Canada and Brazil, or anywhere else that he can get decent long-term data. It’s unlikely he’ll get the quality he got from the federal government, for a price anywhere near “free.”
Big data, the term for scanning loads of information for possibly profitable patterns, is a growing sector of corporate technology. Mostly people think in terms of online behavior, like mouse clicks, LinkedIn affiliations and Amazon shopping choices. But other big databases in the real world, lying around for years, are there to exploit.
A company called the Climate Corporation was formed in 2006 by two former Google employees who wanted to make use of the vast amount of free data published by the National Weather Service on heat and precipitation patterns around the country. At first they called the company WeatherBill, and used the data to sell insurance to businesses that depended heavily on the weather, from ski resorts and miniature golf courses to house painters and farmers.
It did pretty well, raising more than $50 million from the likes of Google Ventures, Khosla Ventures, and Allen & Company. The problem was, it was hard to sell insurance policies to so many little businesses, even using an online shopping model. People like having their insurance explained. The answer was to get even more data, and focus on the agriculture market through the same sales force that sells federal crop insurance.
“We took 60 years of crop yield data, and 14 terabytes of information on soil types, every two square miles for the United States, from the Department of Agriculture,” says David Friedberg, chief executive of the Climate Corporation, a name WeatherBill started using Tuesday. “We match that with the weather information for one million points the government scans with Doppler radar — this huge national infrastructure for storm warnings — and make predictions for the effect on corn, soybeans and winter wheat.”
The product, insurance against things like drought, too much rain at the planting or the harvest, or an early freeze, is sold through 10,000 agents nationwide. The Climate Corporation, which also added Byron Dorgan, the former senator from North Dakota, to its board on Tuesday, will very likely get into insurance for specialty crops like tomatoes and grapes, which do not have federal insurance.
Like the weather information, the data on soils was free for the taking. The hard and expensive part is turning the data into a product. Mr. Friedberg was an early member of the corporate development team at Google. The co-founder, Siraj Khaliq, worked in distributed computing, which involves apportioning big data computing problems across multiple machines. He works as the Climate Corporation’s chief technical officer. Out of the staff of 60 in the company’s San Francisco office (another 30 work in the field) about 12 have doctorates, in areas like environmental science and applied mathematics.
“They like that this is a real-world problem, not just clicks on a Web site,” Mr. Friedberg says.
He figures that the Climate Corporation is one of the world’s largest users of MapReduce, an increasingly popular software technique for making sense of very large data systems. The number crunching is performed on Amazon.com’s Amazon Web Services computers.
The Climate Corporation is working with data intended to judge how different crops will react to certain soils, water and heat. It might be valuable to commodities traders as well, but Mr. Friedberg figures the better business is to expand in farming. Besides the other crops, he is looking at offering the service in Canada and Brazil, or anywhere else that he can get decent long-term data. It’s unlikely he’ll get the quality he got from the federal government, for a price anywhere near “free.”
Monday, October 10, 2011
NYT on Big Data: Government Aims to Build a 'Data Eye in the Sky'
John Markoff The New York Times October 10, 2011
More than 60 years ago, in his “Foundation” series, the science fiction novelist Isaac Asimov invented a new science — psychohistory — that combined mathematics and psychology to predict the future.
Now social scientists are trying to mine the vast resources of the Internet — Web searches and Twitter messages, Facebook and blog posts, the digital location trails generated by billions of cellphones — to do the same thing.
The most optimistic researchers believe that these storehouses of “big data” will for the first time reveal sociological laws of human behavior — enabling them to predict political crises, revolutions and other forms of social and economic instability, just as physicists and chemists can predict natural phenomena.
“This is a significant step forward,” said Thomas Malone, the director of the Center for Collective Intelligence at the Massachusetts Institute of Technology. “We have vastly more detailed and richer kinds of data available as well as predictive algorithms to use, and that makes possible a kind of prediction that would have never been possible before.”
The government is showing interest in the idea. This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for “big data,” according to a research proposal being circulated by the agency. The three-year experiment, to begin in April, is being financed by the Intelligence Advanced Research Projects Activity, or Iarpa (pronounced eye-AR-puh), part of the office of the director of national intelligence.
The automated data collection system is to focus on patterns of communication, consumption and movement of populations. It will use publicly accessible data, including Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries.
It is intended to be an entirely automated system, a “data eye in the sky” without human intervention, according to the program proposal. The research would not be limited to political and economic events, but would also explore the ability to predict pandemics and other types of widespread contagion, something that has been pursued independently by civilian researchers and by companies like Google.
Some social scientists and advocates of privacy rights are deeply skeptical of the project, saying it evokes queasy memories of Total Information Awareness, a post-9/11 Pentagon program that proposed hunting for potential attackers by identifying patterns in vast collections of public and private data: telephone calling records, e-mail, travel data, visa and passport information, and credit card transactions.
“I have Total Information Awareness flashbacks when things like this happen,” said David Price, an anthropologist at St. Martin’s University in Lacey, Wash., who has written about cooperation between social scientists and intelligence agencies. “On the one hand it’s understandable for a nation-state to want to track things like the outbreak of a pandemic, but I have to wonder about the total automation of this and what productive will come of it.”
Iarpa officials declined to discuss the research program, saying they are prohibited from giving interviews until contract awards are made later this year.
A similar project by their military sister organization, the Defense Advanced Research Projects Agency, or Darpa, aims to automatically identify insurgent social networks in Afghanistan.
In its most recent budget proposal, the defense agency argues that its analysis can expose terrorist cells and other stateless groups by tracking their meetings, rehearsals and sharing of material and money transfers.
So far there have been only scattered examples of the potential of mining social media. Last year HP Labs researchers used Twitter data to accurately predict box office revenues of Hollywood movies. In August, the National Science Foundation approved funds for research in using social media like Twitter and Facebook to assess earthquake damage in real time.
The accessibility and computerization of huge databases has already begun to spur the development of new statistical techniques and new software to manage data sets with trillions of entries or more.
“Big data allows one to move beyond inference and statistical significance and move toward meaningful and accurate analyses,” said Norman Nie, a political scientist who was a pioneering developer of statistical tools for social scientists and who recently formed a new company, Revolution Analytics, to develop software for the analysis of immense data sets.
Some scientists are skeptical. They cite the Pentagon’s ill-fated Project Camelot in the 1960s, which also explored the possibility that social science could predict political and economic events, but was canceled in the face of widespread criticism by scholars.
The project focused on Chile, with the goal of developing methods for anticipating “violent changes” and offering ways of averting possible rebellions. It led to an uproar among social scientists, who argued that the study would compromise their professional ethics.
In recent years, however, academic opposition to military financing of research has faded. Since 2008, a Pentagon project called the Minerva Initiative has paid for an array of studies, including research at Arizona State University into political opponents of radical Muslims and a University of Texas study on the effects of climate change on African political stability.
Social scientists who cooperate with the research agencies contend that, on balance, the new technologies will have a positive effect.
“The result will be much better understanding of what is going on in the world, and how well local governments are handling the situation,” said Sandy Pentland, a computer scientist at the M.I.T. Media Laboratory. “I find this all very hopeful rather than scary, because this is perhaps the first real opportunity for all of humanity to have transparency in government.”
But advocates of privacy rights worry that public data and the related techniques developed in the new Iarpa project will be adapted for clandestine “total information” operations.
“These techniques are double-edged,” said Marc Rotenberg, president of the Electronic Privacy Information Center, a privacy rights group based in Washington. “They can be used as easily against political opponents in the United States as they can against threats from foreign countries.”
And some computer scientists expressed skepticism about efforts to predict political instability with indicators like Web searches.
“I’m hard pressed to say that we are witnessing a revolution,” said Prabhakar Raghavan, the director of Yahoo Labs, who is an information retrieval specialist. He noted that much had been written about predicting flu epidemics by looking at Web searches for “flu,” but noted that the predictions did not improve significantly on what could already be found in data from the Centers for Disease Control and Prevention.
“You can look at search queries and divine that flu is about to break out,” he said, “but what our research has highlighted is that many of these new methods don’t add a huge lift.”
Other researchers are far more optimistic. “There is a huge amount of predictive power in this data,” said Albert-Laszlo Barabasi, a physicist at Notre Dame who specializes in network science. “If I have hourly information about your location, with about 93 percent accuracy I can predict where you are going to be an hour or a day later.”
Still, the ease of acquiring and manipulating huge data sets charting Internet behavior causes many researchers to warn that the data mining technologies may be quickly outrunning the ability of scientists to think through questions of privacy and ethics.
There is also the deeper question of whether it will be possible to discern behavioral laws that match the laws of physical sciences. For Isaac Asimov, the predictive powers of psychohistory worked only when it was possible to measure the human population of an entire galaxy.
More than 60 years ago, in his “Foundation” series, the science fiction novelist Isaac Asimov invented a new science — psychohistory — that combined mathematics and psychology to predict the future.
Now social scientists are trying to mine the vast resources of the Internet — Web searches and Twitter messages, Facebook and blog posts, the digital location trails generated by billions of cellphones — to do the same thing.
The most optimistic researchers believe that these storehouses of “big data” will for the first time reveal sociological laws of human behavior — enabling them to predict political crises, revolutions and other forms of social and economic instability, just as physicists and chemists can predict natural phenomena.
“This is a significant step forward,” said Thomas Malone, the director of the Center for Collective Intelligence at the Massachusetts Institute of Technology. “We have vastly more detailed and richer kinds of data available as well as predictive algorithms to use, and that makes possible a kind of prediction that would have never been possible before.”
The government is showing interest in the idea. This summer a little-known intelligence agency began seeking ideas from academic social scientists and corporations for ways to automatically scan the Internet in 21 Latin American countries for “big data,” according to a research proposal being circulated by the agency. The three-year experiment, to begin in April, is being financed by the Intelligence Advanced Research Projects Activity, or Iarpa (pronounced eye-AR-puh), part of the office of the director of national intelligence.
The automated data collection system is to focus on patterns of communication, consumption and movement of populations. It will use publicly accessible data, including Web search queries, blog entries, Internet traffic flow, financial market indicators, traffic webcams and changes in Wikipedia entries.
It is intended to be an entirely automated system, a “data eye in the sky” without human intervention, according to the program proposal. The research would not be limited to political and economic events, but would also explore the ability to predict pandemics and other types of widespread contagion, something that has been pursued independently by civilian researchers and by companies like Google.
Some social scientists and advocates of privacy rights are deeply skeptical of the project, saying it evokes queasy memories of Total Information Awareness, a post-9/11 Pentagon program that proposed hunting for potential attackers by identifying patterns in vast collections of public and private data: telephone calling records, e-mail, travel data, visa and passport information, and credit card transactions.
“I have Total Information Awareness flashbacks when things like this happen,” said David Price, an anthropologist at St. Martin’s University in Lacey, Wash., who has written about cooperation between social scientists and intelligence agencies. “On the one hand it’s understandable for a nation-state to want to track things like the outbreak of a pandemic, but I have to wonder about the total automation of this and what productive will come of it.”
Iarpa officials declined to discuss the research program, saying they are prohibited from giving interviews until contract awards are made later this year.
A similar project by their military sister organization, the Defense Advanced Research Projects Agency, or Darpa, aims to automatically identify insurgent social networks in Afghanistan.
In its most recent budget proposal, the defense agency argues that its analysis can expose terrorist cells and other stateless groups by tracking their meetings, rehearsals and sharing of material and money transfers.
So far there have been only scattered examples of the potential of mining social media. Last year HP Labs researchers used Twitter data to accurately predict box office revenues of Hollywood movies. In August, the National Science Foundation approved funds for research in using social media like Twitter and Facebook to assess earthquake damage in real time.
The accessibility and computerization of huge databases has already begun to spur the development of new statistical techniques and new software to manage data sets with trillions of entries or more.
“Big data allows one to move beyond inference and statistical significance and move toward meaningful and accurate analyses,” said Norman Nie, a political scientist who was a pioneering developer of statistical tools for social scientists and who recently formed a new company, Revolution Analytics, to develop software for the analysis of immense data sets.
Some scientists are skeptical. They cite the Pentagon’s ill-fated Project Camelot in the 1960s, which also explored the possibility that social science could predict political and economic events, but was canceled in the face of widespread criticism by scholars.
The project focused on Chile, with the goal of developing methods for anticipating “violent changes” and offering ways of averting possible rebellions. It led to an uproar among social scientists, who argued that the study would compromise their professional ethics.
In recent years, however, academic opposition to military financing of research has faded. Since 2008, a Pentagon project called the Minerva Initiative has paid for an array of studies, including research at Arizona State University into political opponents of radical Muslims and a University of Texas study on the effects of climate change on African political stability.
Social scientists who cooperate with the research agencies contend that, on balance, the new technologies will have a positive effect.
“The result will be much better understanding of what is going on in the world, and how well local governments are handling the situation,” said Sandy Pentland, a computer scientist at the M.I.T. Media Laboratory. “I find this all very hopeful rather than scary, because this is perhaps the first real opportunity for all of humanity to have transparency in government.”
But advocates of privacy rights worry that public data and the related techniques developed in the new Iarpa project will be adapted for clandestine “total information” operations.
“These techniques are double-edged,” said Marc Rotenberg, president of the Electronic Privacy Information Center, a privacy rights group based in Washington. “They can be used as easily against political opponents in the United States as they can against threats from foreign countries.”
And some computer scientists expressed skepticism about efforts to predict political instability with indicators like Web searches.
“I’m hard pressed to say that we are witnessing a revolution,” said Prabhakar Raghavan, the director of Yahoo Labs, who is an information retrieval specialist. He noted that much had been written about predicting flu epidemics by looking at Web searches for “flu,” but noted that the predictions did not improve significantly on what could already be found in data from the Centers for Disease Control and Prevention.
“You can look at search queries and divine that flu is about to break out,” he said, “but what our research has highlighted is that many of these new methods don’t add a huge lift.”
Other researchers are far more optimistic. “There is a huge amount of predictive power in this data,” said Albert-Laszlo Barabasi, a physicist at Notre Dame who specializes in network science. “If I have hourly information about your location, with about 93 percent accuracy I can predict where you are going to be an hour or a day later.”
Still, the ease of acquiring and manipulating huge data sets charting Internet behavior causes many researchers to warn that the data mining technologies may be quickly outrunning the ability of scientists to think through questions of privacy and ethics.
There is also the deeper question of whether it will be possible to discern behavioral laws that match the laws of physical sciences. For Isaac Asimov, the predictive powers of psychohistory worked only when it was possible to measure the human population of an entire galaxy.
Sunday, October 9, 2011
New Report: The open Internet - Platform for Growth
Access to internet-based content, applications and services has risen up the policy agenda because such applications are recognised to be of growing importance to citizens, consumers, public services, the economy and society; and because some network access providers have engaged in harmful blocking and discrimination against such applications.
It has also been argued by some European network operators that demand and associated traffic growth is a problem rather than an opportunity and that policy makers should support the introduction of new charges in relation to content and application providers.
The question of how the open internet is governed is one of central economic and social importance.
For this reason a group of content, application and service providers commissioned Plum to consider the question of how value is created and distributed along the value chain and what form of governance should apply in order to sustain innovation and investment along the value chain, to the benefit of all.
Our aim in producing this report has been to bring clarity to the issues - to make them as simple as possible but no simpler - and to provide a clear sense of direction for those charged with policy making and policy implementation, without being overly prescriptive given the pace of change and innovation which characterises the market.
It has also been argued by some European network operators that demand and associated traffic growth is a problem rather than an opportunity and that policy makers should support the introduction of new charges in relation to content and application providers.
The question of how the open internet is governed is one of central economic and social importance.
For this reason a group of content, application and service providers commissioned Plum to consider the question of how value is created and distributed along the value chain and what form of governance should apply in order to sustain innovation and investment along the value chain, to the benefit of all.
Our aim in producing this report has been to bring clarity to the issues - to make them as simple as possible but no simpler - and to provide a clear sense of direction for those charged with policy making and policy implementation, without being overly prescriptive given the pace of change and innovation which characterises the market.
WashPost: Innovators, data will set you free
Dominic Basulto The Washington Post October 7, 2011
The pace of data creation on the Internet is growing at an exponential rate. In any 48-hour period in 2010, more data was created than had been created by all of humanity in the past 30,000 years, according to a presentation by entrepreneur Yuri Milner. By the year 2020, that same amount of data will be created in a single hour.
“Information wants to be free,” declared writer Stewart Brand,
Data will set you free.
Digital Libertarianism seems like a radical concept until you consider the fact that - across many industries - regulators are no longer able to keep up with all the data that is being created every day. The most obvious example is the financial markets, where even traders and investors talk about the creation of a vast new "shadow market" where financial instruments are just too complex to regulate and algorithmic traders, using powerful supercomputers to analyze all of the data, can yank the market down in a matter of seconds. Viewed in this way, is it actually possible that it would be in a financial institution’s competitive best-interest to release as much data as possible into the marketplace, in exchange for freedom from regulatory shackles?
How can this be the case? Isn’t proprietary data the key to competitive success in the markets? Not always — consider all of the open innovation initiatives formed to solve the world’s most difficult problems in fields like biology and chemistry and physics. Large corporations have found that full data disclosure, offered in good faith and made available to researchers, can be a way to accelerate growth — especially if the crowd gets involved. It can also be a brake on business practices that harm society. Take taxes, for example. Full data disclosure can also solve for problems like the propensity of individuals and corporations to play fast and loose with their taxes. Governments have always struggled to collect the full tax burden from individuals and corporations.
Consider your own experience on a site like Facebook. The more information and data that you make available to others, the more that you are held accountable to others. On the flip side, all this data leads to new types of experiences, like the ability for corporations to perfectly customize their products to your tastes and needs. Now that Facebook has created the Timeline, people will know exactly what you read, where you were last night, what music you listened to and what Web site you think has the best “LOL kittens.” Your personal life just became public. Imagine if corporations had to disclose as much data about everything they did.
As Supreme Court Justice Louis Brandeis once said: “Sunlight is the best disinfectant“? Perhaps that could be the case here as well.
The idea of making data completely open and transparent is gaining traction in tech circles. Consider New York City's recent overture to the tech crowd, opening up all of its data about the city to developers and entrepreneurs. Government officials could easily hide information about crime reports, public health issues, building complaints and traffic reports. But what if this information could become the stimulus for new innovation by the crowd to solve problems that the government could not?
An increase in the amount of data made freely available to all could lead to governments and businesses doing more good for society. That may sound hopelessly idealistic, but so was the original premise of the Web. As participants in a democracy, we have rights as well as responsibilities. In an age of Wikileaks and forced data transparency, one new responsibility might be to take more than a passing interest in all of the data being created every 48 hours, and instead, use it to hold our society’s institutions fully accountable.
and so does all of this data. In fact, the big data movement — organizations scrambling to make sense of all this data — may be the single most profound trend that the Web has created in the past 12 months. It has also given rise to a group of Digital Libertarians who see the beginning of a new era in which total data transparency makes it possible to regulate the activity of corporations, governments and individuals.
The pace of data creation on the Internet is growing at an exponential rate. In any 48-hour period in 2010, more data was created than had been created by all of humanity in the past 30,000 years, according to a presentation by entrepreneur Yuri Milner. By the year 2020, that same amount of data will be created in a single hour.
“Information wants to be free,” declared writer Stewart Brand,
Data will set you free.
Digital Libertarianism seems like a radical concept until you consider the fact that - across many industries - regulators are no longer able to keep up with all the data that is being created every day. The most obvious example is the financial markets, where even traders and investors talk about the creation of a vast new "shadow market" where financial instruments are just too complex to regulate and algorithmic traders, using powerful supercomputers to analyze all of the data, can yank the market down in a matter of seconds. Viewed in this way, is it actually possible that it would be in a financial institution’s competitive best-interest to release as much data as possible into the marketplace, in exchange for freedom from regulatory shackles?
How can this be the case? Isn’t proprietary data the key to competitive success in the markets? Not always — consider all of the open innovation initiatives formed to solve the world’s most difficult problems in fields like biology and chemistry and physics. Large corporations have found that full data disclosure, offered in good faith and made available to researchers, can be a way to accelerate growth — especially if the crowd gets involved. It can also be a brake on business practices that harm society. Take taxes, for example. Full data disclosure can also solve for problems like the propensity of individuals and corporations to play fast and loose with their taxes. Governments have always struggled to collect the full tax burden from individuals and corporations.
Consider your own experience on a site like Facebook. The more information and data that you make available to others, the more that you are held accountable to others. On the flip side, all this data leads to new types of experiences, like the ability for corporations to perfectly customize their products to your tastes and needs. Now that Facebook has created the Timeline, people will know exactly what you read, where you were last night, what music you listened to and what Web site you think has the best “LOL kittens.” Your personal life just became public. Imagine if corporations had to disclose as much data about everything they did.
As Supreme Court Justice Louis Brandeis once said: “Sunlight is the best disinfectant“? Perhaps that could be the case here as well.
The idea of making data completely open and transparent is gaining traction in tech circles. Consider New York City's recent overture to the tech crowd, opening up all of its data about the city to developers and entrepreneurs. Government officials could easily hide information about crime reports, public health issues, building complaints and traffic reports. But what if this information could become the stimulus for new innovation by the crowd to solve problems that the government could not?
An increase in the amount of data made freely available to all could lead to governments and businesses doing more good for society. That may sound hopelessly idealistic, but so was the original premise of the Web. As participants in a democracy, we have rights as well as responsibilities. In an age of Wikileaks and forced data transparency, one new responsibility might be to take more than a passing interest in all of the data being created every 48 hours, and instead, use it to hold our society’s institutions fully accountable.
and so does all of this data. In fact, the big data movement — organizations scrambling to make sense of all this data — may be the single most profound trend that the Web has created in the past 12 months. It has also given rise to a group of Digital Libertarians who see the beginning of a new era in which total data transparency makes it possible to regulate the activity of corporations, governments and individuals.
Thursday, October 6, 2011
Brian Arthur: Contemplate an economy run entirely by machines
The second economy
Digitization is creating a second economy that’s vast, automatic, and invisible—thereby bringing the biggest change since the Industrial Revolution.
W. Brian Arthur McKinsey Quarterly October 2011
In This Article
· Sidebar: How fast is the second economy growing?
· About the author
In 1850, a decade before the Civil War, the United States’ economy was small—it wasn’t much bigger than Italy’s. Forty years later, it was the largest economy in the world. What happened in-between was the railroads. They linked the east of the country to the west, and the interior to both. They gave access to the east’s industrial goods; they made possible economies of scale; they stimulated steel and manufacturing—and the economy was never the same.
Deep changes like this are not unusual. Every so often—every 60 years or so—a body of technology comes along and over several decades, quietly, almost unnoticeably, transforms the economy: it brings new social classes to the fore and creates a different world for business. Can such a transformation—deep and slow and silent—be happening today
We could look for one in the genetic technologies, or in nanotech, but their time hasn’t fully come. But I want to argue that something deep is going on with information technology, something that goes well beyond the use of computers, social media, and commerce on the Internet. Business processes that once took place among human beings are now being executed electronically. They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential—it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.
Let me begin with two examples. Twenty years ago, if you went into an airport you would walk up to a counter and present paper tickets to a human being. That person would register you on a computer, notify the flight you’d arrived, and check your luggage in. All this was done by humans. Today, you walk into an airport and look for a machine. You put in a frequent-flier card or credit card, and it takes just three or four seconds to get back a boarding pass, receipt, and luggage tag. What interests me is what happens in those three or four seconds. The moment the card goes in, you are starting a huge conversation conducted entirely among machines. Once your name is recognized, computers are checking your flight status with the airlines, your past travel history, your name with the TSA1 (and possibly also with the National Security Agency). They are checking your seat choice, your frequent-flier status, and your access to lounges. This unseen, underground conversation is happening among multiple servers talking to other servers, talking to satellites that are talking to computers (possibly in London, where you’re going), and checking with passport control, with foreign immigration, with ongoing connecting flights. And to make sure the aircraft’s weight distribution is fine, the machines are also starting to adjust the passenger count and seating according to whether the fuselage is loaded more heavily at the front or back.
These large and fairly complicated conversations that you’ve triggered occur entirely among things remotely talking to other things: servers, switches, routers, and other Internet and telecommunications devices, updating and shuttling information back and forth. All of this occurs in the few seconds it takes to get your boarding pass back. And even after that happens, if you could see these conversations as flashing lights, they’d still be flashing all over the country for some time, perhaps talking to the flight controllers—starting to say that the flight’s getting ready for departure and to prepare for that.
Now consider a second example, from supply chain management. Twenty years ago, if you were shipping freight through Rotterdam into the center of Europe, people with clipboards would be registering arrival, checking manifests, filling out paperwork, and telephoning forward destinations to let other people know. Now such shipments go through an RFID2 portal where they are scanned, digitally captured, and automatically dispatched. The RFID portal is in conversation digitally with the originating shipper, other depots, other suppliers, and destinations along the route, all keeping track, keeping control, and reconfiguring routing if necessary to optimize things along the way. What used to be done by humans is now executed as a series of conversations among remotely located servers.
In both these examples, and all across economies in the developed world, processes in the physical economy are being entered into the digital economy, where they are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.
So we can say that another economy—a second economy—of all of these digitized business processes conversing, executing, and triggering further actions is silently forming alongside the physical economy.
Aspen root systems
If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent—a great computer expression—which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.
These last descriptors sound biological—and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, there’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.
The metaphor isn’t perfect: this emerging second-economy root system is more complicated than any aspen system, since it’s also making new connections and new configurations on the fly. But the aspen metaphor is useful for capturing the reality that the observable physical world of aspen trees hides an unseen underground root system just as large or even larger.
How large is the unseen second economy? By a rough back-of-the-envelope calculation (see sidebar, “How fast is the second economy growing?”), in about two decades the digital economy will reach the same size as the physical economy. It’s as if there will be another American economy anchored off San Francisco (or, more in keeping with my metaphor, slipped in underneath the original economy) and growing all the while.
Now this second, digital economy isn’t producing anything tangible. It’s not making my bed in a hotel, or bringing me orange juice in the morning. But it is running an awful lot of the economy. It’s helping architects design buildings, it’s tracking sales and inventory, getting goods from here to there, executing trades and banking operations, controlling manufacturing equipment, making design calculations, billing clients, navigating aircraft, helping diagnose patients, and guiding laparoscopic surgeries. Such operations grow slowly and take time to form. In any deep transformation, industries do not so much adopt the new body of technology as encounter it, and as they do so they create new ways to profit from its possibilities.
The deep transformation I am describing is happening not just in the United States but in all advanced economies, especially in Europe and Japan. And its revolutionary scale can only be grasped if we go beyond my aspen metaphor to another analogy.
A neural system for the economy
Recall that in the digital conversations I was describing, something that occurs in the physical economy is sensed by the second economy—which then gives back an appropriate response. A truck passes its load through an RFID sensor or you check in at the airport, a lot of recomputation takes place, and appropriate physical actions are triggered.
There’s a parallel in this with how biologists think of intelligence. I’m not talking about human intelligence or anything that would qualify as conscious intelligence. Biologists tell us that an organism is intelligent if it senses something, changes its internal state, and reacts appropriately. If you put an E. coli bacterium into an uneven concentration of glucose, it does the appropriate thing by swimming toward where the glucose is more concentrated. Biologists would call this intelligent behavior. The bacterium senses something, “computes” something (although we may not know exactly how), and returns an appropriate response.
No brain need be involved. A primitive jellyfish doesn’t have a central nervous system or brain. What it has is a kind of neural layer or nerve net that lets it sense and react appropriately. I’m arguing that all these aspen roots—this vast global digital network that is sensing, “computing,” and reacting appropriately—is starting to constitute a neural layer for the economy. The second economy constitutes a neural layer for the physical economy. Just what sort of change is this qualitatively?
Think of it this way. With the coming of the Industrial Revolution—roughly from the 1760s, when Watt’s steam engine appeared, through around 1850 and beyond—the economy developed a muscular system in the form of machine power. Now it is developing a neural system. This may sound grandiose, but actually I think the metaphor is valid. Around 1990, computers started seriously to talk to each other, and all these connections started to happen. The individual machines—servers—are like neurons, and the axons and synapses are the communication pathways and linkages that enable them to be in conversation with each other and to take appropriate action.
Is this the biggest change since the Industrial Revolution? Well, without sticking my neck out too much, I believe so. In fact, I think it may well be the biggest change ever in the economy. It is a deep qualitative change that is bringing intelligent, automatic response to the economy. There’s no upper limit to this, no place where it has to end. Now, I’m not interested in science fiction, or predicting the singularity, or talking about cyborgs. None of that interests me. What I am saying is that it would be easy to underestimate the degree to which this is going to make a difference.
I think that for the rest of this century, barring wars and pestilence, a lot of the story will be the building out of this second economy, an unseen underground economy that basically is giving us intelligent reactions to what we do above the ground. For example, if I’m driving in Los Angeles in 15 years’ time, likely it’ll be a driverless car in a flow of traffic where my car’s in a conversation with the cars around it that are in conversation with general traffic and with my car. The second economy is creating for us—slowly, quietly, and steadily—a different world.
A downside
Of course, as with most changes, there is a downside. I am concerned that there is an adverse impact on jobs. Productivity increasing, say, at 2.4 percent in a given year means either that the same number of people can produce 2.4 percent more output or that we can get the same output with 2.4 percent fewer people. Both of these are happening. We are getting more output for each person in the economy, but overall output, nationally, requires fewer people to produce it. Nowadays, fewer people are required behind the desk of an airline. Much of the work is still physical—someone still has to take your luggage and put it on the belt—but much has vanished into the digital world of sensing, digital communication, and intelligent response.
Physical jobs are disappearing into the second economy, and I believe this effect is dwarfing the much more publicized effect of jobs disappearing to places like India and China.
There are parallels with what has happened before. In the early 20th century, farm jobs became mechanized and there was less need for farm labor, and some decades later manufacturing jobs became mechanized and there was less need for factory labor. Now business processes—many in the service sector—are becoming “mechanized” and fewer people are needed, and this is exerting systematic downward pressure on jobs. We don’t have paralegals in the numbers we used to. Or draftsmen, telephone operators, typists, or bookkeeping people. A lot of that work is now done digitally. We do have police and teachers and doctors; where there’s a need for human judgment and human interaction, we still have that. But the primary cause of all of the downsizing we’ve had since the mid-1990s is that a lot of human jobs are disappearing into the second economy. Not to reappear.
Seeing things this way, it’s not surprising we are still working our way out of the bad 2008–09 recession with a great deal of joblessness.
There’s a larger lesson to be drawn from this. The second economy will certainly be the engine of growth and the provider of prosperity for the rest of this century and beyond, but it may not provide jobs, so there may be prosperity without full access for many. This suggests to me that the main challenge of the economy is shifting from producing prosperity to distributing prosperity. The second economy will produce wealth no matter what we do; distributing that wealth has become the main problem. For centuries, wealth has traditionally been apportioned in the West through jobs, and jobs have always been forthcoming. When farm jobs disappeared, we still had manufacturing jobs, and when these disappeared we migrated to service jobs. With this digital transformation, this last repository of jobs is shrinking—fewer of us in the future may have white-collar business process jobs—and we face a problem.
The system will adjust of course, though I can’t yet say exactly how. Perhaps some new part of the economy will come forward and generate a whole new set of jobs. Perhaps we will have short workweeks and long vacations so there will be more jobs to go around. Perhaps we will have to subsidize job creation. Perhaps the very idea of a job and of being productive will change over the next two or three decades. The problem is by no means insoluble. The good news is that if we do solve it we may at last have the freedom to invest our energies in creative acts.
Economic possibilities for our grandchildren
In 1930, Keynes wrote a famous essay, “Economic possibilities for our grandchildren.” Reading it now, in the era of those grandchildren, I am surprised just how accurate it is. Keynes predicts that “the standard of life in progressive countries one hundred years hence will be between four and eight times as high as it is to-day.” He rightly warns of “technological unemployment,” but dares to surmise that “the economic problem [of producing enough goods] may be solved.” If we had asked him and his contemporaries how all this might come about, they might have imagined lots of factories with lots of machines, possibly even with robots, with the workers in these factories gradually being replaced by machines and by individual robots.
That is not quite how things have developed. We do have sophisticated machines, but in the place of personal automation (robots) we have a collective automation. Underneath the physical economy, with its physical people and physical tasks, lies a second economy that is automatic and neurally intelligent, with no upper limit to its buildout. The prosperity we enjoy and the difficulties with jobs would not have surprised Keynes, but the means of achieving that prosperity would have.
Digitization is creating a second economy that’s vast, automatic, and invisible—thereby bringing the biggest change since the Industrial Revolution.
W. Brian Arthur McKinsey Quarterly October 2011
In This Article
· Sidebar: How fast is the second economy growing?
· About the author
In 1850, a decade before the Civil War, the United States’ economy was small—it wasn’t much bigger than Italy’s. Forty years later, it was the largest economy in the world. What happened in-between was the railroads. They linked the east of the country to the west, and the interior to both. They gave access to the east’s industrial goods; they made possible economies of scale; they stimulated steel and manufacturing—and the economy was never the same.
Deep changes like this are not unusual. Every so often—every 60 years or so—a body of technology comes along and over several decades, quietly, almost unnoticeably, transforms the economy: it brings new social classes to the fore and creates a different world for business. Can such a transformation—deep and slow and silent—be happening today
We could look for one in the genetic technologies, or in nanotech, but their time hasn’t fully come. But I want to argue that something deep is going on with information technology, something that goes well beyond the use of computers, social media, and commerce on the Internet. Business processes that once took place among human beings are now being executed electronically. They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential—it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.
Let me begin with two examples. Twenty years ago, if you went into an airport you would walk up to a counter and present paper tickets to a human being. That person would register you on a computer, notify the flight you’d arrived, and check your luggage in. All this was done by humans. Today, you walk into an airport and look for a machine. You put in a frequent-flier card or credit card, and it takes just three or four seconds to get back a boarding pass, receipt, and luggage tag. What interests me is what happens in those three or four seconds. The moment the card goes in, you are starting a huge conversation conducted entirely among machines. Once your name is recognized, computers are checking your flight status with the airlines, your past travel history, your name with the TSA1 (and possibly also with the National Security Agency). They are checking your seat choice, your frequent-flier status, and your access to lounges. This unseen, underground conversation is happening among multiple servers talking to other servers, talking to satellites that are talking to computers (possibly in London, where you’re going), and checking with passport control, with foreign immigration, with ongoing connecting flights. And to make sure the aircraft’s weight distribution is fine, the machines are also starting to adjust the passenger count and seating according to whether the fuselage is loaded more heavily at the front or back.
These large and fairly complicated conversations that you’ve triggered occur entirely among things remotely talking to other things: servers, switches, routers, and other Internet and telecommunications devices, updating and shuttling information back and forth. All of this occurs in the few seconds it takes to get your boarding pass back. And even after that happens, if you could see these conversations as flashing lights, they’d still be flashing all over the country for some time, perhaps talking to the flight controllers—starting to say that the flight’s getting ready for departure and to prepare for that.
Now consider a second example, from supply chain management. Twenty years ago, if you were shipping freight through Rotterdam into the center of Europe, people with clipboards would be registering arrival, checking manifests, filling out paperwork, and telephoning forward destinations to let other people know. Now such shipments go through an RFID2 portal where they are scanned, digitally captured, and automatically dispatched. The RFID portal is in conversation digitally with the originating shipper, other depots, other suppliers, and destinations along the route, all keeping track, keeping control, and reconfiguring routing if necessary to optimize things along the way. What used to be done by humans is now executed as a series of conversations among remotely located servers.
In both these examples, and all across economies in the developed world, processes in the physical economy are being entered into the digital economy, where they are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.
So we can say that another economy—a second economy—of all of these digitized business processes conversing, executing, and triggering further actions is silently forming alongside the physical economy.
Aspen root systems
If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent—a great computer expression—which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.
These last descriptors sound biological—and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, there’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.
The metaphor isn’t perfect: this emerging second-economy root system is more complicated than any aspen system, since it’s also making new connections and new configurations on the fly. But the aspen metaphor is useful for capturing the reality that the observable physical world of aspen trees hides an unseen underground root system just as large or even larger.
How large is the unseen second economy? By a rough back-of-the-envelope calculation (see sidebar, “How fast is the second economy growing?”), in about two decades the digital economy will reach the same size as the physical economy. It’s as if there will be another American economy anchored off San Francisco (or, more in keeping with my metaphor, slipped in underneath the original economy) and growing all the while.
Now this second, digital economy isn’t producing anything tangible. It’s not making my bed in a hotel, or bringing me orange juice in the morning. But it is running an awful lot of the economy. It’s helping architects design buildings, it’s tracking sales and inventory, getting goods from here to there, executing trades and banking operations, controlling manufacturing equipment, making design calculations, billing clients, navigating aircraft, helping diagnose patients, and guiding laparoscopic surgeries. Such operations grow slowly and take time to form. In any deep transformation, industries do not so much adopt the new body of technology as encounter it, and as they do so they create new ways to profit from its possibilities.
The deep transformation I am describing is happening not just in the United States but in all advanced economies, especially in Europe and Japan. And its revolutionary scale can only be grasped if we go beyond my aspen metaphor to another analogy.
A neural system for the economy
Recall that in the digital conversations I was describing, something that occurs in the physical economy is sensed by the second economy—which then gives back an appropriate response. A truck passes its load through an RFID sensor or you check in at the airport, a lot of recomputation takes place, and appropriate physical actions are triggered.
There’s a parallel in this with how biologists think of intelligence. I’m not talking about human intelligence or anything that would qualify as conscious intelligence. Biologists tell us that an organism is intelligent if it senses something, changes its internal state, and reacts appropriately. If you put an E. coli bacterium into an uneven concentration of glucose, it does the appropriate thing by swimming toward where the glucose is more concentrated. Biologists would call this intelligent behavior. The bacterium senses something, “computes” something (although we may not know exactly how), and returns an appropriate response.
No brain need be involved. A primitive jellyfish doesn’t have a central nervous system or brain. What it has is a kind of neural layer or nerve net that lets it sense and react appropriately. I’m arguing that all these aspen roots—this vast global digital network that is sensing, “computing,” and reacting appropriately—is starting to constitute a neural layer for the economy. The second economy constitutes a neural layer for the physical economy. Just what sort of change is this qualitatively?
Think of it this way. With the coming of the Industrial Revolution—roughly from the 1760s, when Watt’s steam engine appeared, through around 1850 and beyond—the economy developed a muscular system in the form of machine power. Now it is developing a neural system. This may sound grandiose, but actually I think the metaphor is valid. Around 1990, computers started seriously to talk to each other, and all these connections started to happen. The individual machines—servers—are like neurons, and the axons and synapses are the communication pathways and linkages that enable them to be in conversation with each other and to take appropriate action.
Is this the biggest change since the Industrial Revolution? Well, without sticking my neck out too much, I believe so. In fact, I think it may well be the biggest change ever in the economy. It is a deep qualitative change that is bringing intelligent, automatic response to the economy. There’s no upper limit to this, no place where it has to end. Now, I’m not interested in science fiction, or predicting the singularity, or talking about cyborgs. None of that interests me. What I am saying is that it would be easy to underestimate the degree to which this is going to make a difference.
I think that for the rest of this century, barring wars and pestilence, a lot of the story will be the building out of this second economy, an unseen underground economy that basically is giving us intelligent reactions to what we do above the ground. For example, if I’m driving in Los Angeles in 15 years’ time, likely it’ll be a driverless car in a flow of traffic where my car’s in a conversation with the cars around it that are in conversation with general traffic and with my car. The second economy is creating for us—slowly, quietly, and steadily—a different world.
A downside
Of course, as with most changes, there is a downside. I am concerned that there is an adverse impact on jobs. Productivity increasing, say, at 2.4 percent in a given year means either that the same number of people can produce 2.4 percent more output or that we can get the same output with 2.4 percent fewer people. Both of these are happening. We are getting more output for each person in the economy, but overall output, nationally, requires fewer people to produce it. Nowadays, fewer people are required behind the desk of an airline. Much of the work is still physical—someone still has to take your luggage and put it on the belt—but much has vanished into the digital world of sensing, digital communication, and intelligent response.
Physical jobs are disappearing into the second economy, and I believe this effect is dwarfing the much more publicized effect of jobs disappearing to places like India and China.
There are parallels with what has happened before. In the early 20th century, farm jobs became mechanized and there was less need for farm labor, and some decades later manufacturing jobs became mechanized and there was less need for factory labor. Now business processes—many in the service sector—are becoming “mechanized” and fewer people are needed, and this is exerting systematic downward pressure on jobs. We don’t have paralegals in the numbers we used to. Or draftsmen, telephone operators, typists, or bookkeeping people. A lot of that work is now done digitally. We do have police and teachers and doctors; where there’s a need for human judgment and human interaction, we still have that. But the primary cause of all of the downsizing we’ve had since the mid-1990s is that a lot of human jobs are disappearing into the second economy. Not to reappear.
Seeing things this way, it’s not surprising we are still working our way out of the bad 2008–09 recession with a great deal of joblessness.
There’s a larger lesson to be drawn from this. The second economy will certainly be the engine of growth and the provider of prosperity for the rest of this century and beyond, but it may not provide jobs, so there may be prosperity without full access for many. This suggests to me that the main challenge of the economy is shifting from producing prosperity to distributing prosperity. The second economy will produce wealth no matter what we do; distributing that wealth has become the main problem. For centuries, wealth has traditionally been apportioned in the West through jobs, and jobs have always been forthcoming. When farm jobs disappeared, we still had manufacturing jobs, and when these disappeared we migrated to service jobs. With this digital transformation, this last repository of jobs is shrinking—fewer of us in the future may have white-collar business process jobs—and we face a problem.
The system will adjust of course, though I can’t yet say exactly how. Perhaps some new part of the economy will come forward and generate a whole new set of jobs. Perhaps we will have short workweeks and long vacations so there will be more jobs to go around. Perhaps we will have to subsidize job creation. Perhaps the very idea of a job and of being productive will change over the next two or three decades. The problem is by no means insoluble. The good news is that if we do solve it we may at last have the freedom to invest our energies in creative acts.
Economic possibilities for our grandchildren
In 1930, Keynes wrote a famous essay, “Economic possibilities for our grandchildren.” Reading it now, in the era of those grandchildren, I am surprised just how accurate it is. Keynes predicts that “the standard of life in progressive countries one hundred years hence will be between four and eight times as high as it is to-day.” He rightly warns of “technological unemployment,” but dares to surmise that “the economic problem [of producing enough goods] may be solved.” If we had asked him and his contemporaries how all this might come about, they might have imagined lots of factories with lots of machines, possibly even with robots, with the workers in these factories gradually being replaced by machines and by individual robots.
That is not quite how things have developed. We do have sophisticated machines, but in the place of personal automation (robots) we have a collective automation. Underneath the physical economy, with its physical people and physical tasks, lies a second economy that is automatic and neurally intelligent, with no upper limit to its buildout. The prosperity we enjoy and the difficulties with jobs would not have surprised Keynes, but the means of achieving that prosperity would have.
his second economy that is silently forming—vast, interconnected, and extraordinarily productive—is creating for us a new economic world. How we will fare in this world, how we will adapt to it, how we will profit from it and share its benefits, is very much up to us.
W. Brian Arthur is a visting researcher with the Intelligent System Lab at the Palo Alto Research Center (PARC) and an external professor at the Santa Fe Institute. He is an economist and technology thinker and a pioneer in the science of complexity. His 1994 book, Increasing Returns and Path Dependence in the Economy (University of Michigan Press, December 1994), contains several of his seminal papers. More recently, Arthur was the author of The Nature of Technology: What it is and How it Evolves (Free Press, August 2009).
Tuesday, October 4, 2011
Rob Atkinson: Government Opportunities to Harness “Big Data”
Posted by Rob Atkinson to Innovation Policy Blog
 |
Recently more attention has been drawn to the emergence of "Big Data"—large scale data sets that businesses and government are using to unlock new value using today's computing and communications power. As a McKinsey Global Institute (MGI) study recently showed, Big Data offers a wide range of commercial opportunities in virtually every sector of the economy for the United States. To take one example, the MGI estimates that better use of big data in health care could generate an additional $300 billion, with approximately two-thirds of that values coming from more efficient delivery of health care.
The use of Big Data should not be confined to just the private sector; data offers incredible new opportunities to the public sector as well.Policymakers have the opportunity to use Big Data to improve government in areas such as public safety, public health, public utilities and public transportation.
ITIF has discussed many of these opportunities before.
ITIF has discussed many of these opportunities before.
Consider the following:
- Electric power utilities can use data analytics and smart meters to better manage resources and avoid blackouts,
- Food inspectors can use data to better track meat and produce safety from farm to fork ,
- Public health officials can use health data to detect infectious disease outbreaks,
- Regulators can track pharmaceutical and medical device safety and effectiveness through better data analytics,
- Police departments can use data analytics to target crime hotspots and prevent crime waves,
- Public utilities can use sensors to collect data on water and sewer usage to detect leaks and reduce water consumption,
- First responders can use sensors, GPS, cameras and better communication systems to let police and fire fighters better protect citizens when responding to emergencies,
- State departments of transportation can use data to reduce traffic, more efficiently deploy resources, and implement congestion pricing systems.
Better use of data can help government agencies, from city agencies to federal bureaucracies, operate more efficiently, create more transparency, and make more informed decisions. And government can use cloud computing to more efficiently develop online systems that provide anytime, anywhere access to information. However, government officials should do more to spur uses of data. Taking advantage of these opportunities will require federal government leadership, such as the Department of Commerce creating a data policy office to spur data innovation and overcome obstacles to adoption, all the while protecting privacy. And going forward, government agencies will increasingly have to deal with issues such as data security and identity management, so these issues do not become impediments to successful utilization of data analytics. Local governments can help pioneer the use of data as well. For example, the city of Boston city sponsored the development of a mobile app "Street Bump" to automatically determine where potholes are based on data collected using citizen's smart phones equipped with GPS and accelerometers. Tools like these are helping create "smart cities" and build a world that is alive with information.
Although there have been many successes in this area, much more can be done. For example, in homeland security, law enforcement must deal with a changing threat landscape. While corporations and individuals can increasingly use better technology to communicate and store data security, criminals can also use these same tools. As a result, law enforcement is increasingly confronting the "Going Dark" problem where they have less access to investigative data, not because of a lack of legal authority, but because of technological hurdles. Yet while law enforcement may have a reduced ability to intercept some types of communication, they now have many more sources of data, such as transactional data, to use to detect threats. As ITIF discussed at an event in 2010 following the Christmas Day terrorist attempt, the intelligence community still needs to develop better analytical tools to "connect the dots" and allow intelligence officers to do a better job. Similarly in many other sectors, Big Data offers government opportunities to reinvent how to operate effectively.
Overall, more investment in data infrastructure and analytics will enable government to better provide and efficiently deliver values and services to its citizens.
Silicon Valley reborn as Smartphone Valley (FT)
By Chris Nuttall in Palo Alto October 3, 2011 4:55 pm Finncial Times
Apple’s unveiling of the iPhone 5 on Tuesday at its Cupertino headquarters is just the latest sign that Silicon Valley is taking on a fresh mantle of Smartphone Valley, with its growing reputation making it a magnet for mobile operators around the world.
AT&T, Verizon and Vodafone have all just opened research, testing and incubation centres in San Francisco and the Valley only weeks apart.
“The reason we are here is this is the centre of the Earth right now when it comes to innovation,” said Fay Arjomandi, head of research and development for Vodafone in the US, at the opening of its Xone Lab in Redwood City last month.
Apple’s iPhone and Google’s Android operating system have set new standards for hardware and software in the mobile industry.
The app culture that both have engendered means there is a race on among operators to feature the newest trends and advances in software first, amid fierce competition for the attention of Silicon Valley developers.
Vodafone’s lab allows developers to incubate their ideas and test how apps and services would perform on the worldwide networks it replicates at its facility.
“We want to bring to market these ideas and accelerate the process – the hope is that six to nine months after we prove something here, it can be in the hands of a user in one of our operating countries,” says Ms Arjomandi.
A week after the Vodafone opening, AT&T launched its Foundry innovation lab in an old French steam laundry in Palo Alto.
It too emphasised the need to move “at web developer rather than carrier speed” – to the extent that much of the office furniture is on wheels and is easily reconfigurable.
“This is part of our transformation – to move from being a telecoms company to a technology company,” said John Donovan, AT&T’s chief technology officer.
The operators have felt the need to change as technology companies such as Apple and Google, which is buying Motorola, have become more influential and handset makers have aligned themselves behind different operating systems.
“They have to show they are a part of the ecosystem, there’s been a big shift in the last year or so, when there has been the threat that carriers just become a dumb pipe [for data],” says Chris Jones, a telecoms analyst based in the Valley for the Canalys research firm.
Trevor Healy, chief executive of the mobile advertising platform Amobee, whose previous Valley company Jajah was bought by Spain’s Telefónica, has also noticed a change in attitude forced upon the operators.
“It’s not long ago that these companies were myopic and closed, but now there’s a carrot and stick for them in the shape of new revenue streams when their core revenues are stagnant and [they face] the stick of other big players like Google and Apple making ground,” he says.
The operators are arriving late to the Valley compared with infrastructure and handset players such as Ericsson and Nokia, who have had research centres here for years and base their chief technology officers in the Bay Area.
“The operators may have abdicated their positions with the operating systems and handsets, but now they’re putting a stake in the ground and focusing on app development, advertising and payments and billing,” says Mr Healy.
The consumer smartphone market may appear to have been carved up between Apple’s App Store and the Android Market, but operators feel they can still offer premium apps and services based on their network’s ability to know location, process payments, serve advertising, communicate with a range of devices and combine disparate services.
Mr Donovan boasted that the Foundry had 106 projects under way and that AT&T was now measuring itself on the openness of its network: 82 APIs (or access points into the network) had been launched over the past year.
Start-up demonstrations at both the AT&T and Vodafone opening events hinted at the “mash-ups” that were possible through mobile networks – for example, how an app recording running performances could link up with ones relaying sensor information on a person’s health.
Augmented reality applications, improved graphics for gaming and the ability for machines to talk to other machines – “the internet of things” – could all benefit from being hard-wired into the operators’ networks, the demonstrations proved.
It remains to be seen whether developers will be attracted in significant numbers to working with the operators, but Ms Arjomandi argues all parties can benefit.
“We would like to have the advantage of [perhaps one-year exclusivity] lead time to the market – to have a differentiation for us, we need to get that lead time,” she said.
“In exchange, we give them access to 340m customers around the world in 30-plus countries – so it’s definitely a win-win situation.”
Apple’s unveiling of the iPhone 5 on Tuesday at its Cupertino headquarters is just the latest sign that Silicon Valley is taking on a fresh mantle of Smartphone Valley, with its growing reputation making it a magnet for mobile operators around the world.
AT&T, Verizon and Vodafone have all just opened research, testing and incubation centres in San Francisco and the Valley only weeks apart.
“The reason we are here is this is the centre of the Earth right now when it comes to innovation,” said Fay Arjomandi, head of research and development for Vodafone in the US, at the opening of its Xone Lab in Redwood City last month.
Apple’s iPhone and Google’s Android operating system have set new standards for hardware and software in the mobile industry.
The app culture that both have engendered means there is a race on among operators to feature the newest trends and advances in software first, amid fierce competition for the attention of Silicon Valley developers.
Vodafone’s lab allows developers to incubate their ideas and test how apps and services would perform on the worldwide networks it replicates at its facility.
“We want to bring to market these ideas and accelerate the process – the hope is that six to nine months after we prove something here, it can be in the hands of a user in one of our operating countries,” says Ms Arjomandi.
A week after the Vodafone opening, AT&T launched its Foundry innovation lab in an old French steam laundry in Palo Alto.
It too emphasised the need to move “at web developer rather than carrier speed” – to the extent that much of the office furniture is on wheels and is easily reconfigurable.
“This is part of our transformation – to move from being a telecoms company to a technology company,” said John Donovan, AT&T’s chief technology officer.
The operators have felt the need to change as technology companies such as Apple and Google, which is buying Motorola, have become more influential and handset makers have aligned themselves behind different operating systems.
“They have to show they are a part of the ecosystem, there’s been a big shift in the last year or so, when there has been the threat that carriers just become a dumb pipe [for data],” says Chris Jones, a telecoms analyst based in the Valley for the Canalys research firm.
Trevor Healy, chief executive of the mobile advertising platform Amobee, whose previous Valley company Jajah was bought by Spain’s Telefónica, has also noticed a change in attitude forced upon the operators.
“It’s not long ago that these companies were myopic and closed, but now there’s a carrot and stick for them in the shape of new revenue streams when their core revenues are stagnant and [they face] the stick of other big players like Google and Apple making ground,” he says.
The operators are arriving late to the Valley compared with infrastructure and handset players such as Ericsson and Nokia, who have had research centres here for years and base their chief technology officers in the Bay Area.
“The operators may have abdicated their positions with the operating systems and handsets, but now they’re putting a stake in the ground and focusing on app development, advertising and payments and billing,” says Mr Healy.
The consumer smartphone market may appear to have been carved up between Apple’s App Store and the Android Market, but operators feel they can still offer premium apps and services based on their network’s ability to know location, process payments, serve advertising, communicate with a range of devices and combine disparate services.
Mr Donovan boasted that the Foundry had 106 projects under way and that AT&T was now measuring itself on the openness of its network: 82 APIs (or access points into the network) had been launched over the past year.
Start-up demonstrations at both the AT&T and Vodafone opening events hinted at the “mash-ups” that were possible through mobile networks – for example, how an app recording running performances could link up with ones relaying sensor information on a person’s health.
Augmented reality applications, improved graphics for gaming and the ability for machines to talk to other machines – “the internet of things” – could all benefit from being hard-wired into the operators’ networks, the demonstrations proved.
It remains to be seen whether developers will be attracted in significant numbers to working with the operators, but Ms Arjomandi argues all parties can benefit.
“We would like to have the advantage of [perhaps one-year exclusivity] lead time to the market – to have a differentiation for us, we need to get that lead time,” she said.
“In exchange, we give them access to 340m customers around the world in 30-plus countries – so it’s definitely a win-win situation.”
Subscribe to:
Posts (Atom)