Thursday, October 6, 2011

Brian Arthur: Contemplate an economy run entirely by machines

The second economy
Digitization is creating a second economy that’s vast, automatic, and invisible—thereby bringing the biggest change since the Industrial Revolution.


W. Brian Arthur    McKinsey Quarterly    October 2011

In This Article
·       Sidebar: How fast is the second economy growing?
·       About the author

In 1850, a decade before the Civil War, the United States’ economy was small—it wasn’t much bigger than Italy’s. Forty years later, it was the largest economy in the world. What happened in-between was the railroads. They linked the east of the country to the west, and the interior to both. They gave access to the east’s industrial goods; they made possible economies of scale; they stimulated steel and manufacturing—and the economy was never the same.

Deep changes like this are not unusual. Every so often—every 60 years or so—a body of technology comes along and over several decades, quietly, almost unnoticeably, transforms the economy: it brings new social classes to the fore and creates a different world for business. Can such a transformation—deep and slow and silent—be happening today

We could look for one in the genetic technologies, or in nanotech, but their time hasn’t fully come. But I want to argue that something deep is going on with information technology, something that goes well beyond the use of computers, social media, and commerce on the Internet. Business processes that once took place among human beings are now being executed electronically. They are taking place in an unseen domain that is strictly digital. On the surface, this shift doesn’t seem particularly consequential—it’s almost something we take for granted. But I believe it is causing a revolution no less important and dramatic than that of the railroads. It is quietly creating a second economy, a digital one.


Let me begin with two examples. Twenty years ago, if you went into an airport you would walk up to a counter and present paper tickets to a human being. That person would register you on a computer, notify the flight you’d arrived, and check your luggage in. All this was done by humans. Today, you walk into an airport and look for a machine. You put in a frequent-flier card or credit card, and it takes just three or four seconds to get back a boarding pass, receipt, and luggage tag. What interests me is what happens in those three or four seconds. The moment the card goes in, you are starting a huge conversation conducted entirely among machines. Once your name is recognized, computers are checking your flight status with the airlines, your past travel history, your name with the TSA1 (and possibly also with the National Security Agency). They are checking your seat choice, your frequent-flier status, and your access to lounges. This unseen, underground conversation is happening among multiple servers talking to other servers, talking to satellites that are talking to computers (possibly in London, where you’re going), and checking with passport control, with foreign immigration, with ongoing connecting flights. And to make sure the aircraft’s weight distribution is fine, the machines are also starting to adjust the passenger count and seating according to whether the fuselage is loaded more heavily at the front or back.

These large and fairly complicated conversations that you’ve triggered occur entirely among things remotely talking to other things: servers, switches, routers, and other Internet and telecommunications devices, updating and shuttling information back and forth. All of this occurs in the few seconds it takes to get your boarding pass back. And even after that happens, if you could see these conversations as flashing lights, they’d still be flashing all over the country for some time, perhaps talking to the flight controllers—starting to say that the flight’s getting ready for departure and to prepare for that.

Now consider a second example, from supply chain management. Twenty years ago, if you were shipping freight through Rotterdam into the center of Europe, people with clipboards would be registering arrival, checking manifests, filling out paperwork, and telephoning forward destinations to let other people know. Now such shipments go through an RFID2 portal where they are scanned, digitally captured, and automatically dispatched. The RFID portal is in conversation digitally with the originating shipper, other depots, other suppliers, and destinations along the route, all keeping track, keeping control, and reconfiguring routing if necessary to optimize things along the way. What used to be done by humans is now executed as a series of conversations among remotely located servers.

In both these examples, and all across economies in the developed world, processes in the physical economy are being entered into the digital economy, where they are “speaking to” other processes in the digital economy, in a constant conversation among multiple servers and multiple semi-intelligent nodes that are updating things, querying things, checking things off, readjusting things, and eventually connecting back with processes and humans in the physical economy.

So we can say that another economy—a second economy—of all of these digitized business processes conversing, executing, and triggering further actions is silently forming alongside the physical economy.

Aspen root systems
If I were to look for adjectives to describe this second economy, I’d say it is vast, silent, connected, unseen, and autonomous (meaning that human beings may design it but are not directly involved in running it). It is remotely executing and global, always on, and endlessly configurable. It is concurrent—a great computer expression—which means that everything happens in parallel. It is self-configuring, meaning it constantly reconfigures itself on the fly, and increasingly it is also self-organizing, self-architecting, and self-healing.

These last descriptors sound biological—and they are. In fact, I’m beginning to think of this second economy, which is under the surface of the physical economy, as a huge interconnected root system, very much like the root system for aspen trees. For every acre of aspen trees above the ground, there’s about ten miles of roots underneath, all interconnected with one another, “communicating” with each other.

The metaphor isn’t perfect: this emerging second-economy root system is more complicated than any aspen system, since it’s also making new connections and new configurations on the fly. But the aspen metaphor is useful for capturing the reality that the observable physical world of aspen trees hides an unseen underground root system just as large or even larger.

How large is the unseen second economy? By a rough back-of-the-envelope calculation (see sidebar, “How fast is the second economy growing?”), in about two decades the digital economy will reach the same size as the physical economy. It’s as if there will be another American economy anchored off San Francisco (or, more in keeping with my metaphor, slipped in underneath the original economy) and growing all the while.

Now this second, digital economy isn’t producing anything tangible. It’s not making my bed in a hotel, or bringing me orange juice in the morning. But it is running an awful lot of the economy. It’s helping architects design buildings, it’s tracking sales and inventory, getting goods from here to there, executing trades and banking operations, controlling manufacturing equipment, making design calculations, billing clients, navigating aircraft, helping diagnose patients, and guiding laparoscopic surgeries. Such operations grow slowly and take time to form. In any deep transformation, industries do not so much adopt the new body of technology as encounter it, and as they do so they create new ways to profit from its possibilities.

The deep transformation I am describing is happening not just in the United States but in all advanced economies, especially in Europe and Japan. And its revolutionary scale can only be grasped if we go beyond my aspen metaphor to another analogy.

A neural system for the economy
Recall that in the digital conversations I was describing, something that occurs in the physical economy is sensed by the second economy—which then gives back an appropriate response. A truck passes its load through an RFID sensor or you check in at the airport, a lot of recomputation takes place, and appropriate physical actions are triggered.

There’s a parallel in this with how biologists think of intelligence. I’m not talking about human intelligence or anything that would qualify as conscious intelligence. Biologists tell us that an organism is intelligent if it senses something, changes its internal state, and reacts appropriately. If you put an E. coli bacterium into an uneven concentration of glucose, it does the appropriate thing by swimming toward where the glucose is more concentrated. Biologists would call this intelligent behavior. The bacterium senses something, “computes” something (although we may not know exactly how), and returns an appropriate response.

No brain need be involved. A primitive jellyfish doesn’t have a central nervous system or brain. What it has is a kind of neural layer or nerve net that lets it sense and react appropriately. I’m arguing that all these aspen roots—this vast global digital network that is sensing, “computing,” and reacting appropriately—is starting to constitute a neural layer for the economy. The second economy constitutes a neural layer for the physical economy. Just what sort of change is this qualitatively?

Think of it this way. With the coming of the Industrial Revolution—roughly from the 1760s, when Watt’s steam engine appeared, through around 1850 and beyond—the economy developed a muscular system in the form of machine power. Now it is developing a neural system. This may sound grandiose, but actually I think the metaphor is valid. Around 1990, computers started seriously to talk to each other, and all these connections started to happen. The individual machines—servers—are like neurons, and the axons and synapses are the communication pathways and linkages that enable them to be in conversation with each other and to take appropriate action.

Is this the biggest change since the Industrial Revolution? Well, without sticking my neck out too much, I believe so. In fact, I think it may well be the biggest change ever in the economy. It is a deep qualitative change that is bringing intelligent, automatic response to the economy. There’s no upper limit to this, no place where it has to end. Now, I’m not interested in science fiction, or predicting the singularity, or talking about cyborgs. None of that interests me. What I am saying is that it would be easy to underestimate the degree to which this is going to make a difference.

I think that for the rest of this century, barring wars and pestilence, a lot of the story will be the building out of this second economy, an unseen underground economy that basically is giving us intelligent reactions to what we do above the ground. For example, if I’m driving in Los Angeles in 15 years’ time, likely it’ll be a driverless car in a flow of traffic where my car’s in a conversation with the cars around it that are in conversation with general traffic and with my car. The second economy is creating for us—slowly, quietly, and steadily—a different world.

A downside
Of course, as with most changes, there is a downside. I am concerned that there is an adverse impact on jobs. Productivity increasing, say, at 2.4 percent in a given year means either that the same number of people can produce 2.4 percent more output or that we can get the same output with 2.4 percent fewer people. Both of these are happening. We are getting more output for each person in the economy, but overall output, nationally, requires fewer people to produce it. Nowadays, fewer people are required behind the desk of an airline. Much of the work is still physical—someone still has to take your luggage and put it on the belt—but much has vanished into the digital world of sensing, digital communication, and intelligent response.

Physical jobs are disappearing into the second economy, and I believe this effect is dwarfing the much more publicized effect of jobs disappearing to places like India and China.

There are parallels with what has happened before. In the early 20th century, farm jobs became mechanized and there was less need for farm labor, and some decades later manufacturing jobs became mechanized and there was less need for factory labor. Now business processes—many in the service sector—are becoming “mechanized” and fewer people are needed, and this is exerting systematic downward pressure on jobs. We don’t have paralegals in the numbers we used to. Or draftsmen, telephone operators, typists, or bookkeeping people. A lot of that work is now done digitally. We do have police and teachers and doctors; where there’s a need for human judgment and human interaction, we still have that. But the primary cause of all of the downsizing we’ve had since the mid-1990s is that a lot of human jobs are disappearing into the second economy. Not to reappear.

Seeing things this way, it’s not surprising we are still working our way out of the bad 2008–09 recession with a great deal of joblessness.

There’s a larger lesson to be drawn from this. The second economy will certainly be the engine of growth and the provider of prosperity for the rest of this century and beyond, but it may not provide jobs, so there may be prosperity without full access for many. This suggests to me that the main challenge of the economy is shifting from producing prosperity to distributing prosperity. The second economy will produce wealth no matter what we do; distributing that wealth has become the main problem. For centuries, wealth has traditionally been apportioned in the West through jobs, and jobs have always been forthcoming. When farm jobs disappeared, we still had manufacturing jobs, and when these disappeared we migrated to service jobs. With this digital transformation, this last repository of jobs is shrinking—fewer of us in the future may have white-collar business process jobs—and we face a problem.

The system will adjust of course, though I can’t yet say exactly how. Perhaps some new part of the economy will come forward and generate a whole new set of jobs. Perhaps we will have short workweeks and long vacations so there will be more jobs to go around. Perhaps we will have to subsidize job creation. Perhaps the very idea of a job and of being productive will change over the next two or three decades. The problem is by no means insoluble. The good news is that if we do solve it we may at last have the freedom to invest our energies in creative acts.

Economic possibilities for our grandchildren
In 1930, Keynes wrote a famous essay, “Economic possibilities for our grandchildren.” Reading it now, in the era of those grandchildren, I am surprised just how accurate it is. Keynes predicts that “the standard of life in progressive countries one hundred years hence will be between four and eight times as high as it is to-day.” He rightly warns of “technological unemployment,” but dares to surmise that “the economic problem [of producing enough goods] may be solved.” If we had asked him and his contemporaries how all this might come about, they might have imagined lots of factories with lots of machines, possibly even with robots, with the workers in these factories gradually being replaced by machines and by individual robots.

That is not quite how things have developed. We do have sophisticated machines, but in the place of personal automation (robots) we have a collective automation. Underneath the physical economy, with its physical people and physical tasks, lies a second economy that is automatic and neurally intelligent, with no upper limit to its buildout. The prosperity we enjoy and the difficulties with jobs would not have surprised Keynes, but the means of achieving that prosperity would have.

his second economy that is silently forming—vast, interconnected, and extraordinarily productive—is creating for us a new economic world. How we will fare in this world, how we will adapt to it, how we will profit from it and share its benefits, is very much up to us.

About the Author
W. Brian Arthur is a visting researcher with the Intelligent System Lab at the Palo Alto Research Center (PARC) and an external professor at the Santa Fe Institute. He is an economist and technology thinker and a pioneer in the science of complexity. His 1994 book, Increasing Returns and Path Dependence in the Economy (University of Michigan Press, December 1994), contains several of his seminal papers. More recently, Arthur was the author of The Nature of Technology: What it is and How it Evolves (Free Press, August 2009).

Tuesday, October 4, 2011

Rob Atkinson: Government Opportunities to Harness “Big Data”

5150336351_ae2a64336a
Recently more attention has been drawn to the emergence of "Big Data"—large scale data sets that businesses and government are using to unlock new value using today's computing and communications power. As a McKinsey Global Institute (MGI) study recently showed, Big Data offers a wide range of commercial opportunities in virtually every sector of the economy for the United States. To take one example, the MGI estimates that better use of big data in health care could generate an additional $300 billion, with approximately two-thirds of that values coming from more efficient delivery of health care.

The use of Big Data should not be confined to just the private sector; data offers incredible new opportunities to the public sector as well.Policymakers have the opportunity to use Big Data to improve government in areas such as public safety, public health, public utilities and public transportation.

ITIF has discussed many of these opportunities before.
Consider the following:
Better use of data can help government agencies, from city agencies to federal bureaucracies, operate more efficiently, create more transparency, and make more informed decisions. And government can use cloud computing to more efficiently develop online systems that provide anytime, anywhere access to information. However, government officials should do more to spur uses of data. Taking advantage of these opportunities will require federal government leadership, such as the Department of Commerce creating a data policy office to spur data innovation and overcome obstacles to adoption, all the while protecting privacy. And going forward, government agencies will increasingly have to deal with issues such as data security and identity management, so these issues do not become impediments to successful utilization of data analytics. Local governments can help pioneer the use of data as well.  For example, the city of Boston city sponsored the development of a mobile app "Street Bump" to automatically determine where potholes are based on data collected using citizen's smart phones equipped with GPS and accelerometers. Tools like these are helping create "smart cities" and build a world that is alive with information.

Although there have been many successes in this area, much more can be done. For example, in homeland security, law enforcement must deal with a changing threat landscape. While corporations and individuals can increasingly use better technology to communicate and store data security, criminals can also use these same tools.  As a result, law enforcement is increasingly confronting the "
Going Dark" problem where they have less access to investigative data, not because of a lack of legal authority, but because of technological hurdles. Yet while law enforcement may have a reduced ability to intercept some types of communication, they now have many more sources of data, such as transactional data, to use to detect threats. As ITIF discussed at an event in 2010 following the Christmas Day terrorist attempt, the intelligence community still needs to develop better analytical tools to "connect the dots" and allow intelligence officers to do a better job. Similarly in many other sectors, Big Data offers government opportunities to reinvent how to operate effectively. 

Overall, more investment in data infrastructure and analytics will enable government to better provide and efficiently deliver values and services to its citizens.

Silicon Valley reborn as Smartphone Valley (FT)

By Chris Nuttall in Palo Alto    October 3, 2011 4:55 pm   Finncial Times

 Apple’s unveiling of the iPhone 5 on Tuesday at its Cupertino headquarters is just the latest sign that Silicon Valley is taking on a fresh mantle of Smartphone Valley, with its growing reputation making it a magnet for mobile operators around the world.

AT&T, Verizon and Vodafone have all just opened research, testing and incubation centres in San Francisco and the Valley only weeks apart.

 “The reason we are here is this is the centre of the Earth right now when it comes to innovation,” said Fay Arjomandi, head of research and development for Vodafone in the US, at the opening of its Xone Lab in Redwood City last month.

Apple’s iPhone and Google’s Android operating system have set new standards for hardware and software in the mobile industry.

The app culture that both have engendered means there is a race on among operators to feature the newest trends and advances in software first, amid fierce competition for the attention of Silicon Valley developers.
Vodafone’s lab allows developers to incubate their ideas and test how apps and services would perform on the worldwide networks it replicates at its facility.

“We want to bring to market these ideas and accelerate the process – the hope is that six to nine months after we prove something here, it can be in the hands of a user in one of our operating countries,” says Ms Arjomandi.

A week after the Vodafone opening, AT&T launched its Foundry innovation lab in an old French steam laundry in Palo Alto.

It too emphasised the need to move “at web developer rather than carrier speed” – to the extent that much of the office furniture is on wheels and is easily reconfigurable.

“This is part of our transformation – to move from being a telecoms company to a technology company,” said John Donovan, AT&T’s chief technology officer.

The operators have felt the need to change as technology companies such as Apple and Google, which is buying Motorola, have become more influential and handset makers have aligned themselves behind different operating systems.

“They have to show they are a part of the ecosystem, there’s been a big shift in the last year or so, when there has been the threat that carriers just become a dumb pipe [for data],” says Chris Jones, a telecoms analyst based in the Valley for the Canalys research firm.

Trevor Healy, chief executive of the mobile advertising platform Amobee, whose previous Valley company Jajah was bought by Spain’s Telefónica, has also noticed a change in attitude forced upon the operators.
“It’s not long ago that these companies were myopic and closed, but now there’s a carrot and stick for them in the shape of new revenue streams when their core revenues are stagnant and [they face] the stick of other big players like Google and Apple making ground,” he says.

The operators are arriving late to the Valley compared with infrastructure and handset players such as Ericsson and Nokia, who have had research centres here for years and base their chief technology officers in the Bay Area.

“The operators may have abdicated their positions with the operating systems and handsets, but now they’re putting a stake in the ground and focusing on app development, advertising and payments and billing,” says Mr Healy.
The consumer smartphone market may appear to have been carved up between Apple’s App Store and the Android Market, but operators feel they can still offer premium apps and services based on their network’s ability to know location, process payments, serve advertising, communicate with a range of devices and combine disparate services.

Mr Donovan boasted that the Foundry had 106 projects under way and that AT&T was now measuring itself on the openness of its network: 82 APIs (or access points into the network) had been launched over the past year.

Start-up demonstrations at both the AT&T and Vodafone opening events hinted at the “mash-ups” that were possible through mobile networks – for example, how an app recording running performances could link up with ones relaying sensor information on a person’s health.

Augmented reality applications, improved graphics for gaming and the ability for machines to talk to other machines – “the internet of things” – could all benefit from being hard-wired into the operators’ networks, the demonstrations proved.

It remains to be seen whether developers will be attracted in significant numbers to working with the operators, but Ms Arjomandi argues all parties can benefit.

“We would like to have the advantage of [perhaps one-year exclusivity] lead time to the market – to have a differentiation for us, we need to get that lead time,” she said.

“In exchange, we give them access to 340m customers around the world in 30-plus countries – so it’s definitely a win-win situation.”

Thursday, September 29, 2011

Big Data Take Center Stage at Health 2.0 Conference

George Lauer, iHealthBeat Contributing Editor September 29, 2011


SAN FRANCISCO -- If the Health 2.0 movement is perceived as an ongoing conversation, the first four years could be seen as determining who was going to be talking and what the means of communication was going to be. Now, in year five, the focus is turning to what, exactly, everybody's going to be talking about.

"Data is everything," said Health 2.0 cofounder Mathew Holt at the Fifth Annual Health 2.0 Conference this week.


Raw information -- the gathering of it, collating, crunching and ultimately putting it to work -- was a major theme of this year's event.


"If we see data in health care as this huge glacier, we really are just now coming to see the tip of the iceberg," said Indu Subaiya, cofounder with Holt. Together, in 2007, they launched the first Health 2.0 event aimed at sparking innovative ways to get consumers involved in actively managing their own health care. Between 400 and 500 people ("mostly techies and nerds," according to Holt) showed up for the first conference. This week, more than 1,500 people from several countries attended.


"We're getting to the stage that is kind of the holy grail of Health 2.0, where the rubber meets the road," Subaiya said. "This year at Health 2.0 we're really beginning to see the intelligent mining of big data that can truly change how the system works. It's all around us, this data, but we have so much more work to do to link it and make use of it."


The kind of data examined during the conference ranged from tiny sensors and devices to monitor heart rate and glucose levels to "Big Data" gathered by the government and huge corporations to analyze big-picture situations ranging from flu epidemics to payer and provider databases.


"Data at every level is going to be what drives things forward," Holt said. 


"Almost everything we talk about here is being generated and displayed on the Internet, and we're only talking about a small, tiny fraction of what's out there," Holt added.


Government Embracing Health 2.0
What started as a grassroots effort to get patients involved with health IT has landed the federal government as a partner. Two HHS representatives participated in the Health 2.0 event this week -- Farzad Mostashari, national coordinator for health IT, and Lygeia Ricciardi, senior adviser for consumer eHealth at the Office of the National Coordinator for Health IT.


"We are embracing this with the passion of a recent convert," Mostashari said during a patient-centered panel presentation. "We come to this community with humility. This is not something that I've been at the forefront fighting for, but I really do believe we're on the right track."


Ricciardi urged attendees to check out the government's new offerings at HealthIT.gov, including a new educational campaign, "Putting the I in Health IT."


"We had this big formal launch in Washington two weeks ago as part of National Health IT Week," Ricciardi said. "One side (of the site's home page) is set up for providers and professionals and the other for patients and families. There's a whole body of information on health IT for the general public to understand it better," Ricciardi added.


Health 2.0 and ONC announced the launch of another in its series of challenges to health care technology innovators -- this one an invitation to multidisciplinary teams of developers to create a user-friendly way for patients to manage cardiovascular health using health IT. The project is part of the larger Health 2.0 Developer Challenge, a partnership between ONC and Health 2.0 designed to spur innovations in the use of technology to improve health care outcomes.


Earlier this month, ONC and Health 2.0 launched two competitions to encourage development of health IT applications to improve outcomes in patients transitioning from hospital to home and to facilitate reporting of adverse events related to medical devices.


'About To Change the World'
Although health care faces many obstacles in trying to bring the industry into the 21st century, the people and ideas involved in Health 2.0 are "on the verge of doing all the things that are about to change the world," according to Mark Smith, president and CEO of the California HealthCare Foundation and the conference's keynote speaker. CHCF publishes California Healthline.


Compared to other parts of modern life, such as banking and research, "in the trenches of American medical practice, we are still about 20 years behind," Smith said.


The good news, Smith said, is that technology is maturing and policy is evolving. The bad news is that time is running out in the race to fix an "enormously complex" health care system with "perverse incentives."
Smith pointed to Bear Stearns and Lehman Brothers, investment banks that folded in the credit crisis of 2008, as examples of "what happens when you run out of time."


"The fact of the matter is we're seeing lots of niceties in health policy swamped by fiscal realities," Smith said. "The amount of time we have is dwindling."


In November 2010, CHCF launched a $10 million fund to support health care innovation.
"We are interested in helping to stimulate innovations that can get some kind of market traction," Smith told a room full of entrepreneurs.


To illustrate the kinds of ideas that don't get traction, Smith showed a slide of the cartoon South Park's Underpants Business Model:
·       Phase One -- Collect underpants
·       Phase Two -- ?????
·       Phase Three -- Profits!!!


"For a lot of health care ideas, that's really what the business model amounts to," Smith said. CHCF wants "to fund successful companies because we're tired of seeing good ideas dying on the vine."


The CHCF Health Innovation Fund invests in for-profit and not-for-profit companies and organizations, with a focus on start-ups. Investments range from seed grants of $50,000 to as much as $3 million over the life of a company or service.


Sampling of Launches, Rollouts
Dozens of new products, projects, partnerships and campaigns were announced during the two-day conference, including:

·       Health eVillages, a partnership between the Robert F. Kennedy Center for Justice and Human Rights and Massachusetts-based Physicians Interactive Holdings. The project aims to get mobile phones and handheld devices to medical professionals in poor, remote and underserved parts of the world. So far, the organization has mounted pilot projects in Haiti, Uganda, Kenya and the U.S. Gulf Coast;

·       Society for Participatory Medicine, which encourages patients to take an active role in their health care and encourages physicians to make that happen, announced a campaign to get 10,000 physicians enrolled in a program to recieve the Society Seal, proclaiming their compliance with three basic tenets of the organization -- providing patients with all medical data, convening a patient advisory group, soliciting input from patients and providing resources to patients; and

·       Clarimed, billed as a "first-of-its-kind" health care rating organization, launched its online rating service. Calling itself the first independent rating agency for medical devices, diseases and health care products and services "in the gap that exists between scientific portals and medical websites for consumers," Clarimed was formed in April, launched a beta site in June and went live this week.


MORE ON THE WEB
·       Health 2.0 San Francisco 2011
·       HealthIT.gov


Read more: http://www.ihealthbeat.org/features/2011/data-take-center-stage-at-health-2-0-conference.aspx#ixzz1ZMzOTLWZ

Information explosion: how rapidly expanding storage spurs innovation

Lee Hutchinson   Ars Technica  | Published a day ago

Moore's Law gets all the press. It's easy to present even to non-technical readers, and the way it's most often expressed is something like, "computers double in speed every year," though that's a bastardization of the axiom, which actually states that the transistor count of integrated circuits tends to double every eighteen months or so. This formulation does succinctly capture how fast computers have gotten in so short a time.

But integrated circuit density hasn't been the only computing tech which has shown extremely rapid progress over the past thirty years. Consider magnetic storage. Modern hard drives are precisely manufactured miracles, products of billions of dollars and decades of research into magnetism and quantum mechanics, squeezing ludicrously large amounts of data into ludicrously tiny spaces. A hard drive with about three terabytes of capacity can be had for less than $150 today; a PC equipped with two or three of these would have more on-board storage than most large enterprises had in aggregate even a decade ago.


That kind of inexpensive capacity has revolutionized the way people keep and use data, both at home and at work. From complex storage- and compute-intensive tasks like oil and gas upstream processing all the way down to editing a home vacation video, the ability to store and manipulate increasingly voluminous data actually drives serious innovation.

A PCM .wav file of a three-minute song might be 30 or 40 MB, while an mp3 of the same song might be 3 MB. If all you've got is a 100 MB hard disk to work with, even the ten-fold decrease in file size doesn't make keeping an mp3 collection practical. But in 1995-6, when mp3s began to flow back and forth on the USENET and FTP and websites, the format suddenly became an extremely attractive way to collect music. With most folks having a handful of gigabytes, having an entire music collection available on a computer suddenly became possible. Copious local storage paved the way for entire new industries—think of early iPods and the iTunes store.

(Despite the sea change currently underway to solid state disks, there's every indication that spinning magnetic storage will continue to be with us for years to come—Seagate already sells a 4 TB external 3.5-inch hard drive.)

But data in and of itself doesn't do much. In this era of YouTube and Facebook, data is useless without a means to transport it from one place to another, a way to share it with others. Unfortunately, data transport has not been an area where the brain-smashing innovation race in integrated circuits and magnetic storage has been mirrored. Just as the density explosion in storage has opened the door to people and companies doing tremendous new things with computers, the lack of commensurate scaling in commonly available telecommunications has greatly hamstrung people and industries.

This has led to one of the most annoying paradoxes of the Internet age. On one hand, the massive increase in storage density encourages creation at all layers, from the biggest of businesses leveraging the ability to analyze and mine targeted data out of petabytes of raw information down to the adorable old couple trying to make a video on their home computer (both of these things can eat up a lot of hard drive space!).

On the other hand, the asymmetrical nature of most broadband solutions available to consumers in the US and Europe and a stagnation in their speed encourages only consumption at the "lower" levels of that stack. Companies that need both the ability to transmit and receive data over distance can usually afford to pay for symmetrical high-speed network links, while consumers (at least in the US) typically can pick from two choices for Internet access—DSL or cable. Both access methods typically provide plenty of download bandwidth for Netflixing and iTunesing and YouTubeing, but comparatively tiny upload bandwidth for sending data (most DSL and cable Internet plans have upload speeds that are less than 25 percent of the download speeds).

This asymmetry of access leads us to a strange place, where most folks have the ability to store and create more amazing things than ever before, while at the same time they lack the ability to quickly and easily share any of those things with each other.

Storage through the decades


Let's take a look back at a typical consumer's (i.e., my) hard disk drive at five-year intervals over the past three decades:

The chart on the top shows what appears to be a roughly linear progression of capacity in storage, but the vertical axes actually increase in powers of ten, showing almost a true exponential growth curve. To better appreciate the growth, consult the chart on the bottom, which uses fixed vertical axes. Starting at 5 MB in 1981, we jump quickly to 20 MB in five years, then to 120 MB in five more years, then to 2 GB, to 40 GB, to 500 GB, and finally to 3 TB.
The capacity increases get truly staggering near the end. This alone doesn't tell us much, since it's an accepted bit of conventional wisdom that hard drives will always grow larger over time. The important thing about this increase in storage capacity isn't that it has let us store more data, but that having fast random access to more data lets us do more with that data.

No one keeps files for the sake of keeping files. Even digital hoarders who have every single episode of Doctor Who ever created don't simply have those files to have files—they have those files because they comprise a collection of data with value as a collection. An Excel or Word document has no value in and of itself; rather, documents gain value when they are consulted or shared, when the data that they contain is compared against or integrated with other data and transformed into new data. With this in mind, the exponential increase in storage density coupled with a decrease in how much it costs per-gigabyte to purchase that storage has driven a huge shift in what it means to create and innovate.

In the early '80s, microcomputers like the IBM PC and the Apple II were used largely for traditional computing tasks—crunching numbers or programming applications that would be used for crunching numbers. (Well, that and games.) A 5MB ProFile hard drive for an Apple Lisa cost about $3,500 when it first became available for that system in 1981, but the price could be justified by the fast and random access that ProFile offered to what at the time was a significant amount of data (equivalent to dozens of double-sided floppy disks). Because of the cost, though, home computers didn't typically have internal hard disk drives until the late 1980s. As their storage capacities increased, so did the things that computers were typically used to do.

Serious music production at home, a trend that arguably got started with module tracking on the Amiga platform and spread rapidly in the early 1990s to the PC, relied on bigger hard drives to hold the digital samples that made up the music files. Accelerating storage density pushed this kind of music production out of the basement and into the home studio, today letting artists like Pomplamoose record and produce entire albums on personal computers without relying on time in hugely expensive recording studios. Entry barriers into a recording career are being quickly demolished and a tidal wave of skilled and talented artists are appearing, doing things that would have been impossible even a few years ago.

Terabytes of fast disks available in most home computers also enable software like iMovie and Windows Movie Maker to exist and thrive, bringing movie-creation tools to the average home user. Not everyone is a talented auteur with a Citizen Kane waiting inside —in fact, most folks tend to create videos of their babies farting or themselves farting or animals farting—but folks like Neill Blomkamp or Bruce Branit have also risen above the noise to create some amazing things.

This sort of storage-driven innovation holds true for the enterprise just as much as it does for the home—the fact that I've been linking to YouTube videos above is telling. Companies like Google (which owns YouTube) wouldn't exist without huge amounts of storage. Google began out of a need to locate and organize information, and it manages and stores petabytes and petabytes of data. Even more to the point, it does so in a way that underlines the commoditzation of information technology in general—most big enterprises these days store data in large monolithic storage arrays, but Google keeps all of its data on throwaway servers with consumer-grade hard disks, relying on redundancy and smart design to route around failed components. As storage densities continue to increase and the transformational things that people can do with that storage scales, more and more enterprises will shift in Google's direction.

Advanced trending and data analytics can also be done now with more data available—for example, T-Mobile was able to take the huge amount of network and subscriber data they store, sift it, and come up with insights into why customers leave (disclaimer: the linked article mentions EMC, and I'm an EMC employee). Having more data at hand lets us make connections or create new things that simply wouldn't have been possible without massive storage.
There's a darker side to all of this shiny creativity, though. It can still be too hard to share it with the world.

Storage enough at last?
There's a famous episode of The Twilight Zone named "Time Enough at Last," wherein a bookish fellow played by Burgess Meredith laments that there's not enough time to indulge in reading. He decides to take his lunch break in the vault of the bank where he works, and while he's down there, the Commies nuke everything. He emerges from the vault stunned at the ruins around him and, finding no one alive but himself, eventually decides to take his own life. Instead, though, he spies a library in the distance; he approaches it and, finding that it's stocked full of undamaged books, realizes that his prayers have been answered—he finally has time, time enough at last to read all he wants! The hook, though—spoiler alert from 1959!—is that as he sits down to read his first book, he breaks his glasses. The episode closes with Burgess howling about the unfairness of it all, amidst stacks of books that he will now be forever unable to read.

This parallels the situation most consumers (at least within the US and much of Europe) find themselves today. We have, finally, storage enough at last to do just about anything we want. We can create and keep terabytes of digital stuff, but the value of that stuff is in what you can do with it, and a home movie doesn't do you any good sitting on your hard drive—its value comes from sharing it. Increasingly, that sharing takes place over the Internet.

And sharing is a problem. Let's take a look at the typical amount of inbound and outbound network bandwidth a US home might have had for the same years in which we look at hard drive capacity. Internet connections to the home weren't terribly common before about 1994, but those of us old enough to experience the days before USENET and the Web will remember local BBS systems and fledgling "information services" like Prodigy and Compuserve, all of which served many of the same purposes then that the Internet does now—connecting with others and exchanging files and messages.

From 300bps in 1981, we travel through 1200bps and the halcyon days of local BBSs at 14.4Kbps, through the brief 56K era and into broadband, going from 1Mb to 3Mb to the 10Mb (and beyond) connections common today. The same kind of exponential-ish increase in speeds is evident, especially in the break between 1996 and 2001 when the widespread shift to broadband connectivity began to pull folks out of needing to use modems over POTS and onto DSL and cable Internet access. However, with that came huge gulfs between the speed at which a person could download things to his computer versus the speed at which he could upload things from it to others.

To be sure, upload/download asymmetry has been around for a long time. Back in my own BBS days, I used to look with envy on the guys with USRobotics Courier HST Dual Standard modems, which could talk to other Courier HST modems at a ridiculously fast (for the time) 16.8Kbps and at the same time have a 450bps back-channel available for control traffic. Similarly, most folks who lived through the 56K days remember that 56K was really only one way—traffic going out from your modem was still limited to 33.8Kbps (and in reality, most folks only saw 22-26 Kbps). But nowhere is the asymmetry as great as with modern cable and DSL connections. The practical reason behind the split between up and down speeds is one of frequency utilization—there's only so many hertz available to carry a signal, and so more of the available frequency space is allocated to carry information to the consumer than from the consumer.

Unfortunately, this common asymmetric split now affects a good chunk of the world's Internet users.

You can't take it with you
Most folks carry what feels like significant chunks of our lives on our hard drives—we express ourselves through the things we create, and we keep things that have powerful meaning for us. A hard drive crash where data is lost can lead to not just technical or financial problems, but actual personal anguish—our drives are surrogates for our memories, and if they die we can lose parts of ourselves.

And yet, with all the innovation that has arrived as a consequence of more and faster data storage, most lack the ability to easily move that data around. We can download multi-gigabyte high definition movies from iTunes or NetFlix in astonishingly short amounts of time, but how many hours does it take to upload a multi-gigabyte file?
Amount of time it would take to upload an entire hard drive at each point in time


Companies like Microsoft and Apple and others continue to roll out cloud sync services, which among other things suggest you put your important files "in the cloud" so they can be accessible anywhere—but why should that be necessary when they could just as easily be accessed in situ on your own computer? I have hundreds of movies that I've ripped from DVDs that I own just sitting on my NAS; why shouldn't I be able to access them wherever I am and stream them to myself at a friend's house as fast as NetFlix can? Why shouldn't I be able to get to any part of my digital identity from anywhere in the world?

Cloud backup services like Backblaze or Mozy have their place, too, providing off-site storage for important data in case of a disaster, but it can take weeks—literally weeks—to upload a few terabytes of data. Most people today don't bother with this kind of backup service, precisely because it takes so long to populate initially. Worse still, for people with metered Internet connections, that initial data copy will almost certainly count against your 50 or 150 or 250 GB monthly cap, which might lead to your Internet access being terminated without explanation by your ISP.
The lack of upstream bandwidth in a typical consumer broadband connection holds back some of the innovation that could come from huge amounts of storage. For instance, with a 10 Mb upload connection to go along with your 10 Mb download connection, content sharing sites like YouTube suddenly lose some of their relevance. "Digital locker" services like RapidShare and MegaUpload serve fewer purposes. An entire infrastructure of sites and services that have evolved as buffers and workarounds for tiny upload speeds are made redundant and can be done away with.
Do more with more
Even if our wings remain clipped by upload restrictions, we're still innovating through storage. We have more and faster storage available to us at home and at work, and the things we can do with that capacity continue to evolve. Parkinson's Law states that "work expands so as to fill the time available for its completion." A common corollary to that law is that "data expands to fill the space available for storage." Put even more simply, as the ghostly voice said to Kevin Costner in Field of Dreams, "If you build it, they will come."
Nothing exists in a vacuum, of course, and it's disingenuous to suggest that increases in storage capacity can alone be responsible for anything, much in the way that it's wrong to say an increase in CPU clock speeds alone is responsible for anything, or that the change from IDE to PCI buses in a PC was by itself responsible for anything. However, the rapid increase in storage capacities to which the average consumer has access has removed barriers and created big new opportunities.


Wednesday, September 14, 2011

Six Provocations for Big Data

Six Provocations for Big Data
danah boyd, Microsoft Research; University of New South Wales (UNSW); Harvard University - Berkman Center for Internet & Society
Kate Crawford, University of New South Wales (UNSW)

September 21, 2011

Abstract:
   
The era of Big Data has begun. Computer scientists, physicists, economists, mathematicians, political scientists, bio-informaticists, sociologists, and many others are clamoring for access to the massive quantities of information produced by and about people, things, and their interactions. Diverse groups argue about the potential benefits and costs of analyzing information from Twitter, Google, Verizon, 23andMe, Facebook, Wikipedia, and every space where large groups of people leave digital traces and deposit data. Significant questions emerge.



Will large-scale analysis of DNA help cure diseases? Or will it usher in a new wave of medical inequality? Will data analytics help make people’s access to information more efficient and effective? Or will it be used to track protesters in the streets of major cities? Will it transform how we study human communication and culture, or narrow the palette of research options and alter what ‘research’ means? Some or all of the above?

This essay offers six provocations that we hope can spark conversations about the issues of Big Data. Given the rise of Big Data as both a phenomenon and a methodological persuasion, we believe that it is time to start critically interrogating this phenomenon, its assumptions, and its biases.

(This paper was presented at Oxford Internet Institute’s “A Decade in Internet Time: Symposium on the Dynamics of theInternet and Society” on September 21, 2011.)

Tuesday, September 13, 2011

Big Data: Spy Agency Seeks Digital Mosaic to Divine Future

The U.S. intelligence community wants to mine lots and lots of the tidbits bopping around on the Internet to suss out trends before they make the news.

Emily Badger     Miller McCune   September 9, 2011

U.S. intelligence agencies hope those tidbits bopping around on the Internet will help discover all kinds of trends before they make the news. (John Foxx/Stockbyte)

Governments have been caught off guard a lot lately: by revolutions, by riots, even by unemployment rates (or, to go back even further, by events like 9/11).

In the information age — where there's no limit to publicly available data on everything from political chatter to gas prices — it seems policymakers should be better at predicting major societal shifts and events than at any point in history. Shouldn't all these little pieces of information be telling us something big? Shouldn't they be telling us about where the next mass migration will come from or where the next riot will be?

The government's Office of the Director of National Intelligence is betting this is the case. Its Intelligence Advanced Research Projects Activity ( or IARPA — which sounds like a less menacing cousin to DARPA) is rolling out a new R&D project to test tools that would mine publicly available data to predict political and humanitarian crises, disease outbreaks, mass violence and instability. The project, the Open Source Indicators Program, is premised on the idea that big events are preceded by population-level changes, and that those population-level changes should be identifiable if we just look in the right places.

Idea Lobby
THE IDEA LOBBY
Miller-McCune's Washington correspondent Emily Badger follows the ideas informing, explaining and influencing government, from the local think tank circuit to academic research that shapes D.C. policy from afar.

The concept isn't new. Allied intelligence officers did something similar during World War II, for example, mining letters to the editors of local newspapers and radio transmissions for clues as to what was going on inside Nazi Germany.

"The difference, the big difference — and the thing this initiative is picking up on — is that whereas we used to have to do this with a relatively small sample of radio shows and newspapers and stuff like that, we're now drinking from the fire hose of the Internet," said Philip Schrodt, a political scientist at Penn State.

An IARPA public affairs officer said officials could not discuss the program while the government is still soliciting proposals. But Schrodt and several other researchers affiliated with academic teams that may eventually wind up working on the project, alongsideprivate contractors, offered a look into an expanding, multidisciplinary field of real-time data analysis that the government hopes may allow it to "beat the news."

This experiment — which will test methods in Latin America, not the U.S. — dovetails with broader trends in computer and social science toward mining large-scale sets of social data. Twitter, for instance, is a jackpot: The network can be scoured both for the content of messages and the connections that are revealed between people writing and reading them.


About half of the populations in most major Western countries are now on Facebook (and a surprising amount of data on Facebook remains accessible to the public). Economic data can be collected from e-commerce sites like Amazon. There are also open-source indicators embedded in Internet news sites, message boards, unemployment data, Web search queries, traffic Web cams and financial markets.

Think of what researchers could learn about the economic mood of consumers if they suddenly discovered hundreds of people (each anonymously identified) all trying to sell their best jewelry on Craigslist tomorrow. Other open-source indicators could provide a trove of information on a topic social scientists have been weighing a lot lately: the effect of price changes — whether for housing, gas or food — on social stability.

And because all this data can be collected in real time and by automated systems, it's more up-to-date, it's larger in sheer quantity, and it's becoming more cost-effective and realistic than ever to analyze.
• • • • • • • • • • • • • • •
The easiest place to understand the potential of all this data is in public health, where live analysis is already helping to identify and track disease epidemics at a speed that was never possible before the Internet. Google Flu Trends pioneered the technique. The tool measures the frequency of certain search terms commonly associated with the flu to identify outbreaks down to the city level. Now it's doing the same with dengue fever.

Researchers at Harvard developed a similar tool five years ago calledHealthMap, which offers a prototype of the model IARPA envisions testing beyond public health. HealthMap continuously scrapes the Web for health-related keywords and deletes noise, then classifies the data, geocodes it, filters it into different categories and maps the results. The project is particularly useful in identifying early disease indicators in countries that have no public health capacity and weak or opaque data reporting (because these are often the same countries where sick people are unlikely to Google their symptoms on a home computer, HealthMap also taps into SMS messages and smartphone data).

"These discussions are taking place by the minute, and we're caching those in real time," said John Brownstein, the director and co-founder of HealthMap. "Our goal is as soon as anybody's talking about an outbreak on the Web, within the hour it ends up on HealthMap."


In theory, similar methods might track contagious ideas, economic unrest or resource shortages — and in a way that would actually move faster than news reports.

Automated systems could certainly be more comprehensive than traditional media. The BBC and Associated Press don't have correspondents in every small town in Mexico, but an automated computer program could analyze in real time reports from every local news site in the country. Or, better yet, it could analyze social network chatter before it even gets to the local newsroom.

"If everybody is suddenly upset about the drought in Texas or something like that," said Schrodt, "we should be able to pick that up immediately without having some reporter go out and talk to some farmer whose cows have died."

This type of analysis could also help policymakers be less surprised by news when it happens, and to monitor it literally in real time.

The bigger question, though, is how researchers take the step from watching trends develop in a live time series to anticipating them before they happen.
Some trends lend themselves more easily to this challenge — disease epidemics are one. HealthMap can start to identify outbreaks before officials even know what disease they're looking at because the tool is designed to scan for basic symptoms like runny noses or a run on aspirin. HealthMap isn't merely crawling the Web for Google searches of "Do I have SARS?"

But how do we anticipate "riots in London" when we don't even know "riots in London" is what we're looking for? Could open-source indicators have identified the rise of the Tea Party before it even started calling itself that?

• • • • • • • • • • • • • • •
"That's the really, really big question, that's the single biggest challenge in this IARPA initiative," Schrodt said. He has worked on other government research projects, including one called the Political Instability Task Force. It's clear, though, in that project, what researchers are looking for: political instability, which is suggested by fewer than five indicators. Here, though, the goal is to identify everything that's publicly available, for anything that might be interesting.

"I can't take a book off my shelf and open it up and say, 'Oh, here's what you do if you're monitoring 100 different indicators using 200 different sources,'" Schrodt said. "That's the new science on this. I don't know if it will work or not." (The political instability project, as an example, has gotten to about 80 percent accuracy forecasting probabilities that countries will collapse.)
Peter Gloor's research at MIT's Center for Collective Intelligence has been trying to solve this unknown prediction problem by identifying the most creative people – both those who are destructively creative, like terrorist plotters, and those who are constructively creative, like Justin Bieber. He's trying to find the people behind trends — information producers, not the information itself — before trends take off.

"We're looking for the trendsetters while they are being born and made," Gloor said. "In Twitter, once you are Ashton Kutcher, everyone knows you; it's clear you are a trendsetter. But Justin Bieber, whenever he was posting his first video on YouTube, it was not as clear."

Gloor has done a lot of this work in Wikipedia, where pages are written and edited according to a revealing pattern: Generally, about 1 percent of contributors write 90 percent of the content; 9 percent of people write another 9 percent of the content; and 90 percent of contributors write just 1 percent of the crowdsourced encyclopedia. If you find the 1 percent of people who are doing all the heavy lifting, those are your future trendsetters (or "coolhunters").

Charles Elkan, a computer scientist at the University of California at San Diego, suspects the biggest value of all these tools will come from quantifying what social scientists already know.

"Social scientists have said for 100 years that revolutions happen at times of rising expectations," he said. "If, for example, society is becoming more prosperous, people start have rising economic expectations, that spills over into rising political expectations, and that can make a revolution more likely. Social scientists have said this for a long time, and now it's beginning to be possible to quantify it."

Finland, for example, is obviously a more stable country than Sudan. But attaching a number to that statement — say, Sudan is eight times more likely to experience a coup or revolution in the next two years — is much trickier.
• • • • • • • • • • • • • • •
One thing open-source indicators probably can't do is write the news before it happens.
"Predicting events is even harder than predicting trends," Elkan said. "If you think of a trend as creating the probability for something, that probability combined with some spark makes the actual event."

And then you have to predict the spark, too.

"We can say unemployment is trending upwards, and that layoffs are increasing," Elkan went on, "but it's still very difficult to predict which manager of which company will wake up in the morning and decide, 'I can't wait any longer, I need to lay off 10 people today.'"

It's possible to predict, for instance, that riots are more likely in London than they are in Orlando. But no one would have predicted a week ahead of time that a specific police shooting would catalyze riots (as plenty of other police shootings — even in other seething areas — don't set off riots). Similarly, no one would have predicted that the suicide of a Tunisian fruit vendor would spark a revolution that would sweep out of North Africa and into the Arabian Peninsula and Mideast.

In this sense, there may be a sizable gap between our imagination of what's possible with trend prediction and the reality of what it can do for policymakers.

The primary limitation isn't the data collection; it's the data analysis. And there's a point in the process where human judgment must take over from automated computer systems. That is inevitably the moment when controversial or expensive action looms: Should we double the FEMA budget, send support to Syrian revolutionaries or shift course on jobs policy to quell a domestic uprising?

The attention span of decision-makers, Elkan cautions, is a limited resource, too.

There's another factor: Governments aren't the only ones who might like to leverage this data. Its business application is obvious. Movie theaters could use trend prediction to book films. Department stores could use it to stock products. Corporations could mine Twitter chatter about them to craft corporate social responsibility policies.

"That's one reason to be skeptical about the extent of the possibility," Elkan said. "If it was really possible, for example, to predict unemployment much better than existing methods do, then people on Wall Street would be doing it and taking advantage of it."

Like the other researchers, he's cautious about where all of this is headed, even as he tries to design computer models to more accurately predict the future.

"We can easily specify, 'Let's monitor every newspaper and every radio station everywhere in the world, then as soon as there's one school that has a local outbreak of some disease, then we can put that into our computer models and make predictions of when it will come to the United States,'" Elkan said. "That's a realistic dream, but it's still a dream at this point."

Monday, September 12, 2011

Big Data : Data-Driven Decisions in Asthma Care

Asthmapolis: Tools to Track, Manage, and Research Asthma

GPS Sensors and Mobile App Track Asthma Symptoms, Triggers, and Inhaler Use

California Healthcare Foundation September 2011

Despite dramatic advances in medication effectiveness, many people with asthma experience episodes of uncontrolled disease, causing them to be at greater risk for acute exacerbations. Historically health care providers have had inadequate tools to understand the severity of a patient's asthma, limiting their ability to predict and prevent serious and costly asthma attacks. To address this problem Asthmapolis developed GPS sensors that record exactly when and where patients use their inhalers and a digital interface to display the information captured. The patient-level data may then help physicians make more informed clinical interventions and help patients track and manage their condition. 

With the support of the California HealthCare Foundation, Asthmapolis has begun a pilot with Woodland HealthCare, a member of Catholic HealthCare West, to determine if the data collected by sensors and presented to physicians and patients results in actions that improve asthma control. This study is being conducted with English- and Spanish-speaking subjects and will include underserved patients in the Sacramento area.

CHCF invested in Asthmapolis because we see the potential for patients to benefit from technology that provides tailored information, allowing them to better understand and manage asthma, which will hopefully lead to fewer serious and costly episodes.

Read more: http://www.chcf.org/projects/2011/asthmapolis#ixzz1Xm0uua3l