Five years ago, Samuel J. Palmisano, then chief executive of I.B.M., gave a speech at the Council on Foreign Relations on the big opportunity ahead for modern computing technology to help businesses, government agencies and other institutions operate more intelligently. His remarks captured an emerging trend that opened the way to a large, lucrative business for I.B.M.
On Thursday evening, his successor, Virginia M. Rometty, is scheduled to deliver a speech in the same setting, looking at how far technology has come and its implications for how business and government are managed.
In 2008, Mr. Palmisano spoke of the accelerating pace of computer processing, storage and software that exploited artificial intelligence. He pointed to the explosion in data from the Web, social networks and sensors. He mentioned I.B.M., if at all, only in passing.
But to the vision he sketched out Mr. Palmisano attached the term "Smarter Planet." It became the tagline for I.B.M.'s highly successful campaign to sell software and services — and sometimes hardware, too — to corporations and governments for an array of "smart" projects.
For Ms. Rometty, the script is similar in the sense that it is intended for an audience beyond the business community and refers to I.B.M. by name only once.
Still, the context is quite different. Mr. Palmisano gave his talk in the fall of 2008, when the financial crisis was sending the world's economies into a tailspin. His speech gained a wider audience only partly because of what he said about technology. He was the chief executive of a major global corporation declaring that it was a good time to invest for the long term rather than pull back in a defensive crouch.
Today the economy is growing, if slowly, and corporate America is thriving. Yet Ms. Rometty's speech — like her predecessor's — seeks to broaden the discussion of the opportunity and implications of technology trends. Hers, in some ways, is a more subtle message about the next step — the broad absorption of this advancing technology.
In an advance copy of the speech, Ms. Rometty points to three effects of the smart technology, which these days flies under the banner of Big Data. She calls them three lessons.
"Lesson 1: Decisions will be based not on 'gut instinct,' but on predictive analytics."
"Lesson 2: The social network is not just the new water cooler; it's the new production line."
"Lesson 3: Value will be created not for 'market segments' or demographics, but for individuals."
Each of the three themes or lessons is illustrated by examples: Memphis police using data analysis to curb crime; a Mexican cement maker tapping the crowdsourced intelligence of its workers in product development: and the Obama campaign's sifting of data to pinpoint get-out-the-vote efforts.
Ms. Rometty concludes by saying, "The challenge is not the technology. The challenge, as always, is culture — changing our entrenched ways of thinking, acting and organizing."
Her point echoes a message management experts have been making for some time. I shared a copy of the speech with Erik Brynjolfsson, director of the Center for Digital Business at MIT's Sloan School of Management.
In an e-mail, Mr. Brynjolfsson said, "I agree very much with the speech. The technology has been available for a few years now to create a management revolution based on big data, and now we're beginning to see more and more companies undertake the much harder job of reinventing their business processes and culture to take full advantage of those technologies."
Any speech by an executive, especially one that aspires to reach a wider audience, walks a line: How much is it fresh information and thoughtful insight versus a self-serving sales pitch?
Decide for yourself. Ms. Rometty's speech is scheduled to begin at 6 P.M. Eastern time on Thursday and to be webcast live on the Council on Foreign Relations site and on I.B.M.'s Smarter Planet blog.
By 7:30 P.M. or so, the text of Ms. Rometty's speech should also be on the I.B.M. blog.
The digital pioneer and visionary behind virtual reality has turned against the very culture he helped create
Ron Rosenbaum, Smithsonian, January 2013
I couldn’t help thinking of John Le Carré’s spy novels as I awaited my rendezvous with Jaron Lanier in a corner of the lobby of the stylish W Hotel just off Union Square in Manhattan. Le Carré’s espionage tales, such as The Spy Who Came In From the Cold, are haunted by the spectre of the mole, the defector, the double agent, who, from a position deep inside, turns against the ideology he once professed fealty to.
And so it is with Jaron Lanier and the ideology he helped create, Web 2.0 futurism, digital utopianism, which he now calls “digital Maoism,” indicting “internet intellectuals,” accusing giants like Facebook and Google of being “spy agencies.” Lanier was one of the creators of our current digital reality and now he wants to subvert the “hive mind,” as the web world’s been called, before it engulfs us all, destroys political discourse, economic stability, the dignity of personhood and leads to “social catastrophe.” Jaron Lanier is the spy who came in from the cold 2.0.
To understand what an important defector Lanier is, you have to know his dossier. As a pioneer and publicizer of virtual-reality technology (computer-simulated experiences) in the ’80s, he became a Silicon Valley digital-guru rock star, later renowned for his giant bushel-basket-size headful of dreadlocks and Falstaffian belly, his obsession with exotic Asian musical instruments, and even a big-label recording contract for his modernist classical music. (As he later told me, he once “opened for Dylan.” )
The colorful, prodigy-like persona of Jaron Lanier—he was in his early 20s when he helped make virtual reality a reality—was born among a small circle of first-generation Silicon Valley utopians and artificial-intelligence visionaries. Many of them gathered in, as Lanier recalls, “some run-down bungalows [I rented] by a stream in Palo Alto” in the mid-’80s, where, using capital he made from inventing the early video game hit Moondust, he’d started building virtual-reality machines. In his often provocative and astute dissenting book You Are Not a Gadget, he recalls one of the participants in those early mind-melds describing it as like being “in the most interesting room in the world.” Together, these digital futurists helped develop the intellectual concepts that would shape what is now known as Web 2.0—“information wants to be free,” “the wisdom of the crowd” and the like.
And then, shortly after the turn of the century, just when the rest of the world was turning on to Web 2.0, Lanier turned against it. With a broadside in Wired called “One-Half of a Manifesto,” he attacked the idea that “the wisdom of the crowd” would result in ever-upward enlightenment. It was just as likely, he argued, that the crowd would devolve into an online lynch mob.
Lanier became the fiercest and weightiest critic of the new digital world precisely because he came from the Inside. He was a heretic, an apostate rebelling against the ideology, the culture (and the cult) he helped found, and in effect, turning against himself.
***
And despite his apostasy, he’s still very much in the game. People want to hear his thoughts even when he’s castigating them. He’s still on the Davos to Dubai, SXSW to TED Talks conference circuit. Indeed, Lanier told me that after our rendezvous, he was off next to deliver the keynote address at the annual meeting of the Ford Foundation uptown in Manhattan. Following which he was flying to Vienna to address a convocation of museum curators, then, in an overnight turnaround, back to New York to participate in the unveiling of Microsoft’s first tablet device, the Surface.
Lanier freely admits the contradictions; he’s a kind of research scholar at Microsoft, he was on a first-name basis with “Sergey” and “Steve” (Brin, of Google, and Jobs, of Apple, respectively). But he uses his lecture circuit earnings to subsidize his obsession with those extremely arcane wind instruments. Following his Surface appearance he gave a concert downtown at a small venue in which he played some of them.
Lanier is still in the game in part because virtual reality has become, virtually, reality these days. “If you look out the window,” he says pointing to the traffic flowing around Union Square, “there’s no vehicle that wasn’t designed in a virtual-reality system first. And every vehicle of every kind built—plane, train—is first put in a virtual-reality machine and people experience driving it [as if it were real] first.”
I asked Lanier about his decision to rebel against his fellow Web 2.0 “intellectuals.”
“I think we changed the world,” he replies, “but this notion that we shouldn’t be self-critical and that we shouldn’t be hard on ourselves is irresponsible.”
For instance, he said, “I’d been an early advocate of making information free,” the mantra of the movement that said it was OK to steal, pirate and download the creative works of musicians, writers and other artists. It’s all just “information,” just 1’s and 0’s.
Indeed, one of the foundations of Lanier’s critique of digitized culture is the very way its digital transmission at some deep level betrays the essence of what it tries to transmit. Take music.
“MIDI,” Lanier wrote, of the digitizing program that chops up music into one-zero binaries for transmission, “was conceived from a keyboard player’s point of view...digital patterns that represented keyboard events like ‘key-down’ and ‘key-up.’ That meant it could not describe the curvy, transient expressions a singer or a saxophone note could produce. It could only describe the tile mosaic world of the keyboardist, not the watercolor world of the violin.”
Quite eloquent, an aspect of Lanier that sets him apart from the HAL-speak you often hear from Web 2.0 enthusiasts (HAL was the creepy humanoid voice of the talking computer in Stanley Kubrick’s prophetic 2001: A Space Odyssey). But the objection that caused Lanier’s turnaround was not so much to what happened to the music, but to its economic foundation.
I asked him if there was a single development that gave rise to his defection.
“I’d had a career as a professional musician and what I started to see is that once we made information free, it wasn’t that we consigned all the big stars to the bread lines.” (They still had mega-concert tour profits.)
“Instead, it was the middle-class people who were consigned to the bread lines. And that was a very large body of people. And all of a sudden there was this weekly ritual, sometimes even daily: ‘Oh, we need to organize a benefit because so and so who’d been a manager of this big studio that closed its doors has cancer and doesn’t have insurance. We need to raise money so he can have his operation.’
“And I realized this was a hopeless, stupid design of society and that it was our fault. It really hit on a personal level—this isn’t working. And I think you can draw an analogy to what happened with communism, where at some point you just have to say there’s too much wrong with these experiments.”
His explanation of the way Google translator works, for instance, is a graphic example of how a giant just takes (or “appropriates without compensation”) and monetizes the work of the crowd. “One of the magic services that’s available in our age is that you can upload a passage in English to your computer from Google and you get back the Spanish translation. And there’s two ways to think about that. The most common way is that there’s some magic artificial intelligence in the sky or in the cloud or something that knows how to translate, and what a wonderful thing that this is available for free.
“But there’s another way to look at it, which is the technically true way: You gather a ton of information from real live translators who have translated phrases, just an enormous body, and then when your example comes in, you search through that to find similar passages and you create a collage of previous translations.”
“So it’s a huge, brute-force operation?” “It’s huge but very much like Facebook, it’s selling people [their advertiser-targetable personal identities, buying habits, etc.] back to themselves. [With translation] you’re producing this result that looks magical but in the meantime, the original translators aren’t paid for their work—their work was just appropriated. So by taking value off the books, you’re actually shrinking the economy.”
The way superfast computing has led to the nanosecond hedge-fund-trading stock markets? The “Flash Crash,” the “London Whale” and even the Great Recession of 2008?
“Well, that’s what my new book’s about. It’s called The Fate of Power and the Future of Dignity, and it doesn’t focus as much on free music files as it does on the world of finance—but what it suggests is that a file-sharing service and a hedge fund are essentially the same things. In both cases, there’s this idea that whoever has the biggest computer can analyze everyone else to their advantage and concentrate wealth and power. [Meanwhile], it’s shrinking the overall economy. I think it’s the mistake of our age.”
The mistake of our age? That’s a bold statement (as someone put it in Pulp Fiction). “I think it’s the reason why the rise of networking has coincided with the loss of the middle class, instead of an expansion in general wealth, which is what should happen. But if you say we’re creating the information economy, except that we’re making information free, then what we’re saying is we’re destroying the economy.”
The connection Lanier makes between techno-utopianism, the rise of the machines and the Great Recession is an audacious one. Lanier is suggesting we are outsourcing ourselves into insignificant advertising-fodder. Nanobytes of Big Data that diminish our personhood, our dignity. He may be the first Silicon populist.
“To my mind an overleveraged unsecured mortgage is exactly the same thing as a pirated music file. It’s somebody’s value that’s been copied many times to give benefit to some distant party. In the case of the music files, it’s to the benefit of an advertising spy like Google [which monetizes your search history], and in the case of the mortgage, it’s to the benefit of a fund manager somewhere. But in both cases all the risk and the cost is radiated out toward ordinary people and the middle classes—and even worse, the overall economy has shrunk in order to make a few people more.”
Lanier has another problem with the techno-utopians, though. It’s not just that they’ve crashed the economy, but that they’ve made a joke out of spirituality by creating, and worshiping, “the Singularity”—the “Nerd Rapture,” as it’s been called. The belief that increasing computer speed and processing power will shortly result in machines acquiring “artificial intelligence,” consciousness, and that we will be able to upload digital versions of ourselves into the machines and achieve immortality. Some say as early as 2020, others as late as 2045. One of its chief proponents, Ray Kurzweil, was on NPR recently talking about his plans to begin resurrecting his now dead father digitally.
Some of Lanier’s former Web 2.0 colleagues—for whom he expresses affection, not without a bit of pity—take this prediction seriously. “The first people to really articulate it did so right about the late ’70s, early ’80s and I was very much in that conversation. I think it’s a way of interpreting technology in which people forgo taking responsibility,” he says. “‘Oh, it’s the computer did it not me.’ ‘There’s no more middle class? Oh, it’s not me. The computer did it.’
“I was talking last year to Vernor Vinge, who coined the term ‘singularity,’” Lanier recalls, “and he was saying, ‘There are people around who believe it’s already happened.’ And he goes, ‘Thank God, I’m not one of those people.’”
In other words, even to one of its creators, it’s still just a thought experiment—not a reality or even a virtual-reality hot ticket to immortality. It’s a surreality.
Lanier says he’ll regard it as faith-based, “Unless of course, everybody’s suddenly killed by machines run amok.”
“Skynet!” I exclaim, referring to the evil machines in the Terminator films.
At last we come to politics, where I believe Lanier has been most farsighted—and which may be the deep source of his turning into a digital Le Carré figure. As far back as the turn of the century, he singled out one standout aspect of the new web culture—the acceptance, the welcoming of anonymous commenters on websites—as a danger to political discourse and the polity itself. At the time, this objection seemed a bit extreme. But he saw anonymity as a poison seed. The way it didn’t hide, but, in fact, brandished the ugliness of human nature beneath the anonymous screen-name masks. An enabling and foreshadowing of mob rule, not a growth of democracy, but an accretion of tribalism.
It’s taken a while for this prophecy to come true, a while for this mode of communication to replace and degrade political conversation, to drive out any ambiguity. Or departure from the binary. But it slowly is turning us into a nation of hate-filled trolls.
Surprisingly, Lanier tells me it first came to him when he recognized his own inner troll—for instance, when he’d find himself shamefully taking pleasure when someone he knew got attacked online. “I definitely noticed it happening to me,” he recalled. “We’re not as different from one another as we’d like to imagine. So when we look at this pathetic guy in Texas who was just outed as ‘Violentacrez’...I don’t know if you followed it?”
“I did.” “Violentacrez” was the screen name of a notorious troll on the popular site Reddit. He was known for posting “images of scantily clad underage girls...[and] an unending fountain of racism, porn, gore” and more, according to the Gawker.com reporter who exposed his real name, shaming him and evoking consternation among some Reddit users who felt that this use of anonymity was inseparable from freedom of speech somehow.
“So it turns out Violentacrez is this guy with a disabled wife who’s middle-aged and he’s kind of a Walter Mitty—someone who wants to be significant, wants some bit of Nietzschean spark to his life.”
Only Lanier would attribute Nietzschean longings to Violentacrez. “And he’s not that different from any of us. The difference is that he’s scared and possibly hurt a lot of people.”
Well, that is a difference. And he couldn’t have done it without the anonymous screen name. Or he wouldn’t have.
And here’s where Lanier says something remarkable and ominous about the potential dangers of anonymity.
“This is the thing that continues to scare me. You see in history the capacity of people to congeal—like social lasers of cruelty. That capacity is constant.”
“Social lasers of cruelty?” I repeat.
“I just made that up,” Lanier says. “Where everybody coheres into this cruelty beam....Look what we’re setting up here in the world today. We have economic fear combined with everybody joined together on these instant twitchy social networks which are designed to create mass action. What does it sound like to you? It sounds to me like the prequel to potential social catastrophe. I’d rather take the risk of being wrong than not be talking about that.”
Here he sounds less like a Le Carré mole than the American intellectual pessimist who surfaced back in the ’30s and criticized the Communist Party he left behind: someone like Whittaker Chambers.
But something he mentioned next really astonished me: “I’m sensitive to it because it murdered most of my parents’ families in two different occasions and this idea that we’re getting unified by people in these digital networks—”
“Murdered most of my parents’ families.” You heard that right. Lanier’s mother survived an Austrian concentration camp but many of her family died during the war—and many of his father’s family were slaughtered in prewar Russian pogroms, which led the survivors to flee to the United States.
It explains, I think, why his father, a delightfully eccentric student of human nature, brought up his son in the New Mexico desert—far from civilization and its lynch mob potential. We read of online bullying leading to teen suicides in the United States and, in China, there are reports of well-organized online virtual lynch mobs forming...digital Maoism.
He gives me one detail about what happened to his father’s family in Russia. “One of [my father’s] aunts was unable to speak because she had survived the pogrom by remaining absolutely mute while her sister was killed by sword in front of her [while she hid] under a bed. She was never able to speak again.”
It’s a haunting image of speechlessness. A pogrom is carried out by a “crowd,” the true horrific embodiment of the purported “wisdom of the crowd.” You could say it made Lanier even more determined not to remain mute. To speak out against the digital barbarism he regrets he helped create.
Tim Wu, The New York Times, December 19, 2012
NOBODY ever said that big cities make for easy living. The apps of the moment, Uber and Airbnb, try to mitigate matters by letting you book a car ride or rent someone’s apartment using your smartphone or computer. They are beloved by those contemplating scarce taxis or $500 hotel rooms. But they’re considerably less popular among city regulators, whose reactions recall Ned Ludd’s response to the automated loom.
Last month, Uber was effectively outlawed by Vancouver, British Columbia (by setting a minimum fare so high it discouraged users), and there are proposals to ban it in New York and other cities. Airbnb is already illegal in cities like San Francisco and New York, where unpredictable enforcement can result in enormous fines for its users.
To be fair, the cities think they’re protecting consumers, and the apps, particularly Airbnb, do raise some real concerns. You might be less of an Airbnb fan if your neighbor ran a hostel for the world’s misbehaving youth. As for Uber, its rates aren’t governed by city agencies, and it raised them in the aftermath of Hurricane Sandy in New York City — which critics labeled price gouging.
Yet many of the complaints are anecdotal, and too many have the odor of industry protectionism. Banning Airbnb helps hotels more than homeowners; banning Uber helps taxi companies more than passengers. Boston, in one egregious example, tried to ban Uber simply because it used GPS to measure fares, instead of an old-fashioned meter. (The ban was later reversed.)
Regulators can do their job and protect consumers against harm without being so heavy-handed. The current approach recalls Prohibition: total bans that are widely violated, with semi-random enforcement and huge fines for unlucky individuals (one Airbnb host in New York recently faced more than $40,000 in potential penalties, before the case was dismissed). It’s a clumsy approach that turns ordinary citizens into scofflaws.
This isn’t to say that the apps should have some kind of special legal immunity. It’s just that there are much smarter and more effective ways to protect consumers against potential harms. The trick is using the same techniques of real-time information access that the apps employ.
Here’s how it might work for Airbnb. Cities could require the company to provide co-op boards and landlords, at their request, with an app that listed any advertised rentals at their addresses. That way, instead of one uniform rule for the city, landlords and boards could decide for themselves how they wanted to handle Airbnb rentals. They could take a laissez-faire approach, or ban tenants from advertising on Airbnb entirely — in which case the app would help them detect and fine any violators. Others might choose to include Airbnb guidelines in rental contracts, take a part of the proceeds or limit usage to certain levels. Over all, getting better information about how Airbnb is actually being used would yield better solutions for all.
Another info-access app could help cities regulate Uber’s rates, if they really were afraid of price gouging. Regulators could simply require Uber to disclose the prices it charged and where its cars were going. If cities wanted to ban rate hikes during emergencies, they could watch to see that the law was obeyed.
This kind of precise, data-driven regulation could protect consumers while also protecting their right to pay for a valuable service. No one can deny that these apps are responding to real demands and helping cities become easier to live in and visit. There are also basic rights at stake. Using Uber to hire a car is basically an exercise in freedom to contract. Taking away the right to rent out your room diminishes the value of your property and drives up costs for visitors.
Change isn’t always pretty, but a healthy city is one where old systems — even the hallowed taxi medallion — stand to be challenged by the winds of creative destruction. Uber and Airbnb are just the first examples of a wave of services trying to match willing buyers and sellers in unexpected ways. That’s why it is so important that regulators get this right, lest they discourage those who are trying to follow their lead. The challenge for regulators is to simultaneously allow change while protecting us from the worst effects of it. It is, in short, a time to think carefully, rather than banning first and asking questions later.
Tim Wu, a professor of law at Columbia, is the author of “The Master Switch: The Rise and Fall of Information Empires.”
Gil Press, Forbes, December 12, 2012
While many saw big data as the winner of the recent elections, I voted for Big Intuition, citing Bill Clinton’s insight and advice as an example of how decisions and data science—in political campaigns or any other endeavor—cannot be automated and must rely on human judgment and domain expertise.
This week, Matthew Jones, a historian at Columbia who is working on the history of data mining, came to a similar conclusion after auditing Rachel Schutt’s introduction to data science class: “Data science depends utterly on algorithms but does not reduce to those algorithms. The use of those algorithms rests fundamentally on what sociologists of science call ‘tacit knowledge’—practical knowledge not easily reducable to articulated rules—or perhaps impossible to reduce to rules.”
This irreducible knowledge is of two kinds: Expertise and experience in a specific domain (as in “Clinton knows how to run political campaigns”); and—specifically for data scientists—experience with and understanding of the tools they apply. Says Jones: “The hubris one might have when using an algorithm must be tempered through a profound familiarity with that algorithm and its particular instantiation.”
Another potentially dangerous hubris is assuming that data trumps algorithms and that “one has an enormous advantage if they have all the existing data.” This is what Josh Wills, a guest lecturer at the Columbia data science class this week, believes according to Cathy O’Neil. Wills even came up with a “big data economic law,” which states that “no individual record is particularly valuable, but having every record is incredibly valuable.”
This “economic law” was debated in many quarters this week. Jeff Hawkins told NYT’s Quentin Hardy that “It only makes sense to look at old data if you think the world doesn’t change. You don’t remember the specific muscles you just used to pick up a coffee cup, or all the words you heard this morning.” Instead, he advocates using and analyzing only (or mostly) streaming data, similar to how the brain functions. Hardy‘s summation: “If no data needs to be saved over a long term and real-time data can stream in all the information that is needed, a big part of the tech industry has a problem.”
Splunk’s Stephen Sorkin also doesn’t buy Wills’ economic law. While pointing out companies don’t analyze data that they collect such as log files, he argues that one false promise that some proponents of big data hold out is that somehow vast oceans of digital data can be sifted for nuggets of pure enterprise gold. The Wall Street Journal reports on his comments at the Silicon Valley Comes to Oxford annual event: “It is not going to happen magically. The software only finds correlations, not causations…If you take any sufficiently large data sets, you are going to find correlations. You need a human in the loop to work out which are important.”
Similarly, Michael Wu, Principal Scientist of Analytics at Lithium contends that “the fallacy of big data is that more data doesn’t mean you will get ‘proportionately’ more information. In fact, the more data you have, the less information you gain as a proportion of the data. That means the information you can extract from any big data asymptotically diminishes as your data volume increases.” Sounds to me like an “economic law,” possibly to be labeled “the data implosion law,” to counter Wills’ “bigger is better law.” And you need humans to turn data implosion into value explosion. Rob Lux, CIO of Freddie Mac, tells the Wall Street Journal (subscription required): “We’ve got lots of dirt—data. And there’s definitely gold in our data. But to refine it into information, you need people that have been doing this for years.”
Given the scarcity of people with the right algorithmic and/or domain experience, modern-day gold miners come with a hefty price tag. In the same Wall Street Journal article, Tom Davenport puts the annual salaries of some data scientist at $300,000, which is “pretty good for somebody that doesn’t have anyone else working for them.”
Maybe that’s another economic law—“the value of data scientists rises with the rise in the amount of data to be mined and decreases with the rise in the availability of data miners.” IDC this week confirmed our suspicions that the first part—data growth—is not abating, forecasting 4 trillion gigabytes of new data in 2013, almost 50% more than 2012 volumes and nearly a quadrupling of 2010 volumes.
The second part, the scarcity of data scientists, was taken up this week by IBM which announced a new center for data analytics and a partnership with Ohio State University to develop new data science course materials. Other venues for training of data scientists are the planned Master’s program in marketing analytics from the University of Maryland and existing and planned programs described in this USNews article.
Given that data scientists are hard to find, there’s a lot going on in terms of successful data science projects. News of big data in practice reported this week included:
· Amadeus is beta testing new flight search technology after spending the past five years building a custom big data platform to power it. The new ‘Featured Results’ product will present travel agents and their customers with four targeted flight search results while waiting for a more comprehensive list of options to load. Amadeus provides each of its travel agency customers with an in-memory cache of “several hundreds of millions” of live airline prices and half-a-billion reservation records that could be up to a year old.
· Stem, a startup combining real-time energy monitoring software and onsite battery storage is delivering new levels of cost savings to two San Francisco hotels.
· Cars.com manages its website more efficiently with Splunk’s help, analyzing 2.5 million weblogs per hour, representing a terabyte per week.
· Norfolk Southern combines data from remote sensors, weather forecasts, and other sources to help dispatchers and train engineers make better decisions that help the railroad reduce fuel consumption by around 6%, cutting $1.6 billion in costs.
· Health care decisions: “Up until the time we had big data analytics, physicians were using guidelines based on broad population averages that were not tailored to any one person,” according to Katrina Montinola, VP of Engineering at Archimedes, a healthcare modeling organization. “With Univa Grid Engine, Archimedes’ technology is able to make connections between one event and another event in a way that one human brain wasn’t able to do. It is able to compare a person’s electronic health record with risk factors and all sorts of data. The computer is able to make connections on events and risk factors that would be overlooked by the average doctor—or even the best doctor.”
· Targeted marketing: “Now we have almost five years of data on most of our members and customers, so we’re able to do more nuanced marketing and communication,” says Alexis Maybank, co-founder of the Gilt Groupe. “Within a single minute at noon everyday, there are over 3000 versions of our message that go out to customers, based on what they shop for, what they like, even what sizes they wear. It’s tailored, one-to-one communication with the customer…. On the engineering front, it means that you’re hiring statisticians and data engineers who are really good at building algorithms to take advantage of this information.”
Two big data announcements came from two large companies, one founded in 1994, the other in 1890. Amazon announced Redshift, a cloud-based data warehouse service that it says will deliver better scalability and performance than conventional on-premises data warehouses at dramatically lower costs. Amazon also revealed that its S3 storage service now stores a total of 1.3 trillion objects and handles over 830,000 requests per second. GE announced that it plans to launch a series of “Industrial Internet Quests” with Kaggle’s help and two are already under way—reducing flight delays and improving hospital experience for doctors and patients. GE also issued a report on the “Industrial Internet” (“Intelligent devices, intelligent systems, and intelligent decisioning”) and said it will invest $1.5 billion in the idea over a three-year period.
In funding news, Sumo Logic has raised $30 million Series C round of venture capital funding led by Accel Partners. It uses cloud-based software to analyze, monitor and visualize data generated by machines in real time. AgilOne has raised $10 million from Mayfield Fund with participation from Sequoia Capital. It provides cloud-based predictive marketing intelligence to help marketers figure out what their customers are going to do next.
Big data byte of the week: On November 23rd, Psy, a South Korean pop star, claimed the title for the most-watched online video when his “Gangnam Style” clip was viewed more than 805m times. The Economist reports (quoting Sandvine) that the average monthly usage of online subscribers (fixed access) in North America has gone up from 15 gigabyte to 51.3 gigabytes in less than two years and that real-time entertainment now accounts for almost two-thirds of peak-time Internet traffic.
You can contribute your own bytes on December 12 by watching Live from New York, It’s Big Data.
Is the push to free up government data resulting in economic activity and startup creation?
Alex Howard, O'Reilly Radar, December 11, 2012
Over the past several years, I’ve been writing about how government data is moving into the marketplaces, underpinning ideas, products and services. Open government data and application programming interfaces to distribute it, more commonly known as APIs, increasingly look like fundamental public infrastructure for digital government in the 21st century.
What I’m looking for now is more examples of startups and businesses that have been created using open data or that would not be able to continue operations without it. If big data is a strategic resource, it’s important to understand how and where organizations are using it for public good, civic utility and economic benefit.
Sometimes government data has been proactively released, like the federal government’s work to revolutionize the health care industry by making health data as useful as weather data or New York City’s approach to becoming a data platform.
In other cases, startups like Panjiva or BrightScope have liberated government data through Freedom of Information Act requests and automated means. By doing so, they’ve helped the American people and global customers understand the supply chain, the fees associated with 401(k) plans and the history of financial advisors.
I’ve hypothesized that open data will have an overall effect on the economy akin to that of open source and small business. Gartner’s research has posited that open data creates value in the public and private sector. If government acts as a platform to enable people inside and outside government to innovate on top of it, what are the outcomes?
Over the past four years, the world has heard a rising chorus for raw data from voices like the creator of the World Wide Web, Tim Berners-Lee, and the chief technology officer of the United States, Todd Park. Park, in particular, has been working to scale open data across the federal government as the nation’s “entrepreneur in residence.”
McKinsey and Associates estimated the annual economic value of big, open liquid health data at some $350 billion annually. While that number is eye-opening, which companies and startups stand to change health care using open health data?
Some examples are clear, from mobile apps like iTriage (now owned by Aetna) to Castlight, but they aren’t sufficient to understand what’s happening out there.
Other promising startups are in the consumer finance space, where so-called “smart disclosure” initiatives are enabling people to put their personal data to use. Startups like Billshrink.com and Hello Wallet now are enabling people to make smarter financial decisions.
I know there are more stories out there, and in sectors beyond health care and consumer finance — including transit, energy, education and media. Over the next several months, I’ll be identifying and profiling more civic startups, such as those from the first class in the Code for America accelerator, like Captricity, to specialized search engines, like Zillow, Panjiva and DataMarket.
In the course of that work, I hope to answer some big questions. What are the sustainable business models that successful civic startups are using, whether they use legislative data or other reuse of public sector information? What are the real costs associated with opening up government data to make it usable, both for government and entrepreneurs? And how does it balance against what datasets, at the federal, state or local levels, are the most valuable? Are they open and usable? If so, who’s using them and to what effect? If not, why not?
At the end of this particular project, in February, we’ll publish a report on what I’ve found. In it, I hope to be able to share some answers to several core questions on the topic. Where I need your help is in identifying new startups that are using or consuming government data or in highlighting how existing companies use it in their operations, good or services. Who is doing the most interesting work — and where? If you have research and evidence to share on the questions I posed above, feel free to ring in on that count as well.
Please weigh in through the comments or drop me a line at alex@oreilly.com or at @digiphile on Twitter.
Jeff Immelt, GigaOm, November 28, 2012
Thanks to the internet, almost anything consumers might want is just a click away. But for businesses, the gains have been much less dramatic. That is about to change, with the arrival of the Industrial Internet.
As we know it today, the internet has been largely about connecting people to information, people to people, and people to business. Monetization strategies range as widely as the options available, and for all the success, there are more failures. While many of the advancements have been extraordinary – even unthinkable a short time ago – too often we’re still left asking, “to what end?”
The internet can give consumers nearly anything with just a click, but global economies remain challenged. The internet has become the biggest library in the world, but education is just now beginning to take advantage and change. The internet can provide businesses with unprecedented data, but true insight remains contentious and change is slow.
The real opportunity for change is still ahead of us, surpassing the magnitude of the development and adoption of the consumer internet. It is what we call the “Industrial Internet,” an open, global network that connects people, data and machines. The Industrial Internet is aimed at advancing the critical industries that power, move and treat the world.
There are now many millions of machines across the world, ranging from simple electric motors to highly advanced MRI machines. There are tens of thousands of fleets of sophisticated machinery, ranging from power plants that produce electricity to aircraft that move people and cargo around the world. There are thousands of complex networks ranging from power grids to railroad systems, which tie machines and fleets together.
This vast physical world of machines, facilities, fleets and networks can more deeply merge with the connectivity, big data and analytics of the digital world. This is what the Industrial Internet Revolution is all about.
Productivity Revolution
The Industrial Internet leverages the power of the cloud to connect machines embedded with sensors and sophisticated software to other machines (and to us) so we can extract data, make sense of it and find meaning where it did not exist before. Machines – from jet engines to gas turbines to CT scanners – will have the analytical intelligence to self-diagnose and self-correct. They will be able to deliver the right information to the right people, all in real time. When machines can sense conditions and communicate, they become instruments of understanding. They create knowledge from which we can act quickly, saving money and producing better outcomes.
As an example, we have pushed the boundaries of physical and material sciences in our aircraft engines to the point where these engines are more powerful and efficient than ever. We will continue to improve them physically, but at the same time we can use software, monitoring and big data analytics to attack the $284 billion in annual waste in the airline industry that is caused by fuel inefficiency, unscheduled aircraft maintenance, and delayed flights.
Consider that just a one percent improvement in aircraft engine maintenance efficiency can reduce related costs by $250 million annually. A similar one percent fuel savings in power generation could add more than $4 billion annually to the global economy.
Whether in terms of operations, performance or maintenance excellence, all industries are looking for their next major productivity gains. Health care is burdened by a system where doctors and caregivers have to go searching for vital information; it is inefficient at best and life threatening at worst. We need to make the data more intelligent and integrated, more predictive and proactive, so information finds the doctor instead of the other way around.
Intelligent data flows speed up care delivery and can prevent chronic conditions by getting the treatment right the very first time. Similarly, in terms of health management costs, “intelligent” hospitals are deploying systems that behave like air traffic control for medical staff and devices, and provide a full detailed view of hospital resources. Better utilization cuts capital expenses. Better asset location leaves nurses more time to focus on patients. Better management improves patient flow, cuts operating costs, and saves hospitals millions.
There are similar scenarios in every other major industry, and the economic benefit can be huge. Assuming growth similar to what prevailed during the internet boom, the Industrial Internet revolution will add about $15 trillion to global GDP by 2030. That’s the equivalent of adding another U.S. economy to the world.
The amazing aspect of this growth is that it stems from what appears to be minor productivity improvements. At GE, we have 5,000 software engineers and another 9,000 IT engineers. We’re focused on mining for just one percent gains in productivity. The potential is irresistible.
Roadmap to the Revolution
In the near future, I expect nothing short of an open, global fabric of highly intelligent machines that connect, communicate and cooperate with us. This Industrial Internet is not about a world run by robots, it is about combining the world’s best technologies to solve our biggest challenges. It’s about economically and environmentally sustainable energy, curing the incurable diseases, and preparing our infrastructure and cities for the next 100 years.
To do this industry and government need to work together on two critical areas: standardization and security. We need to establish common standards so that innovative minds can develop the best solutions for the machines and systems that move our world. Just as the advancement of mobile devices and operating systems have brought forth a prosperous “app” economy, a standard language for machines will unleash waves of innovation that will truly change how the world works. This is a critical step and needs government policies that favor advancement.
Attaining the vision set forth for the Industrial Internet will also require an effective internet security regime. Cyber security should be considered in terms of both network security (a defense strategy specific to the Cloud) and the security of devices that are connected to the network. We need industry to effectively secure facilities and networks and governments to enforce a regulatory regime that promotes innovative solutions and international standards.
The Industrial Internet era has already begun. And during a time when the global economy is recovering but remains volatile and where resources are constrained for people, governments, and companies, what we need most is to not lose sight of a real opportunity to create meaningful change around the world.
After all, this is what revolutions are all about.
Jeff Immelt is the chairman and CEO of GE.
Analysts Discuss Key Issues Facing the IT Industry During Gartner Symposium/ITxpo 2012, October 21-25, in Orlando
ORLANDO, Fla.--(BUSINESS WIRE)-- Worldwide IT spending is forecast to surpass $3.7 trillion in 2013, a 3.8 percent increase from 2012 projected spending of $3.6 trillion, but it's the outlook for big data that is creating much excitement, according to Gartner, Inc.
"By 2015, 4.4 million IT jobs globally will be created to support big data, generating 1.9 million IT jobs in the United States," said Peter Sondergaard, senior vice president at Gartner and global head of Research. "In addition, every big data-related role in the U.S. will create employment for three people outside of IT, so over the next four years a total of 6 million jobs in the U.S. will be generated by the information economy."
"But there is a challenge. There is not enough talent in the industry. Our public and private education systems are failing us. Therefore, only one-third of the IT jobs will be filled. Data experts will be a scarce, valuable commodity," Mr. Sondergaard said. "IT leaders will need immediate focus on how their organization develops and attracts the skills required. These jobs will be needed to grow your business. These jobs are the future of the new information economy."
Mr. Sondergaard provided the latest outlook for the IT industry today to an audience of more than 8,000 CIOs and IT leaders at Gartner Symposium/ITxpo, which is taking place here through October 25. He said the IT industry is entering the Nexus of Forces, which includes a confluence and integration of cloud, social collaboration, mobile and information.
"This is a time of accelerating change, where your current IT architecture will be rendered obsolete," Mr. Sondergaard said. "You must lead through this change, selectively destroy low impact systems, and aggressively change your IT cost structure. This is the New World of the Nexus, the next age of computing."
Cloud
The cloud is the carrier for the three other Forces: mobile is personal cloud, social media is only possible via the cloud, and big data is the killer app for the cloud. Cloud will be the permanent fixture, the foundation.
"Cloud is not merely about cost-cutting, the end game is not just cheap on-demand services. In fact, 90 percent of these services are still subscription based, not pay-as-you-go," Mr. Sondergaard said. "We are just at the beginning of realizing the cost benefits of cloud, but organizations moving to the cloud are also attracted by the new capabilities they do not get today. It is bringing new approaches to designing applications, specifically for the cloud, and providing more resilience by architecturing failure as a design concept. Cloud also teaches us about services and service levels, and the contrast between what the business wants for outcomes versus IT's old methods of getting there."
Mobile
In 2016, more than 1.6 billion smart mobile devices will be purchased globally. Two-thirds of the mobile workforce will own a smartphone, and 40 percent of the workforce will be mobile. The challenge for IT leaders is determining what to do with this new channel to their customers and employees.
"Mobile is about computing at the right time, in the moment. It is the point of entry for all applications, delivering personalized, contextual experiences," Mr. Sondergaard said. "It means: marketing gets more time with the customer; employees become more productive; and process flows get dramatically cut."
In less than two years, iPads will be more common in business than Blackberries. Mr. Sondergaard said some CIOs are now placing orders for tens of thousands of iPads at a time. Productivity is the driver. Two years from now, 20 percent of sales organizations will use tablets as the primary mobile platform for their field sales force. As a result, by 2018, 70 percent of mobile workers will use a tablet or a hybrid device that has tablet-like characteristics.
Gartner forecasts that in 2016, half of all non-PC devices will be purchased by employees. By the end of the decade, half of all devices in business will be purchased by employees.
Social Computing
In the next three years, the dominant consumer social networks will the limits of their growth. However, social computing will become even more important. Companies are establishing social media as a discipline. Gartner predicts that in three years, 10 organizations will each spend more than $1 billion on social media.
"Social computing is moving from being just on the outside of the organization to being at the core of business operations," Mr. Sondergaard said. "It is changing the fundamentals of management: how you establish a sense of purpose and motivate people to act. Social computing will move organizations from hierarchical structures and defined teams to communities that can cross any organizational boundary."
Big Data
By tapping a continual stream of information from internal and external sources, businesses today have an endless array of new opportunities for: transforming decision-making; discovering new insights; optimizing the business; and innovating their industries.
Big data creates a new layer in the economy which is all about information, turning information, or data, into revenue. This will accelerate growth in the global economy and create jobs.
"Big data is about looking ahead, beyond what everybody else sees," Mr. Sondergaard said. "You need to understand how to deal with hybrid data, meaning the combination of structured and unstructured data, and how you shine a light on 'dark data.' Dark data is the data being collected, but going unused despite its value. Leading organizations of the future will be distinguished by the quality of their predictive algorithms. This is the CIO challenge, and opportunity."
About Gartner Symposium/ITxpo
Gartner Symposium/ITxpo is the world's most important gathering of CIOs and senior IT executives. This event delivers independent and objective content with the authority and weight of the world's leading IT research and advisory organization, and provides access to the latest solutions from key technology providers. Gartner's annual Symposium/ITxpo events are key components of attendees' annual planning efforts. IT executives rely on Gartner Symposium/ITxpo to gain insight into how their organizations can use IT to address business challenges and improve operational efficiency.
Additional information about Gartner Symposium/ITxpo in Orlando, is available at www.gartner.com/symposium/us. Video replays of keynotes and sessions are available on Gartner Events on Demand at www.gartnerondemand.com. Follow news, photos and video coming from Gartner Symposium/ITxpo on Facebook at http://www.facebook.com/GartnerSymposium, and on Twitter at http://twitter.com/Gartner_inc and using #GartnerSym.
