5 real-world uses of big data

5 real-world uses of big data
By David Smith Jul. 17, 2011,
http://gigaom.com/cloud/5-real-world-uses-of-big-data/

In the past year, big data has emerged as one of the most closely watched trends in IT. Organizations today are generating more data in a single day than that the entire Internet was generated as recently as 2000. The explosion of “big data”–much of it in complex and unstructured formats–has presented companies with a tremendous opportunity to leverage their data for better business insights through analytics.

Wal-Mart was one of the early pioneers in this field, using predictive analytics to better identify customer preferences on a regional basis and stock their branch locations accordingly. It was an incredibly effective tactic that yielded strong ROI and allowed them to separate themselves from the retail pack. Other industries took notice of Wal-Mart’s tactics — and the success they gleaned from processing and analyzing their data — and began to employ the same tactics.

While data analytics was once considered a competitive advantage, it’s increasingly being seen as a necessity for enterprises–to the point that those that aren’t employing some kind of analytics are seen to be at a competitive disadvantage. Driven by the rise of modern statistical languages like R, there’s been a surge in enterprises hiring data analysts–which has in turn given rise to the larger data science movement. Data is a huge asset for enterprises, and they’re beginning to treat it accordingly.

For all the talk about the need to effectively analyze your data, though, there’s been relatively little written about how organizations are using data to achieve actionable results. With that in mind, here are five use cases involving analyses of large data sets that brought about valuable new insight:

·       NYU Ph.D. student conducts comprehensive analysis of Wikileaks data for greater insight into the Afghanistan conflict: Drew Conway is a Ph.D. student at New York University who also runs the popular, data-centric Zero Intelligence Agents blog. Last year, he analyzed several terabytes worth of Wikileaks data to determine key trends around U.S. and coalition troop activity in Afghanistan. Conway used the R statistics language first to sort the overall flow of information in the five Afghanistan regions, categorized by type of activity (enemy, neutral, ally), and then to identify key patterns from the data. His findings gave credence to a number of popular theories on troop activity there–that there were seasonal spikes in conflict with the Taliban and most coalition activity stemmed from the “Ring Road” that surrounds the capitol, Kabul, to name a few. Through this work, Conway helped the public glean additional insight into the state of affairs for American troops in Afghanistan and the high degree of combat they experienced there.

·       International non-profit organization uses data science to confirm Guatemalan genocide: Benetech is a non-profit organization that has been contracted by the likes of Amnesty International and Human Rights Watch to address controversial geopolitical issues through data science. Several years ago, they were contracted to analyze a massive trove of secret files from Guatemala’s National Police that were discovered in an abandoned munitions depot. The documents, of which there were over 80 million, detailed state-sanctioned arrests and disappearances that occurred during the country’s decades-long civil conflict that occurred between 1960 and 1996. There had long been whispers of a genocide against the country’s Mayan population during that period, but no hard evidence had previously emerged to verify these claims. Benetech’s scientists set up a random sample of the data to analyze its content for details on missing victims from the decades-long conflict. After exhaustive analysis, Benetech was able to come to the grim conclusion that genocide had in fact occurred in Guatemala. In the process, they were able to give closure to grieving relatives that had wondered about the fate of their loved ones for decades.

·       Statistician develops innovative metrics tracking for baseball players, gains widespread recognition and a job with the Boston Red Sox: Bill James (he of Moneyball fame) is a well-known figure in the world of both baseball and statistics at this point, but that has not always been the case. James, a classically trained statistician and avid baseball fan, began publishing research in the early 1970s that took a more quantitative approach to analyzing the performance of baseball players. His work focused on providing specific metrics that could empirically support or refute specific claims about players, be it the amount of runs they contributed to in a given season or how their defensive abilities contributed to or detracted from a team’s success. James’ approach became known as sabermetrics and has since expanded to incorporate a wide range of quantitative analyses for measuring baseball metrics. Over time, sabermetrics has gained wide recognition in baseball to the point that it’s now employed by all 30 Major League Baseball teams for tracking player metrics. In 2003, James was named Senior Advisor of Baseball Operations by the Boston Red Sox, a position he holds to this day.

·       U.S. government uses R to coordinate disaster response to BP oil spill: In the early days of last year’s Deepwater Horizon disaster, the flow of oil rate from the spill was of primary concern; estimating it accurately was key to coordinating the scale and scope of the U.S. government’s response to the emergency. The National Institute of Science and Technology (NIST) was charged with making sense of the varying estimates that existed from both BP and independent third-parties. To do so, NIST used the open source R language to run an uncertainty analysis that harmonized the estimates from various sources to come up with actionable intelligence around which disaster response efforts could be coordinated.

·       Medical diagnostics company analyzes millions of lines of data to develop first non-intrusive test for predicting coronary artery disease: CardioDX is a relatively small, Palo Alto, Calif.-based company that performs genomic research. One of their major initiatives over the past several years was developing a predictive test that could identify coronary artery disease in its most nascent stages. To do so, researchers at the company analyzed over 100 million gene samples to ultimately identify the 23 primary predictive genes for coronary artery disease. The resulting test, known as the “Corus CAD Test,” was recognized as on of the “Top Ten Medical Breakthroughs of 2010” by TIME Magazine.
These are but a few brief examples of the exciting work that’s being undertaken in the rapidly growing discipline of data science. More and more, data analysis is being relied on to provide context for critical business decisions, a trend that promises to increase as data sets grow larger and more complex and scientists continue to push the limits of statistical innovation.
David Smith is vice president of community at Revolution Analytics, a company founded in 2007 to foster R analytics by creating programs to make it easier for data scientists to analyze large amounts of data.

PM-ISE Releases the 2011 ISE Annual Report to the Congress

The PM-ISE has officially released its 2011 ISE Annual Report to the Congress and we are proud of the information sharing success stories featured in the Report – stories that describe the outstanding accomplishments of our mission partners across the federal, state, local, and tribal governments, the private sector, and foreign allies.

The Annual Report is required by law to provide the Congress “a progress report on the extent to which the ISE has been implemented.”[1] The Report highlights major ISE activities since July 2010 and is organized around five themes:

Strengthening Management and Oversight - The Annual Report describes the work of the Information Sharing and Access Interagency Policy Committee (ISA IPC) and its sub-committees and working groups; of particular note, the Report highlights how these bodies expanded to include representatives of non-federal organizations and are reaching out to engage the private sector in developing the ISE, as well.

Improving Information Sharing Activities - Among the many activities presented, the Report describes how the Nationwide Suspicious Activity Reporting Initiative has made substantial progress toward streamlining reporting and analysis within fusion centers by implementing new standards, policies, and processes. Another notable interagency effort involved the Baseline Capabilities Assessment, during which federal, state, and local officials completed the first nationwide, in-depth assessment of fusion centers to baseline their capabilities.

Establishing Standards for Responsible Information Sharing and Protection - Standards are critical to powering the ISE, and so the Report describes the efforts by the PM-ISE, its mission partners, and standards organizations to identify the best existing standards for reuse and implementation across the ISE.

Enabling Assured Interoperability across Networks - The Report details the tremendous progress made toward implementing a Simplified Sign On that will enable federal, state, local, and tribal law enforcement officers and analysts to more easily access a rich variety of data services provided by Assured Sensitive but Unclassified (SBU) networks. The Report also describes similar efforts for classified information sharing.

Enhancing Privacy, Civil Rights, and Civil Liberties Protections - Balancing the need for national security with the need to protect privacy and civil liberties, the Report provides information on policies and training activities designed to enhance these protections.

These are only a few of the activities that are helping the nation build a robust information sharing environment. And, while the Annual Report is primarily focused on terrorism-related initiatives, it also describes mission partner accomplishments that may not have been developed explicitly to support CT, but which may ultimately become best practices for information sharing and collaboration government-wide.

FT: A brave new networked world

July 18, 2011 11:15 pm
A brave new networked world
By Philip Delves Broughton
http://www.ft.com/intl/cms/s/0/2fe38490-b176-11e0-9444-00144feab49a.html#axzz1SSrsjjWo
We are under siege by social networks, from Facebook to LinkedIn to school, university and even corporate alumni organisations, which are ever more aggressive and sophisticated in their networking efforts.
Facebook has more than 750m users, LinkedIn 100m, and Twitter is handling 1bn tweets a week. Technology-driven social networks have been credited with propelling the revolutionaries of the Arab spring.
Marketers and venture investors salivate over any company promising to identify and assemble networks of like-minded consumers. Social network analysis software has become a fast-growing sector of IT services. IBM alone has spent $11bn in the past five years buying makers of such software.

But there are sceptics questioning the power of these networks. Malcolm Gladwell wrote in The New Yorker last year that the impact of new forms of communication in fomenting political change was exaggerated. He distinguished between the strong ties that bind groups of revolutionaries and the weak ties that link 1m Facebook fans.

“The platforms of social media are built around weak ties,” he wrote. “Twitter is a way of following (or being followed by) people you may never have met. Facebook is a tool for efficiently managing your acquaintances, for keeping up with the people you would not otherwise be able to stay in touch with. That’s why you can have a thousand ‘friends’ on Facebook, as you never could in real life.”

While the significance of social networks for political activists may be open to question, companies are starting to find real value in mapping and analysing the strength and frequency of connections between employees and customers, and their behaviour.

Social network analysis is being used to measure job performance and forecast turnover, to rate employees for promotion, monitor their ethical standards and improve the systems for collaboration. It is now a standard diagnostic and prescriptive tool for management consultants advising companies.

By measuring who talks to whom, when and for how long, companies can uncover hidden stars. They can develop an index that reveals where real power lies – and it may not be with the people who hold the grandest titles. When the US was tracking Saddam Hussein it mapped the networks of his former chauffeurs, which led to his hideout. The Iraqi elite had no idea where he was.

Businesses also use social network analysis to help categorise customers. Nike tracks bloggers to find out whose posts most influence sales of their shoes. Telecoms companies seek out the “influencers”, those thrifty types who shop around for the best plan and take their Facebook friends and Twitter followers with them when they switch.

Rob Cross is a professor of management at the University of Virginia’s McIntire School of Commerce and a consultant on corporate social networks to companies ranging from American Express to Intel. Through web-based surveys, he maps “who creates enthusiasm within organisations and who drains it. This turns out to be wildly predictive of success or failure in innovation”.

Prof Cross also finds that social networks can create another level of work for employees. “Due to the recession, people’s workloads and spans of accountability have increased, the tools of collaboration have multiplied, and we’re seeing people can’t keep up with collaborative demands any more,” he says.

Social media tools have only made this worse. Much of his work is therefore about reducing unnecessary collaboration. Organisations can benefit from protecting key people, such as research scientists or top salespeople, from the deluge of fruitless networking and information sharing.

Social network sites, Prof Cross says, are good for creating interest groups or sharing tactical information, but in situations where you need to solve a problem it is necessary to resort to old-fashioned means of developing trust and sharing information.

At agribusiness Monsanto, executives spread across the world who were forced to implement a new global transaction system were much more productive, more quickly, when they had previous experience of collaborating.

Another useful aspect of social network sites is in reactivating dormant ties. In research published in MIT Sloan Management Review, Daniel Levin, Jorge Walter and Keith Murnighan have shown that we underestimate the power of our dormant relationships.

People we once knew well but have not seen for a while turn out to be delighted to hear from us. They also have novel insights and are happy to help. Social network sites make these dormant relationships easier to rediscover and resume, but they are only a start.

“People who haven’t seen each other in a while are delighted to e-mail, until something goes beyond expectation in a negative way,” says Prof Murnighan. “Then they might realise why the relationship was dormant.”

For substantive interactions, you still need to get on the phone or meet in person. But the lesson of his research into dormant ties, he says, is that “people move on with their lives, but they don’t forget each other. An awful lot of substance remains”.

No amount of e-mailing or Facebook poking can substitute for extended time spent with others.
Lauren Cohen and Christopher Malloy of Harvard Business School have written a series of papers on the continuing importance of old-fashioned ties in investment performance. They found mutual fund managers and sell-side analysts made much better investment returns and recommendations in companies where they had strong university alumni connections.

In the US, they found that before new regulations came into effect in 2000 to limit selective disclosure of corporate information, the return premium from the old school tie was 8.16 per cent a year. Since 2000, it has fallen to zero. But in the UK, where the rules about selective disclosure are less rigorous, the old school tie premium persists.

Prof Cohen says that the premium is explained by the many clubs and networking events laid on by universities and the informal networks and gossip shared among old friends.

People who went to the same university, she says, are “more likely to have met each other or have common acquaintances. They understand what it means if a person belonged to a certain club or participated in a specific study programme. They may know people who hired them previously. All this helps them better assess executives’ potential as leaders and business owners”.

Technology may have made everyone accessible, but it has not yet made all social networks equal.
Harnessing networks
Not all networks are equal but all have their uses. So it is important to identify the differing strength of connections within a network in order to determine how best to use it.

Weak ties Good for forming groups around shared interests, hobbies and projects as well as tactical information sharing. Twitter hashtags and Facebook groups can cluster people with similar interests, but not for long unless strong ties can be developed.

Strong ties Vital for trust-based activities and complex collaboration among remote groups. Monsanto found that executives who had worked together in the past, even though now dispersed around the world, were much more effective at complex, collaborative tasks.

Dormant ties People whom we once knew well are happy to be “reactivated”. They can be great sources of new information and perspectives.

Old school ties Alumni networks still provide access to information which can affect hiring and improve investor returns.