Paul Taylor, The Financial Times, June 19, 2012
Companies are awash with data, some generated by their customers or systems, some by third parties. These data are growing so fast – by about 2.5 exabytes a day – that 90 per cent of the stored data in the world today has been created in just the past two years, earning it the geeky moniker “big data”.
For the uninitiated, one exabyte is 1bn gigabytes.
Some of this material is traditional structured data, such as store point-of-sale information, bank cash machine transactions or mobile phone records. But much of it is unstructured information gleaned from non-traditional sources such as blogs, Facebook posts, tweets, email messages, smartphone apps, electronic sensors, pictures and YouTube video clips.
This high velocity, high volume and high variety make big data difficult to interpret using traditional database and data analytics, says Cyrus Mewawalla, an independent investment researcher.
But by combing through it with more sophisticated data analytics tools and techniques such as in-memory computing, it can provide companies with a better understanding of their customers and partners. It can help them spot trends in near real time, make more accurate forecasts and adjust their operations quickly to changing demand or new business opportunities.
“Companies benefit from a multidimensional view of their business when they add insight from big data to the traditional types of information they collect and analyse,” says Mr Mewawalla.
For example, he says, a company that operates a retail website can use big data to understand site visitors’ activities, such as paths through the site, pages viewed and comments posted. This knowledge can be combined with purchasing history. From this, the company gains a better understanding of customers and can fine-tune offers to target their interests.
Since companies such as Google, Amazon and Facebook pioneered the collection, processing and analysis of big data, it has become one of the hottest trends in corporate information technology alongside cloud computing, mobility and enterprise social networking.
WinterCorp, a specialist big data consultancy, says: “As enterprises harness big data, they are discovering opportunities better to understand and predict the interests and behaviour of their customers, especially in connection with ecommerce and social networking,”.
Meanwhile high-performance analytics is helping industries from banking to retail, healthcare and insurance to glean insights from big data that once took days or weeks in just hours, minutes or seconds.
In engineering and manufacturing, for example, companies are finding new opportunities to predict maintenance problems, enhance manufacturing quality and manage costs using big data. In healthcare, there are new opportunities to predict and react more rapidly to critical clinical events, resulting in better care for patients and more effective cost management.
Many retailers are already using big data analytics to improve the accuracy of forecasts, anticipate changes in demand and then react accordingly. For example, Brooks Brothers, one of the oldest retailers in the US, introduced business analytics developed by SAS, the US-based business analytics software group, to help improve stock management.
Using analytics to forecast accurately its global stock, store managers were able to make better decisions about stock levels and pricing. As a result, the number of times stores were out of stock when customers came in to buy an item were reduced, and stock decreased by 27 per cent.
In the financial services sector, most investment banks still rely on overnight batch data to make trading decisions. This means their risk management models constantly rely on out-of-date data. By using big data analytics in real time, banks can make better trading and risk decisions, safeguarding them against the threat of collapse and, subsequently, protecting the financial markets.
Using technology from SAS, air traffic controllers at Frankfurt airport in Germany receive early warnings of storms, and managers can access an overview of all key performance indicators in near real time, including average times for luggage delivery, delays and airport security levels.
All of this happens on the go – business data are refreshed every five minutes and both managers and operations experts monitor reports via their PCs and mobile devices.
Even governments, including those of the UK and US, are jumping on the big data bandwagon. A recent study undertaken by SAS and the Centre for Economics and Business Research, the UK-based think-tank, suggested that if the UK government capitalised on big data it could save £2bn in fraud detection, create 2,000 new jobs and generate £3.6bn in savings through better management of processes by, for example, integrating patient data to improve healthcare IT systems.
In March, the US government and six federal agencies launched their own big data initiative backed by a $200m investment. Calling it one of the most important public investments in technology since the rise of supercomputing and the internet, the White House Office of Science and Technology Policy (OSTP) said the investment was aimed at “greatly improving the tools and techniques needed to access, organise and glean discoveries from huge volumes of digital data”.
“The way we look at big data is that it is a confluence of three technology trends: big transaction data, big interaction data and big data processing,” says Sohaib Abbasi, chief executive of Informatica, whose software products help companies to clean, integrate and sift through huge volumes of data.
“Big data is not just about the volume of data,” he says, “it is also about new types and sources of data that can be used to gain new insight and deliver business advantage.”
For years, he explains, corporate IT departments have managed transactional data held within relational databases. “The promise of big data is to do better analysis of transactional data – the more data, the more reliable the data and the higher the quality of the analysis,” he says. But while this has been growing in scale and complexity, there is now an additional source of data that enterprises need to take notice: big interaction data.
This is a new type of data that represents social media interactions (human-generated interactions) and machine interactions (device-generated interactions). In both cases, the data are extremely large and continuously growing, and exist both within and beyond the corporate security firewall. “The challenge for businesses is how to understand and extract useful intelligence from these complex, unstructured data sources.”
T-Mobile USA, Deutsche Telekom’s US-based mobile unit, has used Informatica’s PowerCenter to integrate big data across its disparate federated architecture and predict customer defections based on the analysis of its 33m customer data records, web logs, billing data and social media information. By combining big transaction and big interaction data, T-Mobile has gained a better view of the reasons behind customer defections, which it was able to cut in half in a single quarter.
Similarly, US Xpress, the US trucking company, collects 900 data elements from tens of thousands of trucking systems: sensor data for tyre and fuel usage and engine operation, geospatial data for fleet tracking, and complaints posted on trucker blogs. Using Hadoop, a type of open-source database that is often used for big data projects, and Informatica, US Xpress processes and analyses this data to optimise fleet usage, saving millions of dollars a year.
It is not just big companies that are using big data. As McLaren’s Formula One cars speed round the track they send a stream of data back to the team that are processed and analysed in real time using SAP’s Hana in-memory technology. Hana uses sophisticated data compression to store information in Ram, which is 10,000 times faster than hard disks, enabling analysis of the data in seconds rather than hours.
This real-time analysis of car sensor data is compared with historical data and predictive models, helping the McLaren team to make immediate proactive corrections, avoid costly, dangerous incidents and win races.
“On every lap of every Grand Prix, practice or test session, our cars generate vast quantities of performance data. Our ability to process that data and act on it rapidly is crucial to creating the kind of prescriptive intelligence that enables us to transform the outcome of races. And that need resonates through every other facet of our business,” says Ron Dennis, executive chairman of McLaren.
As McLaren diversifies its business, its electronic systems, including telemetry, modelling and real-time simulations, are being used widely in other areas, such as to help Olympic athletes hone their performance and in rapid transit systems in the US to optimise traffic flow.
By harnessing big data, organisations can improve operational efficiency, reduce data management costs and better manage brands and customer relationships. But some IT leaders and analysts warn that few organisations are equipped to capitalise on the business value of big data, and that by neglecting to adapt effectively to big data, organisations also invite unforeseen cost, complexity and risk.
“Just collecting and storing big data doesn’t drive a cent of value to an organisation’s bottom line,” says Stephen Brobst, chief technology officer at Teradata, the analytics company.
So while many businesses are already capturing big data from sources such as web logs, machine data and text, and inexpensively storing it in open source “Hadoop” systems, complexity and incompatibility issues often make it difficult for them to use standard business intelligence applications and tools to access and analyse the many types of data.
“Business users are clamouring for more access to big data analytics,” says Scott Gnau, president of Teradata Labs, which recently rolled out new software called Aster SQL-H, which is designed to make it easier for business analysts to use raw, multi-structured data in Hadoop files to develop new insights for competitive advantage.
Jim Hagemann Snabe, SAP’s co-chief executive, believes that transforming information into intelligence in real time is increasingly critical for the future of every company. Similarly Informatica’s Mr Abbasi suggests that the wealth of new data now available to companies can bring unprecedented business opportunity.
Whether big data becomes an organisation’s greatest asset or one of its gravest liabilities depends on the strategies and solutions it puts in place to deal with the epic growth in data volumes, complexity, diversity and velocity.
This message seems to be getting through. In a global survey of 600 executives this month by Capgemini and the Economist Intelligence Unit, nine out of 10 respondents identified data as being the fourth factor of production – as fundamental to business as land, labour and capital.
Among the survey’s other findings, respondents said the use of big data has improved businesses’ performance, on average, by 26 per cent and that the impact will grow to 41 per cent over the next three years.
Almost 60 per cent of companies said they planned to make a bigger investment in big data over the next three years, suggesting that the era of big data and big data analytics has already arrived.
Copyright The Financial Times Limited 2012. You may share using our article tools.
Please don't cut articles from FT.com and redistribute by email or post to the web.
