Saturday, March 31, 2012
Engineering Our Health
Vivek Wadhwa, American Society for Engineering Education - Prism, March 2012
New technology and DNA access promise a medical revolution.
In 2000, scientists at a private company called Celera announced they had raced ahead of the U.S. government in decoding the DNA of a human being. Using the latest sequencing technology, plus the data available from the Human Genome project, Celera scientists created a working draft of the genome. These efforts cost more than $1 billion. Today, a complete genome sequence costs about $3,000 and takes about a week. One company, Life Technologies Corp. in Carlsbad, Calif., just announced that it will provide the service for $1,000 and in 24 hours. At this rate, within three years, the cost will be less than that of a simple blood test and the results will be almost instantaneous.
This type of data opens up an amazing set of possibilities.
A genome map is the source code for the software that constitutes living organisms. Imagine doing a Google search on your own genome to learn the health predispositions and likely abilities of people genetically similar to you. You can learn about what medications or lifestyle changes may best prevent a disease.
Today, medicines are generally prescribed on a “one size fits all” basis. When a particular medication causes a significant negative reaction with a small part of the population, it is prevented from being available to anyone. With genetic information, we could be prescribing specific types and dosages of medicines based on a person’s DNA.
And new advances in DNA printing promise to revolutionize biology and more.
Craig Venter, who led the research at Celera, made another stunning announcement a decade later, in May 2010. His team had, for the first time in history, built a synthetic life-form — by “writing” DNA. The slow-growing, harmless bacterium they created was made of a synthetic genome with 1,077,947 DNA base pairs. Today, a number of DNA “print” providers offer DNA synthesis and assembly operations as a service. Current pricing is by the number of base pairs — the chemical “bits” that make up a gene — to be assembled. Today’s rate is about 30 cents per base pair, but prices are falling exponentially. Within a few years, it could cost 1/100th this amount. Eventually, like laser printers, DNA printers could become inexpensive home devices that enable legions of garage biotechnologists and DIY-ers to solve big health problems.
Venter is now using “synthetic biology” technology to try to solve the problems of energy by developing biofuels from genetically engineered algae. The idea is to extract “hydrocarbonlike” liquid that can be turned into transportation fuel.
Tissue engineering and 3-D printing technologies are advancing rapidly and beginning to merge. These advances will allow us to print organs and personalized medicines. A company called Tengion synthesized full-size replacement bladders in 2008, and surgeons in Sweden recently carried out the world’s first synthetic organ transplant — a synthetic trachea/windpipe structure created and seeded with the patient’s own progenitor cells.
Soon, we will be able to “print” sophisticated medical devices. And sensor technologies are becoming ubiquitous, so continually monitoring and recording every aspect of our health from the time we wake to when we sleep will be within reach before long.
All of these advances play to the strengths of engineers. The skills needed for diagnosing health problems are similar to those for analyzing the structure of bridges. Search engines for the genome aren’t that different from search engines for text and video. The same rigor needed for programming new bacteria is similar to what’s necessary for building an industrial robot: developing functional requirements, designing and testing components, integrating these components, and testing for effectiveness, reliability, and safety.
With these new developments, it is clear that our engineers have begun using their skills to solve the problems of human health. Now, it’s time they tackled many other grand challenges that face humanity.
Vivek Wadhwa is a scholar specializing in entrepreneurship. He is vice president of academics and innovation at Singularity University and is also affiliated with Stanford University, Duke University’s Pratt School of Engineering, and Emory University.
Thursday, March 29, 2012
McAfee in FT: Companies must use digital technologies to transform, not substitute
Didier Bonnet, Andrew McAfee and George Westerman, The Financial Times, March 29, 2012
Digital transformation, the use of technology to radically improve the performance or reach of enterprises, has become one of the major issues taxing the minds and abilities of executives worldwide.
Organisations in all industries are using digital advances in analytics, mobility, social media and smart embedded devices (such as radio frequency ID tags - RFID) to change their customer relationships, internal processes, and value propositions.
Looking at just how fast digital technology has disrupted other industries, particularly their shell-shocked colleagues in the media industry, the corporate community has been given a clear wake-up call. To understand more, we interviewed some 160 senior executives confronting this challenge in 50 large billion-dollar organisations in 13 countries.
We quickly learned that organisations are facing common pressures and are therefore investing heavily in a common set of digital responses. However, even companies in the same industry investing in the same areas are getting wildly different results.
Digital efforts in many firms lack scale, are disjointed and are often the pet project of one silo within the organisation. However, there are a select few who are taking a different tack and see this digital challenge as an opportunity to truly improve their holistic performance across the entire company.
These organisations are approaching this digital transformation from the top-down and are aligning their efforts under a common vision and coordination structure. However, of the organisations we studied, less than a third is taking this strategic approach and truly reshaping their businesses through digital technologies.
While our research helped us understand how organisations are approaching digital transformation, we also looked at what areas they are transforming. We found that the use of analytics, mobile and social media are widespread in these large organisations to improve both customer experience and operational processes.
Analytics is used by 72 per cent of respondents, mobile and social media by 62 per cent of organisations, and embedded devices by 24 per cent, despite their relative nascence within most industries (see figure 1).
Applications of social media to improve customer experience are more common than for internal uses, reflecting awareness of the fact that consumers are rapidly adopting these technologies, as well as some scepticism about how effective they will be for internal collaboration.
The use of analytics too, is more common in customer experience applications, reflecting theincreasing efforts of organisationsto personally target marketing activities and personalise customer experiences better than in the past.
Mobile is also used widely in both domains but embedded devices are still an emerging technology, with less than a quarter of companies using these devices, although many executives can foresee ways to use the devices as they become cheaper and more reliable.
More important is the difference in how companies are using the technologies. While many companies are using these technologies achieve only incremental improvements, others are using these technologies as an opportunity to re-envision and radically improve their businesses.
Our study identified three levels of digital usage currently in place within major enterprises today:
• Substitution is using new technology as an alternative or replacement for substantially the same function that the enterprise already performs with a different technology and process. For example, BlackBerrys and iPhones acting as a substitute for PCs in both internal operations (email) and customer experience (web access).
Many employees use both methods depending on their needs at a point in time. Analytics, in the form of basic business intelligence tools and detailed reporting, substitutes for similar information sources available through other methods. Substitution is useful when it improves performance over, or is more convenient than, traditional methods.
• Extension is significantly improving performance or functionality of a process through technology. For example, a manufacturer providing full information to field workers on mobile devices, thus eliminating the need for workers to spend time in the office at the start and end of shifts. Or apharma company’s social media physician community enabling the company to learn about issues and opportunities by allowing doctors to talk with one another.
• Transformation is fundamentally redefining a process or function through technology. For example, a manufacturer using a combination of embedded devices and analytics to create fully-automated factories that deliver a higher level of efficiency, quality and environmental sustainability than manually-tended factories. Or a hospitality firm using predictive analytics to engage in location-based marketing with its customers.
The study shows that the most common usage of each technology is for substitution, for example mobile email replacing desktop email or Facebook pages and ads as a substitute for traditional Web advertising (see Figure 2). However, companies are making inroads in each of the three approaches.
Roughly 30 per cent of firms are engaging in extension or transformation with mobile, social media or analytics technologies. Although usage of embedded devices lags behind other technologies, some companies are already extending or transforming their processes through these devices. The researchalso shows an interesting trend toward identifying transformative opportunities that combine multiple technologies with new management practices. For example:
• Factories including both mobile and analytics technologies to improve production significantly.
• Mobile phones are increasingly being used as embedded devices for customer-facing applications.
• Retailers are aiming to integrate their social and web-based marketing approaches with digital product design capabilities and embedded devices in products to make themselvesmore responsive to emerging consumer preferences.
• Concept stores are combining multiple technologies to test or deliver a new buying experience.
Major technologies that developed in isolation are now maturing at the same time. This presents a major strategic opportunity for large firms in all industries. Forward-looking executives are beginning to envision possibilities that cross organisational and technological silos, letting the possibilities drive technology and organisation rather than the reverse.
Click here to access the full report.
Didier Bonnet is a Senior Vice-President and head of global practices for Capgemini Consulting. He is responsible for the digital transformation global research programme.
Andrew McAfee is principal research scientist at the MIT Centre for Digital Business, MIT Sloan School of Management. He is a co-author, along with Erik Brynjolfsson, of the 2011 book “Race Against The Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy.” He also wrote the 2009 book “Enterprise 2.0” (Harvard Business School Press).
George Westerman is a research scientist at the MIT Centre for Digital Business, MIT Sloan School of Management. George is co-author of two books on building competitive advantage from digital technology: “The Real Business of IT” and “IT Risk.”
White House: Big Data is a Big Deal
Tom Kalil, The White House, March 29, 2012
Editor's Note: Watch the live webcast today at 2pm ET of the Big Data Research and Development event at http://live.science360.gov/bigdata/]
Today, the Obama Administration is announcing the “Big Data Research and Development Initiative.” By improving our ability to extract knowledge and insights from large and complex collections of digital data, the initiative promises to help accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning.
To launch the initiative, six Federal departments and agencies will announce more than $200 million in new commitments that, together, promise to greatly improve the tools and techniques needed to access, organize, and glean discoveries from huge volumes of digital data. Learn more about ongoing Federal government programs that address the challenges of, and tap the opportunities afforded by, the big data revolution in our Big Data Fact Sheet.
We also want to challenge industry, research universities, and non-profits to join with the Administration to make the most of the opportunities created by Big Data. Clearly, the government can’t do this on its own. We need what the President calls an “all hands on deck” effort.
Some companies are already sponsoring Big Data-related competitions, and providing funding for university research. Universities are beginning to create new courses—and entire courses of study—to prepare the next generation of “data scientists.” Organizations like Data Without Borders are helping non-profits by providing pro bono data collection, analysis, and visualization. OSTP would be very interested in supporting the creation of a forum to highlight new public-private partnerships related to Big Data.
Tom Kalil is Deputy Director for Policy at OSTP
New U.S. Research Will Aim at Flood of Digital Data
Steve Lohr, The New York Times, March 29, 2012
The federal government is beginning a major research initiative in big data computing. The effort, which will be announced on Thursday, involves several government agencies and departments, and commitments for the programs total $200 million.
Administration officials compare the initiative to past government research support for high-speed networking and supercomputing centers, which have had an impact in areas like climate science and Web browsing software.
“This is that level of importance,” said Tom Kalil, deputy director of the White House Office of Science and Technology Policy. “The future of computing is not just big iron. It’s big data.”
Big data refers to the rising flood of digital data from many sources, including the Web, biological and industrial sensors, video, e-mail and social network communications. The emerging opportunity arises from combining these diverse data sources with improving computing tools to pinpoint profit-making opportunities, make scientific discoveries and predict crime waves, for example.
“Data, in my view, is a transformative new currency for science, engineering, education, commerce and government,” said Farnam Jahanian, head of the National Science Foundation’s computer and information science and engineering directorate. “Foundational research in data management and data analytics promise breakthrough discoveries and innovations across all disciplines.”
On Thursday, the National Science Foundation will announce a joint program with the National Institutes of Health seeking new techniques and technologies for data management, data analysis and machine learning, which is a branch of artificial intelligence.
Other departments and agencies that will be announcing big data programs at a gathering on Thursday at the American Association for the Advancement of Science in Washington include the United States Geological Survey, the Defense Department, the Defense Advanced Research Projects Agency and the Energy Department. These initiatives will mostly be seeking the best ideas from university and corporate researchers for collaborative projects.
The private sector is the leader in many applications of big data computing. Internet powers like Google and Facebook are masters at instantaneously mining Web data, click streams, search queries and messages to finely target users for online advertisements. Many major software companies, including I.B.M., Microsoft, Oracle, SAP and SAS Institute, and a growing band of start-ups, are focused on the opportunity in big data computing.
Still, there is an important complementary role for the government to play where the incentives for private investment are lacking, according to administration officials and computer scientists. Such areas, they say, include scientific discovery in fields like astronomy and physics, research into policy issues like privacy, and funding for research at universities, where the high-technology work force of the future is educated.
But for government departments and agencies promoting and mastering big data computing, there is self-interest as well.
“There is recognition by a broad range of federal agencies that further advances in big data management and analysis are critical to achieving their missions,” said Edward Lazowska, a computer scientist at the University of Washington. “It doesn’t matter whether the mission is national defense, energy efficiency, evidence-based health care, education or scientific discovery.”
At the session on Thursday, there will be presentations by scientists who are experts in big data computing.
Astronomy is a pioneering discipline for the approach. The Sloan Digital Sky Survey has used digital sensors to scan distant galaxies from an optical telescope in New Mexico, collecting vast amounts of image data that are processed with powerful computers.
The resulting three-dimensional mapping has yielded a “visual representation of the evolution of the universe,” said Alexander Szalay, a professor at Johns Hopkins University. He calls the digital sky program a “cosmic genome project.”
At Stanford University, an intriguing big-data experiment in online education is under way. Last year, three computer science courses, including videos and assignments, were put online. Hundreds of thousands of students have registered and participated in the courses.
The courses generate huge amounts of data on how students learn, what teaching strategies work best and what models do not, said Daphne Koller, a professor at the Stanford Artificial Intelligence Laboratory.
In most education research, teaching methods are tested in small groups, comparing results in different classrooms, Ms. Koller explained. With small sample groups, research conclusions tend to be uncertain, she said, and results are often not available until tests at the end of school semesters.
But in an online class of 20,000 students, whose every mouse click is tracked in real time, the research can be more definitive and more immediate, Ms. Koller said.
“If 5,000 people had the same wrong answer, it’s obvious a concept is not getting through, and you have a clear path that shows where students went wrong,” she said.
That kind of data tracking in education, she said, provides “an opportunity no one has exploited yet.”
The federal government is beginning a major research initiative in big data computing. The effort, which will be announced on Thursday, involves several government agencies and departments, and commitments for the programs total $200 million.
Administration officials compare the initiative to past government research support for high-speed networking and supercomputing centers, which have had an impact in areas like climate science and Web browsing software.
“This is that level of importance,” said Tom Kalil, deputy director of the White House Office of Science and Technology Policy. “The future of computing is not just big iron. It’s big data.”
Big data refers to the rising flood of digital data from many sources, including the Web, biological and industrial sensors, video, e-mail and social network communications. The emerging opportunity arises from combining these diverse data sources with improving computing tools to pinpoint profit-making opportunities, make scientific discoveries and predict crime waves, for example.
“Data, in my view, is a transformative new currency for science, engineering, education, commerce and government,” said Farnam Jahanian, head of the National Science Foundation’s computer and information science and engineering directorate. “Foundational research in data management and data analytics promise breakthrough discoveries and innovations across all disciplines.”
On Thursday, the National Science Foundation will announce a joint program with the National Institutes of Health seeking new techniques and technologies for data management, data analysis and machine learning, which is a branch of artificial intelligence.
Other departments and agencies that will be announcing big data programs at a gathering on Thursday at the American Association for the Advancement of Science in Washington include the United States Geological Survey, the Defense Department, the Defense Advanced Research Projects Agency and the Energy Department. These initiatives will mostly be seeking the best ideas from university and corporate researchers for collaborative projects.
The private sector is the leader in many applications of big data computing. Internet powers like Google and Facebook are masters at instantaneously mining Web data, click streams, search queries and messages to finely target users for online advertisements. Many major software companies, including I.B.M., Microsoft, Oracle, SAP and SAS Institute, and a growing band of start-ups, are focused on the opportunity in big data computing.
Still, there is an important complementary role for the government to play where the incentives for private investment are lacking, according to administration officials and computer scientists. Such areas, they say, include scientific discovery in fields like astronomy and physics, research into policy issues like privacy, and funding for research at universities, where the high-technology work force of the future is educated.
But for government departments and agencies promoting and mastering big data computing, there is self-interest as well.
“There is recognition by a broad range of federal agencies that further advances in big data management and analysis are critical to achieving their missions,” said Edward Lazowska, a computer scientist at the University of Washington. “It doesn’t matter whether the mission is national defense, energy efficiency, evidence-based health care, education or scientific discovery.”
At the session on Thursday, there will be presentations by scientists who are experts in big data computing.
Astronomy is a pioneering discipline for the approach. The Sloan Digital Sky Survey has used digital sensors to scan distant galaxies from an optical telescope in New Mexico, collecting vast amounts of image data that are processed with powerful computers.
The resulting three-dimensional mapping has yielded a “visual representation of the evolution of the universe,” said Alexander Szalay, a professor at Johns Hopkins University. He calls the digital sky program a “cosmic genome project.”
At Stanford University, an intriguing big-data experiment in online education is under way. Last year, three computer science courses, including videos and assignments, were put online. Hundreds of thousands of students have registered and participated in the courses.
The courses generate huge amounts of data on how students learn, what teaching strategies work best and what models do not, said Daphne Koller, a professor at the Stanford Artificial Intelligence Laboratory.
In most education research, teaching methods are tested in small groups, comparing results in different classrooms, Ms. Koller explained. With small sample groups, research conclusions tend to be uncertain, she said, and results are often not available until tests at the end of school semesters.
But in an online class of 20,000 students, whose every mouse click is tracked in real time, the research can be more definitive and more immediate, Ms. Koller said.
“If 5,000 people had the same wrong answer, it’s obvious a concept is not getting through, and you have a clear path that shows where students went wrong,” she said.
That kind of data tracking in education, she said, provides “an opportunity no one has exploited yet.”
Wednesday, March 28, 2012
White House: Challenges and Opportunities in Big Data
On Thursday, March 29, 2012, from 2-3 pm ET, federal government science heads from OSTP, NSF, NIH, DOE, DOD, DARPA and USGS will outline how their agencies are engaged in Big Data research.
Their remarks will be followed by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times. The event will take place at the American Association for the Advancement of Science in Washington, D.C.
Webcast at http://live.science360.gov/bigdata/
Their remarks will be followed by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times. The event will take place at the American Association for the Advancement of Science in Washington, D.C.
Webcast at http://live.science360.gov/bigdata/
Monday, March 26, 2012
Improving Decision Making in the World of Big Data
Christopher Frank, Forbes, March 25, 2012
Christopher J Frank is Vice President in Global Marketplace Insights at American Express. He is co-author of Drinking From the Fire Hose: Making Smarter Decisions in a Data Overloaded World.
'Big Data' is having its 15-minutes of fame. It is the hot term referring to the increasingly large datasets of information being amassed as a result of our social, mobile, and digital world. In the past 12-months, the use of the term in the U.S. has increased 1,211% on the internet.
'Big Data' is having its 15-minutes of fame. It is the hot term referring to the increasingly large datasets of information being amassed as a result of our social, mobile, and digital world. In the past 12-months, the use of the term in the U.S. has increased 1,211% on the internet.
Lack of data is not the issue. Lack of strategy is.
IBM estimates that every day 2.5 quintillion bytes of data are created – so much that 90% of the data in the world today has been created in the last two years. It is mind-boggling. The irony is we have more information available, but we feel less informed.
Bridging the gap between big data and big strategy starts with effectively framing the problem. If you have read Moneyball (Lewis; Norton) or seen the movie, you witnessed the power of big data – it is the story about the ability to compete and win with few resources and limited dollars. This sums up the hopes and challenge of business today.
The Oakland A's did it in baseball, but some companies have their own Moneyball story. In the thought-provoking book Scoring Points (Humby, Hunt, Phillips; Kogan Page), the authors discuss how Tesco, the UK-based grocer, analyzed customer purchase behavior to redefine their brand, drive loyalty, and fuel rapid growth. In SuperCrunches: Why Thinking-by-Numbers Is the New Way to Be Smart (Ayres; Random House), the authors show how number- crunching affects you in ways you might never imagine. Recently, the New York Times Magazine ran a fascinating article "How Companies Learn Your Secrets", discussing how Target Corporation studies consumer habits and data to drive explosive sales.
These examples illustrate the potential of big data – being smarter with the right resources. Success in doing so is becoming a reality with faster information, better modeling and cheaper computing power. A small army of consultants are amassing to help analyze the large datasets. In a study by McKinsey's Business Technology Office and MGI the firm calculated the U.S. faces a shortage of 140,000 to 190,000 people with analytical expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis. I want to emphasize this last point — the biggest gap is the lack of skilled managers to make decisions based on analysis by a factor of 10x.
Growing talent and building teams to make analytic-based decisions is the key to realizing the value of big data.
One of the most effective ways to start building these capabilities on your team is the "IWIK" discussion. The concept is based on asking a simple but powerful question: 'I wish I knew' (IWIK). The question is a catalyst for a brainstorming discussion to identify wants and needs with respect to big data. IWIK will uncover the strategic, essential questions that can be answered through big data. It will define the indispensable information you need to move forward.
Identifying and answering the relevant IWIKs for your business will enable you to become smarter along two paths: avoid getting lost in big data, and using the right data to discover the real wants and needs of your customers. In the new reality of big data, hyper-competitive businesses are linking and labeling perception information, transactional data and layering in social behaviors and then using the IWIK to define the data rails to shape analytic-based decisions. Therefore, the focus should be less on 'big' data and more on 'essential' data.
What is your IWIK?
Christopher J Frank is Vice President in Global Marketplace Insights at American Express. He is co-author of Drinking From the Fire Hose: Making Smarter Decisions in a Data Overloaded World.
IBM estimates that every day 2.5 quintillion bytes of data are created – so much that 90% of the data in the world today has been created in the last two years. It is mind-boggling. The irony is we have more information available, but we feel less informed.
Bridging the gap between big data and big strategy starts with effectively framing the problem. If you have read Moneyball (Lewis; Norton) or seen the movie, you witnessed the power of big data – it is the story about the ability to compete and win with few resources and limited dollars. This sums up the hopes and challenge of business today.
The Oakland A's did it in baseball, but some companies have their own Moneyball story. In the thought-provoking book Scoring Points (Humby, Hunt, Phillips; Kogan Page), the authors discuss how Tesco, the UK-based grocer, analyzed customer purchase behavior to redefine their brand, drive loyalty, and fuel rapid growth. In SuperCrunches: Why Thinking-by-Numbers Is the New Way to Be Smart (Ayres; Random House), the authors show how number- crunching affects you in ways you might never imagine. Recently, the New York Times Magazine ran a fascinating article "How Companies Learn Your Secrets", discussing how Target Corporation studies consumer habits and data to drive explosive sales.
These examples illustrate the potential of big data – being smarter with the right resources. Success in doing so is becoming a reality with faster information, better modeling and cheaper computing power. A small army of consultants are amassing to help analyze the large datasets. In a study by McKinsey's Business Technology Office and MGI the firm calculated the U.S. faces a shortage of 140,000 to 190,000 people with analytical expertise and 1.5 million managers and analysts with the skills to understand and make decisions based on the analysis. I want to emphasize this last point — the biggest gap is the lack of skilled managers to make decisions based on analysis by a factor of 10x.
Growing talent and building teams to make analytic-based decisions is the key to realizing the value of big data.
One of the most effective ways to start building these capabilities on your team is the "IWIK" discussion. The concept is based on asking a simple but powerful question: 'I wish I knew' (IWIK). The question is a catalyst for a brainstorming discussion to identify wants and needs with respect to big data. IWIK will uncover the strategic, essential questions that can be answered through big data. It will define the indispensable information you need to move forward.
Identifying and answering the relevant IWIKs for your business will enable you to become smarter along two paths: avoid getting lost in big data, and using the right data to discover the real wants and needs of your customers. In the new reality of big data, hyper-competitive businesses are linking and labeling perception information, transactional data and layering in social behaviors and then using the IWIK to define the data rails to shape analytic-based decisions. Therefore, the focus should be less on 'big' data and more on 'essential' data.
What is your IWIK?
Christopher J Frank is Vice President in Global Marketplace Insights at American Express. He is co-author of Drinking From the Fire Hose: Making Smarter Decisions in a Data Overloaded World.
Sunday, March 25, 2012
Big Data in NYT: Just the Facts. Yes, All of Them.
Quentin Hardy, The New York Times, March 24, 2012
AT 7 years old, Gilad Elbaz wrote, “I want to be a rich mathematician and very smart.” That, he figured, would help him “discover things like time machines, robots and machines that can answer any question.”
In the 34 years since, Mr. Elbaz has accomplished big chunks of these goals. He has built Web-traversing software robots and answered some very big questions for Google, along the way becoming a millionaire several hundred times over.
His time-machine plans, however, have been ditched for something he finds more important: trying to identify every fact in the world, and to hold them all in a company he calls Factual.
“The world is one big data problem,” Mr. Elbaz says from his headquarters, a quiet office 14 floors above the Los Angeles Country Club. He is a slim, soft-spoken man who weaves in his chair when an idea excites him. “What if you could spot any error, as soon as you wrote it? Factual is definitely a new thing that will change business, and a valuable new tool for computing.”
In the booming world of Big Data, where once-unimaginably huge amounts of information are scoured for world-changing discoveries, Mr. Elbaz may be the most influential inventor and investor. Besides Factual, he has interests in 30 start-ups, including an incubator in San Francisco dedicated to Big Data. Factual’s headquarters, in a high-rise on the Avenue of the Stars, hosts seminars for a data community he hopes to foster in the Los Angeles area.
Mr. Elbaz also serves on the boards of the California Institute of Technology, his alma mater, and the X Prize Foundation, which offers cash prizes to teams that meet challenges in space flight, medicine and genomics. The company he sold to Google, Applied Semantics, is the basis of Google’s AdSense business, which brings Google close to $10 billion in revenue annually.
While valued for his investments and guidance, Mr. Elbaz remains relatively little-known. He is so self-effacing that he recently walked through a conference of 3,000 data scientists, recognized only by the staff members of one of his investments. He lives quietly with his wife, a former federal prosecutor, and his three children in a modest ranch house in West Hollywood. For fun, he plays basketball at a local sports club.
His mental and financial assets, he says, are like gifts he needs to deploy so the world works better.
“If all data was clear, a lot fewer people would subtract value from the world,” he says. “A lot more people would add value.”
Creating clear, reliable data could also make Factual a very big company.
“Gil is pretty far ahead of the rest of us, the one entrepreneur where it takes a few meetings before I really understand everything he is talking about,” says Ben Horowitz, a venture capitalist who backed Factual through his firm, Andreessen Horowitz. “Three years ago, he thought Factual was his biggest chance to change the world. Over time, the world has moved his way.”
Since its start in 2008, Factual has absorbed what Mr. Elbaz terms “many billions of individual facts we’ve collated.”
Geared to both big companies and smaller software developers, it includes available government data, terabytes of corporate data and information on 60 million places in 50 countries, each described by 17 to 40 attributes. Factual knows more than 800,000 restaurants in 30 different ways, including location, ownership and ratings by diners and health boards. It also contains information on half a billion Web pages, a list of America’s high schools and data on the offices, specialties and insurance preferences of 1.8 million United States health care professionals. There are also listings of 14,000 wine grape varietals, of military aircraft accidents from 1950 to 1974, and of body masses of major celebrities. Odd facts matter too, Mr. Elbaz notes.
He keeps 500 terabytes of storage near Factual’s headquarters. That’s about twice the amount needed to hold the entire Library of Congress. He has more data stored inside Amazon’s giant cloud of computers. His statisticians have cleaned and corrected data to account for things like how different health departments score sanitation, whether the term “middle school” means two years or three in a particular town, and whether there were revisions between an original piece of data and its duplicate.
Factual’s plan, outlined in a big orange room with a few tables and walled with whiteboards, is to build the world’s chief reference point for thousands of interconnected supercomputing clouds. The digital world is expected to hold a collective 2.7 zettabytes of data by year-end, an amount roughly equivalent to 700 billion DVDs. Factual, which now has 50 employees, could prove immensely valuable as this world grows and these databases begin to interact.
FACTUAL sells data to corporations and independent software developers on a sliding scale, based on how much the information is used. Small data feeds for things like prototypes are free; contracts with its biggest customers run into the millions. Sometimes, Factual trades data with other companies, building its resources.
Some current uses are for adding information like restaurant locations to cellphone maps, or for planning sales campaigns. But more broadly, Factual is meant for the heart of a great business of our age: using all the cloud-based data and algorithms to find patterns in nature and society, for scientists to observe and businesses to exploit.
“Data has always been seen as just a side effect in computing, something you look up while you are doing work,” Mr. Elbaz says. “We see it as a whole separate layer that everyone is going to have to tap into, data you want to solve a problem, but that you might not have yourself, and completely reliable.”
A restaurant chain, for example, might use Factual to figure out whether a new location is near the competition, and how the locals have talked about the place on Yelp, the social ratings site. Checking for gas stations near the restaurant can indicate how many cars come off the highway. The chain can also employ Factual to see where it is mentioned on the Web, or to correct what other people are saying about it.
Financed with $27 million by a constellation of Silicon Valley luminaries, Factual remains closely held. But it already has thousands of customers. Facebook, CitySearch, AT&T and others use it for information about places. Newsweek used the database to help rank America’s greenest companies.
Others use Factual data for tasks like product planning and customer care. There are no profits yet, as Mr. Elbaz puts money into more data sets and talent, which already includes advanced mathematicians, data scientists from LinkedIn and Google, and at least one specialist in late Roman archaeology.
Competitors in the new industry include Microsoft, which says its Windows Azure Marketplace has “trillions of data points,” as well as a language translator. People can sell data sets to Azure, too. Infochimps offers geographic and social data, among other kinds, while companies like Gnip and Datasift offer insights from Twitter and other social sites. Wolfram Alpha, founded by another mathematician, has both data and computations that are used by Apple’s Siri, among others.
And a young company called ClearStory, also financed by Andreessen Horowitz, is trying to tie together all of these companies, often called data marts, in a way ordinary people can use. There are also several open-source data repositories, with public and private information that developers plug into their algorithms.
Several other data specialists, mostly from Google, have left their jobs to wrangle lots of information in new ways. David Friedberg, a former product manager at Google, has started the Climate Corporation, which uses government data on weather, soil porosity and the root structures of wheat and soybeans to write crop insurance.
Mr. Elbaz is also an investor in Kaggle, which awards cash for finding data patterns. It was used by NASA, for example, to find a better way to measure the shape of galaxies; in the first week of competition, a Ph.D. student in glacier mapping had outperformed NASA’s algorithms. He has also put money into ZestCash, which makes payday loans that are cheaper than the industry’s average, judging risk via criteria like cellphone bills and how its applicants read the ZestCash Web site.
The ZestCash C.E.O., Douglas Merrill, once ran Google’s internal information systems.
“We feel like all data is credit data, we just don’t know how to use it yet,” he says. “This is the math we all learned at Google. A page was important for what was on it, but also for how good the grammar was, what the type font was, when it was created or edited. Everything. What Gil is doing at Factual is the same. Data matters. More data is always better.”
MR. ELBAZ was born in Washington, D.C., and grew up in Ohio, Texas and Florida. His father, who was born in Morocco and grew up in Israel, was a school principal and professor of Hebrew literature. His mother, a journalist, died when Mr. Elbaz was 18. At age 3, he began writing a repeating series of numbers at preschool. He read almanacs and enjoyed watching the crawl of stock prices on TV, seeking patterns.
“He would go to a lot of math competitions, and come out with three or four prizes,” says Nissim Elbaz, Mr. Elbaz’s father. “In between the math contests he’d take tests in physics for fun. When I would tell him what a genius he was, he’d give me a dirty look, so I learned to keep it in my heart.”
The elder Mr. Elbaz recalls that when he tried to explain the Isreali-Palestinian conflict to him, the son replied that the hatred would end if the two sides could just agree on the facts.
From an early age, Mr. Elbaz would also figure out math-related businesses — like buying the entire supply of a single brand of baseball cards in El Paso, Tex., then reselling them at three times the money at a memorabilia convention.
“We’d do lotteries based on guessing the number of marbles in a jar,” says Eytan Elbaz, his younger brother, who has worked with Gil and now has two start-ups of his own. “When he was 16, he held a contest based on rolling a Yahtzee dice. He stayed up the night before making a spreadsheet to figure out all the payouts against what we’d take in.” Mr. Elbaz’s other brother, Noam, has spent the last decade studying at a yeshiva in Israel.
At Caltech, Mr. Elbaz majored in applied science and economics. Interested in the subject of monopolies, he won an award for a paper that determined that companies would take financial losses to corner their markets.
He worked for I.B.M. for two years, looking at the use of computers in problems of manufacturing, then went to Sybase, a database company. This was in the early 1990s, when I.B.M. was stumbling in the transition from mainframe computers to servers and PCs.
His younger brother says he thinks that the experience changed him. Many employees were “just trying to hold on to their jobs, not working together for the company,” Eytan says. He recalls how Gil, concerned about how employees were hoarding their data, “started talking about how much better it would be if people shared data.”
Mr. Elbaz then joined a semiconductor start-up called Microunity and became a consultant, saving money and playing the stock market to help finance his own first business. His father gave him $10,000 to invest for him, which Mr. Elbaz tripled in 18 months. When Mr. Elbaz and a Caltech friend decided to form a company in 1998 — it became Applied Semantics — his father told him to put the stock winnings into it.
Applied Semantics software quickly scanned thousands of Web pages for their meaning. By parsing content, it could tell businesses what kind of ads would work well on a particular page. It had 45 employees and was profitable when Google acquired it in 2003 for $102 million in cash and pre-I.P.O. stock.
While Mr. Elbaz would not say how much he made from the deal, his father’s $30,000 from the stock investments was eventually worth $18 million. “He certainly changed my retirement,” Nissim Elbaz says.
Mr. Elbaz became the head of Google’s engineering office in Santa Monica, Calif., near where he lives with his wife, Elyssa, and three sons. He has donated several million dollars to a handful of causes, including science education, environmental efforts and an organization that helps Los Angeles nonprofit groups. He has also donated to Common Crawl, a Google-type Web examination tool that researchers can access through Amazon’s computers.
“Having money is overrated when you are brought up not to believe you are entitled to it,” he says. “You can make enough money to not need things, or you can just not need things.”
In 2007, Mr. Elbaz left Google to start Factual. When Mr. Horowitz, who along with Mark Andreessen runs Andreessen Horowitz, was asked to invest in Factual in 2009, he struggled with the idea that Mr. Elbaz would want to work hard on another start-up when he was already rich. But when Mr. Elbaz described his palace of facts, Mr. Horowitz said he recognized a true believer.
“I asked him, ‘Aren’t you too rich to build this company?’ ” Mr. Horowitz recalls. “He gave me one of the longest and most thought-out answers I’d ever heard. He thinks this is a chance to change the world — that matters to him more than money.” Mr. Horowitz says Mr. Elbaz told him he needed the money as an incentive for the engineers, and that he needed to reach his goal while his mind was still strong enough.
“I eventually realized this was not a ‘too rich to work hard’ problem.” Mr. Horowitz said.
Other investors in Factual include Ron Conway, Esther Dyson, Index Ventures and the Founder Collective.
FACTUAL also has offices in Shanghai and in Palo Alto, Calif., where Mr. Elbaz wants to add more talent from Silicon Valley. His first two employees in Palo Alto were Tim Chklovski, with a doctorate from M.I.T. in artificial intelligence, and Tyler Bell, who worked at Yahoo on maps after a decade at Oxford piecing together how ancient Rome became early Europe.
“Roman amphorae get dug up around Europe, and get described in all sorts of different ways, in different languages,” Mr. Bell says. “What we do here is the same kind of restoration. We started to learn it well when we had one million data sets uploaded. The early stuff was things like senators and how they voted, or ZIP codes, types of cigars, lists of video games. Merging ZIP codes is easy. Merging different ways people feel about toys is hard.”
Part of the difficulty, even among employees, is deciding how much data is enough. “For sure, we want the correct name and location of every gas station on the globe,” Mr. Bell says. “Not the price changes at every station.”
“Wait a minute, I’d like to know every gallon of gasoline that flows around the world,” Mr. Chklovski cuts in. “That might take us 20 years, but it would be interesting.”
At most start-ups, talk about doing the same kind of thing, only bigger and better, 20 years from now might seem like a marriage of the delusional and the dull. Mr. Elbaz and his team, however, say they feel that it makes sense. Telling everyone the true facts of the world is at least the work of a lifetime.
”Lately, I’ve been thinking that we need to get more personal data,” Mr. Elbaz says. He doesn’t mean names and addresses, but their genetic information, what they ate, when and where they exercised — ideally, for everyone on the planet, now and forever. “I want to figure out a way,” he says, “to get people to leave their data to science.”
AT 7 years old, Gilad Elbaz wrote, “I want to be a rich mathematician and very smart.” That, he figured, would help him “discover things like time machines, robots and machines that can answer any question.”
In the 34 years since, Mr. Elbaz has accomplished big chunks of these goals. He has built Web-traversing software robots and answered some very big questions for Google, along the way becoming a millionaire several hundred times over.
His time-machine plans, however, have been ditched for something he finds more important: trying to identify every fact in the world, and to hold them all in a company he calls Factual.
“The world is one big data problem,” Mr. Elbaz says from his headquarters, a quiet office 14 floors above the Los Angeles Country Club. He is a slim, soft-spoken man who weaves in his chair when an idea excites him. “What if you could spot any error, as soon as you wrote it? Factual is definitely a new thing that will change business, and a valuable new tool for computing.”
In the booming world of Big Data, where once-unimaginably huge amounts of information are scoured for world-changing discoveries, Mr. Elbaz may be the most influential inventor and investor. Besides Factual, he has interests in 30 start-ups, including an incubator in San Francisco dedicated to Big Data. Factual’s headquarters, in a high-rise on the Avenue of the Stars, hosts seminars for a data community he hopes to foster in the Los Angeles area.
Mr. Elbaz also serves on the boards of the California Institute of Technology, his alma mater, and the X Prize Foundation, which offers cash prizes to teams that meet challenges in space flight, medicine and genomics. The company he sold to Google, Applied Semantics, is the basis of Google’s AdSense business, which brings Google close to $10 billion in revenue annually.
While valued for his investments and guidance, Mr. Elbaz remains relatively little-known. He is so self-effacing that he recently walked through a conference of 3,000 data scientists, recognized only by the staff members of one of his investments. He lives quietly with his wife, a former federal prosecutor, and his three children in a modest ranch house in West Hollywood. For fun, he plays basketball at a local sports club.
His mental and financial assets, he says, are like gifts he needs to deploy so the world works better.
“If all data was clear, a lot fewer people would subtract value from the world,” he says. “A lot more people would add value.”
Creating clear, reliable data could also make Factual a very big company.
“Gil is pretty far ahead of the rest of us, the one entrepreneur where it takes a few meetings before I really understand everything he is talking about,” says Ben Horowitz, a venture capitalist who backed Factual through his firm, Andreessen Horowitz. “Three years ago, he thought Factual was his biggest chance to change the world. Over time, the world has moved his way.”
Since its start in 2008, Factual has absorbed what Mr. Elbaz terms “many billions of individual facts we’ve collated.”
Geared to both big companies and smaller software developers, it includes available government data, terabytes of corporate data and information on 60 million places in 50 countries, each described by 17 to 40 attributes. Factual knows more than 800,000 restaurants in 30 different ways, including location, ownership and ratings by diners and health boards. It also contains information on half a billion Web pages, a list of America’s high schools and data on the offices, specialties and insurance preferences of 1.8 million United States health care professionals. There are also listings of 14,000 wine grape varietals, of military aircraft accidents from 1950 to 1974, and of body masses of major celebrities. Odd facts matter too, Mr. Elbaz notes.
He keeps 500 terabytes of storage near Factual’s headquarters. That’s about twice the amount needed to hold the entire Library of Congress. He has more data stored inside Amazon’s giant cloud of computers. His statisticians have cleaned and corrected data to account for things like how different health departments score sanitation, whether the term “middle school” means two years or three in a particular town, and whether there were revisions between an original piece of data and its duplicate.
Factual’s plan, outlined in a big orange room with a few tables and walled with whiteboards, is to build the world’s chief reference point for thousands of interconnected supercomputing clouds. The digital world is expected to hold a collective 2.7 zettabytes of data by year-end, an amount roughly equivalent to 700 billion DVDs. Factual, which now has 50 employees, could prove immensely valuable as this world grows and these databases begin to interact.
FACTUAL sells data to corporations and independent software developers on a sliding scale, based on how much the information is used. Small data feeds for things like prototypes are free; contracts with its biggest customers run into the millions. Sometimes, Factual trades data with other companies, building its resources.
Some current uses are for adding information like restaurant locations to cellphone maps, or for planning sales campaigns. But more broadly, Factual is meant for the heart of a great business of our age: using all the cloud-based data and algorithms to find patterns in nature and society, for scientists to observe and businesses to exploit.
“Data has always been seen as just a side effect in computing, something you look up while you are doing work,” Mr. Elbaz says. “We see it as a whole separate layer that everyone is going to have to tap into, data you want to solve a problem, but that you might not have yourself, and completely reliable.”
A restaurant chain, for example, might use Factual to figure out whether a new location is near the competition, and how the locals have talked about the place on Yelp, the social ratings site. Checking for gas stations near the restaurant can indicate how many cars come off the highway. The chain can also employ Factual to see where it is mentioned on the Web, or to correct what other people are saying about it.
Financed with $27 million by a constellation of Silicon Valley luminaries, Factual remains closely held. But it already has thousands of customers. Facebook, CitySearch, AT&T and others use it for information about places. Newsweek used the database to help rank America’s greenest companies.
Others use Factual data for tasks like product planning and customer care. There are no profits yet, as Mr. Elbaz puts money into more data sets and talent, which already includes advanced mathematicians, data scientists from LinkedIn and Google, and at least one specialist in late Roman archaeology.
Competitors in the new industry include Microsoft, which says its Windows Azure Marketplace has “trillions of data points,” as well as a language translator. People can sell data sets to Azure, too. Infochimps offers geographic and social data, among other kinds, while companies like Gnip and Datasift offer insights from Twitter and other social sites. Wolfram Alpha, founded by another mathematician, has both data and computations that are used by Apple’s Siri, among others.
And a young company called ClearStory, also financed by Andreessen Horowitz, is trying to tie together all of these companies, often called data marts, in a way ordinary people can use. There are also several open-source data repositories, with public and private information that developers plug into their algorithms.
Several other data specialists, mostly from Google, have left their jobs to wrangle lots of information in new ways. David Friedberg, a former product manager at Google, has started the Climate Corporation, which uses government data on weather, soil porosity and the root structures of wheat and soybeans to write crop insurance.
Mr. Elbaz is also an investor in Kaggle, which awards cash for finding data patterns. It was used by NASA, for example, to find a better way to measure the shape of galaxies; in the first week of competition, a Ph.D. student in glacier mapping had outperformed NASA’s algorithms. He has also put money into ZestCash, which makes payday loans that are cheaper than the industry’s average, judging risk via criteria like cellphone bills and how its applicants read the ZestCash Web site.
The ZestCash C.E.O., Douglas Merrill, once ran Google’s internal information systems.
“We feel like all data is credit data, we just don’t know how to use it yet,” he says. “This is the math we all learned at Google. A page was important for what was on it, but also for how good the grammar was, what the type font was, when it was created or edited. Everything. What Gil is doing at Factual is the same. Data matters. More data is always better.”
MR. ELBAZ was born in Washington, D.C., and grew up in Ohio, Texas and Florida. His father, who was born in Morocco and grew up in Israel, was a school principal and professor of Hebrew literature. His mother, a journalist, died when Mr. Elbaz was 18. At age 3, he began writing a repeating series of numbers at preschool. He read almanacs and enjoyed watching the crawl of stock prices on TV, seeking patterns.
“He would go to a lot of math competitions, and come out with three or four prizes,” says Nissim Elbaz, Mr. Elbaz’s father. “In between the math contests he’d take tests in physics for fun. When I would tell him what a genius he was, he’d give me a dirty look, so I learned to keep it in my heart.”
The elder Mr. Elbaz recalls that when he tried to explain the Isreali-Palestinian conflict to him, the son replied that the hatred would end if the two sides could just agree on the facts.
From an early age, Mr. Elbaz would also figure out math-related businesses — like buying the entire supply of a single brand of baseball cards in El Paso, Tex., then reselling them at three times the money at a memorabilia convention.
“We’d do lotteries based on guessing the number of marbles in a jar,” says Eytan Elbaz, his younger brother, who has worked with Gil and now has two start-ups of his own. “When he was 16, he held a contest based on rolling a Yahtzee dice. He stayed up the night before making a spreadsheet to figure out all the payouts against what we’d take in.” Mr. Elbaz’s other brother, Noam, has spent the last decade studying at a yeshiva in Israel.
At Caltech, Mr. Elbaz majored in applied science and economics. Interested in the subject of monopolies, he won an award for a paper that determined that companies would take financial losses to corner their markets.
He worked for I.B.M. for two years, looking at the use of computers in problems of manufacturing, then went to Sybase, a database company. This was in the early 1990s, when I.B.M. was stumbling in the transition from mainframe computers to servers and PCs.
His younger brother says he thinks that the experience changed him. Many employees were “just trying to hold on to their jobs, not working together for the company,” Eytan says. He recalls how Gil, concerned about how employees were hoarding their data, “started talking about how much better it would be if people shared data.”
Mr. Elbaz then joined a semiconductor start-up called Microunity and became a consultant, saving money and playing the stock market to help finance his own first business. His father gave him $10,000 to invest for him, which Mr. Elbaz tripled in 18 months. When Mr. Elbaz and a Caltech friend decided to form a company in 1998 — it became Applied Semantics — his father told him to put the stock winnings into it.
Applied Semantics software quickly scanned thousands of Web pages for their meaning. By parsing content, it could tell businesses what kind of ads would work well on a particular page. It had 45 employees and was profitable when Google acquired it in 2003 for $102 million in cash and pre-I.P.O. stock.
While Mr. Elbaz would not say how much he made from the deal, his father’s $30,000 from the stock investments was eventually worth $18 million. “He certainly changed my retirement,” Nissim Elbaz says.
Mr. Elbaz became the head of Google’s engineering office in Santa Monica, Calif., near where he lives with his wife, Elyssa, and three sons. He has donated several million dollars to a handful of causes, including science education, environmental efforts and an organization that helps Los Angeles nonprofit groups. He has also donated to Common Crawl, a Google-type Web examination tool that researchers can access through Amazon’s computers.
“Having money is overrated when you are brought up not to believe you are entitled to it,” he says. “You can make enough money to not need things, or you can just not need things.”
In 2007, Mr. Elbaz left Google to start Factual. When Mr. Horowitz, who along with Mark Andreessen runs Andreessen Horowitz, was asked to invest in Factual in 2009, he struggled with the idea that Mr. Elbaz would want to work hard on another start-up when he was already rich. But when Mr. Elbaz described his palace of facts, Mr. Horowitz said he recognized a true believer.
“I asked him, ‘Aren’t you too rich to build this company?’ ” Mr. Horowitz recalls. “He gave me one of the longest and most thought-out answers I’d ever heard. He thinks this is a chance to change the world — that matters to him more than money.” Mr. Horowitz says Mr. Elbaz told him he needed the money as an incentive for the engineers, and that he needed to reach his goal while his mind was still strong enough.
“I eventually realized this was not a ‘too rich to work hard’ problem.” Mr. Horowitz said.
Other investors in Factual include Ron Conway, Esther Dyson, Index Ventures and the Founder Collective.
FACTUAL also has offices in Shanghai and in Palo Alto, Calif., where Mr. Elbaz wants to add more talent from Silicon Valley. His first two employees in Palo Alto were Tim Chklovski, with a doctorate from M.I.T. in artificial intelligence, and Tyler Bell, who worked at Yahoo on maps after a decade at Oxford piecing together how ancient Rome became early Europe.
“Roman amphorae get dug up around Europe, and get described in all sorts of different ways, in different languages,” Mr. Bell says. “What we do here is the same kind of restoration. We started to learn it well when we had one million data sets uploaded. The early stuff was things like senators and how they voted, or ZIP codes, types of cigars, lists of video games. Merging ZIP codes is easy. Merging different ways people feel about toys is hard.”
Part of the difficulty, even among employees, is deciding how much data is enough. “For sure, we want the correct name and location of every gas station on the globe,” Mr. Bell says. “Not the price changes at every station.”
“Wait a minute, I’d like to know every gallon of gasoline that flows around the world,” Mr. Chklovski cuts in. “That might take us 20 years, but it would be interesting.”
At most start-ups, talk about doing the same kind of thing, only bigger and better, 20 years from now might seem like a marriage of the delusional and the dull. Mr. Elbaz and his team, however, say they feel that it makes sense. Telling everyone the true facts of the world is at least the work of a lifetime.
”Lately, I’ve been thinking that we need to get more personal data,” Mr. Elbaz says. He doesn’t mean names and addresses, but their genetic information, what they ate, when and where they exercised — ideally, for everyone on the planet, now and forever. “I want to figure out a way,” he says, “to get people to leave their data to science.”
Tuesday, March 20, 2012
Internet accounts for 4.7% of U.S. economy
Annalyn Censky @CNNMoney March 19, 2012
![]()
NEW YORK (CNNMoney) -- The Internet contributes more to the American economy than the entire federal government, according to a new study by the Boston Consulting Group.
The Internet accounted for $684 billion, or 4.7% of all U.S. economic activity in 2010, Boston Consulting Group found. By way of comparison, the federal government, contributed $625 billion, or 4.3%, to the nation's output.
If it was considered its own separate industry, the Internet would also be larger than America's education, construction or agricultural sectors.
In the retail sphere alone, e-commerce accounted for 5% of U.S. sales in 2010.
"All businesses are increasingly digital and need to think about how to take advantage of these opportunities," said Dominic Field, a BCG partner and co-author of the report. "And for policymakers, we would hope they recognize the importance of Internet growth and making sure their countries are taking advantage of these opportunities."
As a share of gross domestic product, only three countries have larger Internet economies: the United Kingdom, South Korea and China. The U.S. is tied with Japan.
Boston Consulting Group predicts the Internet will grow about 10% a year through 2016 in the Group of 20 nations. It will grow nearly twice as fast in emerging markets as in developed economies, with Argentina and India accounting for the fastest growth, the study said.
"The U.S. is relatively mature as an Internet economy, whereas some of the developing economies are further behind -- so their growth rates are higher," Field said.
Granted, measuring the full impact of the Internet can be a fuzzy matter. In the Boston Consulting Group study, the researchers included the impact of e-commerce, what consumers pay to access the Internet and money spent by businesses and the government on building Internet infrastructure.
The study also looked at some other slightly silly metrics to measure how consumers value the Internet. It found for example, that 77% of Americans would be willing to give up chocolate for an entire year, rather than go without the Internet. Only 21% though would give up sex, and an even smaller 7% would go without a shower.
Americans perceive the Internet to be worth about $3,000 a year, even though on average, they spend only $472 each year on devices, applications, services and access.![]()
NEW YORK (CNNMoney) -- The Internet contributes more to the American economy than the entire federal government, according to a new study by the Boston Consulting Group.
The Internet accounted for $684 billion, or 4.7% of all U.S. economic activity in 2010, Boston Consulting Group found. By way of comparison, the federal government, contributed $625 billion, or 4.3%, to the nation's output.
If it was considered its own separate industry, the Internet would also be larger than America's education, construction or agricultural sectors.
In the retail sphere alone, e-commerce accounted for 5% of U.S. sales in 2010.
"All businesses are increasingly digital and need to think about how to take advantage of these opportunities," said Dominic Field, a BCG partner and co-author of the report. "And for policymakers, we would hope they recognize the importance of Internet growth and making sure their countries are taking advantage of these opportunities."
As a share of gross domestic product, only three countries have larger Internet economies: the United Kingdom, South Korea and China. The U.S. is tied with Japan.
Boston Consulting Group predicts the Internet will grow about 10% a year through 2016 in the Group of 20 nations. It will grow nearly twice as fast in emerging markets as in developed economies, with Argentina and India accounting for the fastest growth, the study said.
"The U.S. is relatively mature as an Internet economy, whereas some of the developing economies are further behind -- so their growth rates are higher," Field said.
Granted, measuring the full impact of the Internet can be a fuzzy matter. In the Boston Consulting Group study, the researchers included the impact of e-commerce, what consumers pay to access the Internet and money spent by businesses and the government on building Internet infrastructure.
The study also looked at some other slightly silly metrics to measure how consumers value the Internet. It found for example, that 77% of Americans would be willing to give up chocolate for an entire year, rather than go without the Internet. Only 21% though would give up sex, and an even smaller 7% would go without a shower.
Americans perceive the Internet to be worth about $3,000 a year, even though on average, they spend only $472 each year on devices, applications, services and access.
Monday, March 19, 2012
Big Data for the Rest of Us, In One Startup
Quentin Hardy, March 19, 2012, The New York Times
The best insight into the current state of the Big Data business may be a five person start-up in Palo Alto, Calif. While the industry mostly gathers lots of data, or stores it in new cloud-based databases, the startup, ClearStory Data, is building the biggest possible base of data consumers.
ClearStory, which was formed last summer in Palo Alto, Calif., is making software for ordinary business professionals. They will be able to blend their own corporate data with the large amounts of publicly-available data, in search of new statistical insights. There just aren’t enough statisticians, the company figures, to address the demand corporations will have for Big Data.
“We’re about making data consumable,” says Sharmila Shahani-Mulligan, ClearStory’s founder and chief executive. “The world is talking about the size of Big Data sources, but at the end of the day it will be about the ease of consumption.” Indeed, the McKinsey Global Institute has projected that the U.S. needs up to 190,000 people with “analytical expertise” than it has.
The idea of combining proprietary corporate data with public data is hardly new; fast food companies have long looked at census data to figure out where to put outlets, and what to feed people there. The public data sources for ClearStory, will largely be the new category of “Data Marts,” like Datasift, Factual, the Windows Azure Data Marketplace from Microsoft, and Infochimps. These data marts collectively have perhaps millions of data types, from Twitter feeds to mean temperatures and store locations.
“There aren’t enough experts at companies to handle all these data sources,” Ms. Shahani-Mulligan says. “Brick and mortar stores in the Midwest can’t get the kind of data scientists LinkedIn has. We can give them that.” The company offers a range of algorithms to wrangle the data. Longer term, ClearStory hopes to get companies to start sharing more of their own internal data too, what Ms. Shahani-Mulligan calls “the gold locked up in 30 years of relational databases.”
Drawing off all of the data sources could be powerful, if ClearStory can manage to maintain an even level of quality and reliability among its data sources. That already bedevils many data marts. If ClearStory offers sloppy data to customers, however, it will hurt ClearStory.
Another challenge will be designing an intuitive Web-based user experience, so the average marketer or communications professional isn’t flummoxed by statistics. Good “U.E.” people are among the toughest people to find in tech.
ClearStory is a good trend barometer in other ways. The product, which is expected to be ready for public consumption late this summer, will be offered free to casual users. The idea is that enough customers far away from the corporate information technology department will come to depend on ClearStory, and eventually I.T. will want to incorporate a more sophisticated paid version. It is a plan that worked for Yammer, a corporate messaging tool, and Dropbox, with offers storage in the cloud, as well as the Linux operating system.
Ms. Shahani-Mulligan, a marketing veteran of Netscape, Opsware, and Cloudera, is backed by Google Ventures, Andreessen Horowitz, Khosla Ventures, among others. That makes it one of the more “hot backer”-compliant ventures: Google Ventures is specializing in companies with a statistical bent, Andreessen Horowitz, is assembling an ecology of companies in cloud computing, and Vinod Khosla is a renowned tech visionary.
Even with backers like that, ClearStory is making itself visible a lot earlier than most startups risking it will hit the radar of potential competitors even before it has a product. In another sign of the times, and the boom (or, possibly, the bubble) in Big Data, ClearStory needs the publicity in order to hire in an increasingly competitive talent market, however.
“There is a war for talent,” says Ben Horowitz, ClearStory’s backer at Andreessen Horowitz. “Getting the first five people is easy, but the next 20 is hard, when you are selling on your vision and momentum.” Besides, he says, “big companies don’t react to potential threats. They only react to real threats. There are just too many threats out there.”
The best insight into the current state of the Big Data business may be a five person start-up in Palo Alto, Calif. While the industry mostly gathers lots of data, or stores it in new cloud-based databases, the startup, ClearStory Data, is building the biggest possible base of data consumers.
ClearStory, which was formed last summer in Palo Alto, Calif., is making software for ordinary business professionals. They will be able to blend their own corporate data with the large amounts of publicly-available data, in search of new statistical insights. There just aren’t enough statisticians, the company figures, to address the demand corporations will have for Big Data.
“We’re about making data consumable,” says Sharmila Shahani-Mulligan, ClearStory’s founder and chief executive. “The world is talking about the size of Big Data sources, but at the end of the day it will be about the ease of consumption.” Indeed, the McKinsey Global Institute has projected that the U.S. needs up to 190,000 people with “analytical expertise” than it has.
The idea of combining proprietary corporate data with public data is hardly new; fast food companies have long looked at census data to figure out where to put outlets, and what to feed people there. The public data sources for ClearStory, will largely be the new category of “Data Marts,” like Datasift, Factual, the Windows Azure Data Marketplace from Microsoft, and Infochimps. These data marts collectively have perhaps millions of data types, from Twitter feeds to mean temperatures and store locations.
“There aren’t enough experts at companies to handle all these data sources,” Ms. Shahani-Mulligan says. “Brick and mortar stores in the Midwest can’t get the kind of data scientists LinkedIn has. We can give them that.” The company offers a range of algorithms to wrangle the data. Longer term, ClearStory hopes to get companies to start sharing more of their own internal data too, what Ms. Shahani-Mulligan calls “the gold locked up in 30 years of relational databases.”
Drawing off all of the data sources could be powerful, if ClearStory can manage to maintain an even level of quality and reliability among its data sources. That already bedevils many data marts. If ClearStory offers sloppy data to customers, however, it will hurt ClearStory.
Another challenge will be designing an intuitive Web-based user experience, so the average marketer or communications professional isn’t flummoxed by statistics. Good “U.E.” people are among the toughest people to find in tech.
ClearStory is a good trend barometer in other ways. The product, which is expected to be ready for public consumption late this summer, will be offered free to casual users. The idea is that enough customers far away from the corporate information technology department will come to depend on ClearStory, and eventually I.T. will want to incorporate a more sophisticated paid version. It is a plan that worked for Yammer, a corporate messaging tool, and Dropbox, with offers storage in the cloud, as well as the Linux operating system.
Ms. Shahani-Mulligan, a marketing veteran of Netscape, Opsware, and Cloudera, is backed by Google Ventures, Andreessen Horowitz, Khosla Ventures, among others. That makes it one of the more “hot backer”-compliant ventures: Google Ventures is specializing in companies with a statistical bent, Andreessen Horowitz, is assembling an ecology of companies in cloud computing, and Vinod Khosla is a renowned tech visionary.
Even with backers like that, ClearStory is making itself visible a lot earlier than most startups risking it will hit the radar of potential competitors even before it has a product. In another sign of the times, and the boom (or, possibly, the bubble) in Big Data, ClearStory needs the publicity in order to hire in an increasingly competitive talent market, however.
“There is a war for talent,” says Ben Horowitz, ClearStory’s backer at Andreessen Horowitz. “Getting the first five people is easy, but the next 20 is hard, when you are selling on your vision and momentum.” Besides, he says, “big companies don’t react to potential threats. They only react to real threats. There are just too many threats out there.”
Big Data: The NSA Is Building the Country's Biggest Spy Center (Watch What You Say)
James Bamford, Wired, March 15, 2012
The spring air in the small, sand-dusted town has a soft haze to it, and clumps of green-gray sagebrush rustle in the breeze. Bluffdale sits in a bowl-shaped valley in the shadow of Utah’s Wasatch Range to the east and the Oquirrh Mountains to the west. It’s the heart of Mormon country, where religious pioneers first arrived more than 160 years ago.
But new pioneers have quietly begun moving into the area, secretive outsiders who say little and keep to themselves. Like the pious polygamists, they are focused on deciphering cryptic messages that only they have the power to understand. Just off Beef Hollow Road, less than a mile from brethren headquarters, thousands of hard-hatted construction workers in sweat-soaked T-shirts are laying the groundwork for the newcomers’ own temple and archive, a massive complex so large that it necessitated expanding the town’s boundaries.
Once built, it will be more than five times the size of the US Capitol.
Rather than Bibles, prophets, and worshippers, this temple will be filled with servers, computer intelligence experts, and armed guards. And instead of listening for words flowing down from heaven, these newcomers will be secretly capturing, storing, and analyzing vast quantities of words and images hurtling through the world’s telecommunications networks. In the little town of Bluffdale, Big Love and Big Brother have become uneasy neighbors.
The NSA has become the largest, most covert, and potentially most intrusive intelligence agency ever.
Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade. Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.”
It is, in some measure, the realization of the “total information awareness” program created during the first term of the Bush administration—an effort that was killed by Congress in 2003 after it caused an outcry over its potential for invading Americans’ privacy.
But “this is more than just a data center,” says one senior intelligence official who until recently was involved with the program. The mammoth Bluffdale center will have another important and far more secret role that until now has gone unrevealed. It is also critical, he says, for breaking codes. And code-breaking is crucial, because much of the data that the center will handle—financial information, stock transactions, business deals, foreign military and diplomatic secrets, legal documents, confidential personal communications—will be heavily encrypted. According to another top official also involved with the program, the NSA made an enormous breakthrough several years ago in its ability to cryptanalyze, or break, unfathomably complex encryption systems employed by not only governments around the world but also many average computer users in the US. The upshot, according to this official: “Everybody’s a target; everybody with communication is a target.”
For the NSA, overflowing with tens of billions of dollars in post-9/11 budget awards, the cryptanalysis breakthrough came at a time of explosive growth, in size as well as in power. Established as an arm of the Department of Defense following Pearl Harbor, with the primary purpose of preventing another surprise assault, the NSA suffered a series of humiliations in the post-Cold War years. Caught offguard by an escalating series of terrorist attacks—the first World Trade Center bombing, the blowing up of US embassies in East Africa, the attack on the USS Cole in Yemen, and finally the devastation of 9/11—some began questioning the agency’s very reason for being. In response, the NSA has quietly been reborn. And while there is little indication that its actual effectiveness has improved—after all, despite numerous pieces of evidence and intelligence-gathering opportunities, it missed the near-disastrous attempted attacks by the underwear bomber on a flight to Detroit in 2009 and by the car bomber in Times Square in 2010—there is no doubt that it has transformed itself into the largest, most covert, and potentially most intrusive intelligence agency ever created.
In the process—and for the first time since Watergate and the other scandals of the Nixon administration—the NSA has turned its surveillance apparatus on the US and its citizens. It has established listening posts throughout the nation to collect and sift through billions of email messages and phone calls, whether they originate within the country or overseas. It has created a supercomputer of almost unimaginable speed to look for patterns and unscramble codes. Finally, the agency has begun building a place to store all the trillions of words and thoughts and whispers captured in its electronic net. And, of course, it’s all being done in secret. To those on the inside, the old adage that NSA stands for Never Say Anything applies more than ever.
UTAH DATA CENTER
1 Visitor control center
3 Data halls
Four 25,000-square-foot facilities house rows and rows of servers.
4 Backup generators and fuel tanks
Can power the center for at least three days.
5 Water storage and pumping
Able to pump 1.7 million gallons of liquid per day.
6 Chiller plant
About 60,000 tons of cooling equipment to keep servers from overheating.
7 Power substation
An electrical substation to meet the center’s estimated 65-megawatt demand.
8 Security
Video surveillance, intrusion detection, and other protection will cost more than $10 million.
A short time later, Inglis arrived in Bluffdale at the site of the future data center, a flat, unpaved runway on a little-used part of Camp Williams, a National Guard training site. There, in a white tent set up for the occasion, Inglis joined Harvey Davis, the agency’s associate director for installations and logistics, and Utah senator Orrin Hatch, along with a few generals and politicians in a surreal ceremony. Standing in an odd wooden sandbox and holding gold-painted shovels, they made awkward jabs at the sand and thus officially broke ground on what the local media had simply dubbed “the spy center.” Hoping for some details on what was about to be built, reporters turned to one of the invited guests, Lane Beattie of the Salt Lake Chamber of Commerce. Did he have any idea of the purpose behind the new facility in his backyard? “Absolutely not,” he said with a self-conscious half laugh. “Nor do I want them spying on me.”
For his part, Inglis simply engaged in a bit of double-talk, emphasizing the least threatening aspect of the center: “It’s a state-of-the-art facility designed to support the intelligence community in its mission to, in turn, enable and protect the nation’s cybersecurity.” While cybersecurity will certainly be among the areas focused on in Bluffdale, what is collected, how it’s collected, and what is done with the material are far more important issues. Battling hackers makes for a nice cover—it’s easy to explain, and who could be against it? Then the reporters turned to Hatch, who proudly described the center as “a great tribute to Utah,” then added, “I can’t tell you a lot about what they’re going to be doing, because it’s highly classified.”
And then there was this anomaly: Although this was supposedly the official ground-breaking for the nation’s largest and most expensive cybersecurity project, no one from the Department of Homeland Security, the agency responsible for protecting civilian networks from cyberattack, spoke from the lectern. In fact, the official who’d originally introduced the data center, at a press conference in Salt Lake City in October 2009, had nothing to do with cybersecurity. It was Glenn A. Gaffney, deputy director of national intelligence for collection, a man who had spent almost his entire career at the CIA. As head of collection for the intelligence community, he managed the country’s human and electronic spies.
Within days, the tent and sandbox and gold shovels would be gone and Inglis and the generals would be replaced by some 10,000 construction workers. “We’ve been asked not to talk about the project,” Rob Moore, president of Big-D Construction, one of the three major contractors working on the project, told a local reporter. The plans for the center show an extensive security system: an elaborate $10 million antiterrorism protection program, including a fence designed to stop a 15,000-pound vehicle traveling 50 miles per hour, closed-circuit cameras, a biometric identification system, a vehicle inspection facility, and a visitor-control center.
Inside, the facility will consist of four 25,000-square-foot halls filled with servers, complete with raised floor space for cables and storage. In addition, there will be more than 900,000 square feet for technical support and administration. The entire site will be self-sustaining, with fuel tanks large enough to power the backup generators for three days in an emergency, water storage with the capability of pumping 1.7 million gallons of liquid per day, as well as a sewage system and massive air-conditioning system to keep all those servers cool. Electricity will come from the center’s own substation built by Rocky Mountain Power to satisfy the 65-megawatt power demand. Such a mammoth amount of energy comes with a mammoth price tag—about $40 million a year, according to one estimate.
Given the facility’s scale and the fact that a terabyte of data can now be stored on a flash drive the size of a man’s pinky, the potential amount of information that could be housed in Bluffdale is truly staggering. But so is the exponential growth in the amount of intelligence data being produced every day by the eavesdropping sensors of the NSA and other intelligence agencies. As a result of this “expanding array of theater airborne and other sensor networks,” as a 2007 Department of Defense report puts it, the Pentagon is attempting to expand its worldwide communications network, known as the Global Information Grid, to handle yottabytes (1024 bytes) of data. (A yottabyte is a septillion bytes—so large that no one has yet coined a term for the next higher magnitude.)
It needs that capacity because, according to a recent report by Cisco, global Internet traffic will quadruple from 2010 to 2015, reaching 966 exabytes per year. (A million exabytes equal a yottabyte.) In terms of scale, Eric Schmidt, Google’s former CEO, once estimated that the total of all human knowledge created from the dawn of man to 2003 totaled 5 exabytes. And the data flow shows no sign of slowing. In 2011 more than 2 billion of the world’s 6.9 billion people were connected to the Internet. By 2015, market research firm IDC estimates, there will be 2.7 billion users. Thus, the NSA’s need for a 1-million-square-foot data storehouse. Should the agency ever fill the Utah center with a yottabyte of information, it would be equal to about 500 quintillion (500,000,000,000,000,000,000) pages of text.
The data stored in Bluffdale will naturally go far beyond the world’s billions of public web pages. The NSA is more interested in the so-called invisible web, also known as the deep web or deepnet—data beyond the reach of the public. This includes password-protected data, US and foreign government communications, and noncommercial file-sharing between trusted peers. “The deep web contains government reports, databases, and other sources of information of high value to DOD and the intelligence community,” according to a 2010 Defense Science Board report. “Alternative tools are needed to find and index data in the deep web … Stealing the classified secrets of a potential adversary is where the [intelligence] community is most comfortable.” With its new Utah Data Center, the NSA will at last have the technical capability to store, and rummage through, all those stolen secrets. The question, of course, is how the agency defines who is, and who is not, “a potential adversary.”
The NSA’S SPY NETWORK
Focuses on intercepts from Latin America and, since 9/11, the Middle East and Europe. Some 2,000 workers staff the operation. The NSA recently completed a $100 million renovation on a mega-data center here—a backup storage facility for the Utah Data Center.
5 NSA Hawaii, Oahu
Focuses on intercepts from Asia. Built to house an aircraft assembly plant during World War II, the 250,000-square-foot bunker is nicknamed the Hole. Like the other NSA operations centers, it has since been expanded: Its 2,700 employees now do their work aboveground from a new 234,000-square-foot facility.
6 Domestic listening posts
The NSA has long been free to eavesdrop on international satellite communications. But after 9/11, it installed taps in US telecom “switches,” gaining access to domestic traffic. An ex-NSA official says there are 10 to 20 such installations.
7 Overseas listening posts
According to a knowledgeable intelligence source, the NSA has installed taps on at least a dozen of the major overseas communications links, each capable of eavesdropping on information passing by at a high data rate.
8 Utah Data Center, Bluffdale, Utah
At a million square feet, this $2 billion digital storage facility outside Salt Lake City will be the centerpiece of the NSA’s cloud-based data strategy and essential in its plans for decrypting previously uncrackable documents.
9 Multiprogram Research Facility, Oak Ridge, Tennessee
Some 300 scientists and computer engineers with top security clearance toil away here, building the world’s fastest supercomputers and working on cryptanalytic applications and other secret projects.
Before yottabytes of data from the deep web and elsewhere can begin piling up inside the servers of the NSA’s new center, they must be collected. To better accomplish that, the agency has undergone the largest building boom in its history, including installing secret electronic monitoring rooms in major US telecom facilities. Controlled by the NSA, these highly secured spaces are where the agency taps into the US communications networks, a practice that came to light during the Bush years but was never acknowledged by the agency. The broad outlines of the so-called warrantless-wiretapping program have long been exposed—how the NSA secretly and illegally bypassed the Foreign Intelligence Surveillance Court, which was supposed to oversee and authorize highly targeted domestic eavesdropping; how the program allowed wholesale monitoring of millions of American phone calls and email. In the wake of the program’s exposure, Congress passed the FISA Amendments Act of 2008, which largely made the practices legal. Telecoms that had agreed to participate in the illegal activity were granted immunity from prosecution and lawsuits. What wasn’t revealed until now, however, was the enormity of this ongoing domestic spying program.
For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail. William Binney was a senior NSA crypto-mathematician largely responsible for automating the agency’s worldwide eavesdropping network. A tall man with strands of black hair across the front of his scalp and dark, determined eyes behind thick-rimmed glasses, the 68-year-old spent nearly four decades breaking codes and finding new ways to channel billions of private phone calls and email messages from around the world into the NSA’s bulging databases. As chief and one of the two cofounders of the agency’s Signals Intelligence Automation Research Center, Binney and his team designed much of the infrastructure that’s still likely used to intercept international and foreign communications.
He explains that the agency could have installed its tapping gear at the nation’s cable landing stations—the more than two dozen sites on the periphery of the US where fiber-optic cables come ashore. If it had taken that route, the NSA would have been able to limit its eavesdropping to just international communications, which at the time was all that was allowed under US law. Instead it chose to put the wiretapping rooms at key junction points throughout the country—large, windowless buildings known as switches—thus gaining access to not just international communications but also to most of the domestic traffic flowing through the US. The network of intercept stations goes far beyond the single room in an AT&T building in San Francisco exposed by a whistle-blower in 2006. “I think there’s 10 to 20 of them,” Binney says. “That’s not just San Francisco; they have them in the middle of the country and also on the East Coast.”
The eavesdropping on Americans doesn’t stop at the telecom switches. To capture satellite communications in and out of the US, the agency also monitors AT&T’s powerful earth stations, satellite receivers in locations that include Roaring Creek and Salt Creek. Tucked away on a back road in rural Catawissa, Pennsylvania, Roaring Creek’s three 105-foot dishes handle much of the country’s communications to and from Europe and the Middle East. And on an isolated stretch of land in remote Arbuckle, California, three similar dishes at the company’s Salt Creek station service the Pacific Rim and Asia.
The former NSA official held his thumb and forefinger close together: “We are that far from a turnkey totalitarian state.”
Binney left the NSA in late 2001, shortly after the agency launched its warrantless-wiretapping program. “They violated the Constitution setting it up,” he says bluntly. “But they didn’t care. They were going to do it anyway, and they were going to crucify anyone who stood in the way. When they started violating the Constitution, I couldn’t stay.” Binney says Stellar Wind was far larger than has been publicly disclosed and included not just eavesdropping on domestic phone calls but the inspection of domestic email. At the outset the program recorded 320 million calls a day, he says, which represented about 73 to 80 percent of the total volume of the agency’s worldwide intercepts. The haul only grew from there. According to Binney—who has maintained close contact with agency employees until a few years ago—the taps in the secret rooms dotting the country are actually powered by highly sophisticated software programs that conduct “deep packet inspection,” examining Internet traffic as it passes through the 10-gigabit-per-second cables at the speed of light.
The software, created by a company called Narus that’s now part of Boeing, is controlled remotely from NSA headquarters at Fort Meade in Maryland and searches US sources for target addresses, locations, countries, and phone numbers, as well as watch-listed names, keywords, and phrases in email. Any communication that arouses suspicion, especially those to or from the million or so people on agency watch lists, are automatically copied or recorded and then transmitted to the NSA.
The scope of surveillance expands from there, Binney says. Once a name is entered into the Narus database, all phone calls and other communications to and from that person are automatically routed to the NSA’s recorders. “Anybody you want, route to a recorder,” Binney says. “If your number’s in there? Routed and gets recorded.” He adds, “The Narus device allows you to take it all.” And when Bluffdale is completed, whatever is collected will be routed there for storage and analysis.
According to Binney, one of the deepest secrets of the Stellar Wind program—again, never confirmed until now—was that the NSA gained warrantless access to AT&T’s vast trove of domestic and international billing records, detailed information about who called whom in the US and around the world. As of 2007, AT&T had more than 2.8 trillion records housed in a database at its Florham Park, New Jersey, complex.
Verizon was also part of the program, Binney says, and that greatly expanded the volume of calls subject to the agency’s domestic eavesdropping. “That multiplies the call rate by at least a factor of five,” he says. “So you’re over a billion and a half calls a day.” (Spokespeople for Verizon and AT&T said their companies would not comment on matters of national security.)
After he left the NSA, Binney suggested a system for monitoring people’s communications according to how closely they are connected to an initial target. The further away from the target—say you’re just an acquaintance of a friend of the target—the less the surveillance. But the agency rejected the idea, and, given the massive new storage facility in Utah, Binney suspects that it now simply collects everything. “The whole idea was, how do you manage 20 terabytes of intercept a minute?” he says. “The way we proposed was to distinguish between things you want and things you don’t want.” Instead, he adds, “they’re storing everything they gather.” And the agency is gathering as much as it can.
Once the communications are intercepted and stored, the data-mining begins. “You can watch everybody all the time with data- mining,” Binney says. Everything a person does becomes charted on a graph, “financial transactions or travel or anything,” he says. Thus, as data like bookstore receipts, bank statements, and commuter toll records flow in, the NSA is able to paint a more and more detailed picture of someone’s life.
The NSA also has the ability to eavesdrop on phone calls directly and in real time. According to Adrienne J. Kinne, who worked both before and after 9/11 as a voice interceptor at the NSA facility in Georgia, in the wake of the World Trade Center attacks “basically all rules were thrown out the window, and they would use any excuse to justify a waiver to spy on Americans.” Even journalists calling home from overseas were included. “A lot of time you could tell they were calling their families,” she says, “incredibly intimate, personal conversations.” Kinne found the act of eavesdropping on innocent fellow citizens personally distressing. “It’s almost like going through and finding somebody’s diary,” she says.
In secret listening rooms nationwide, NSA software examines every email, phone call, and tweet as they zip by.
But there is, of course, reason for anyone to be distressed about the practice. Once the door is open for the government to spy on US citizens, there are often great temptations to abuse that power for political purposes, as when Richard Nixon eavesdropped on his political enemies during Watergate and ordered the NSA to spy on antiwar protesters. Those and other abuses prompted Congress to enact prohibitions in the mid-1970s against domestic spying.
Before he gave up and left the NSA, Binney tried to persuade officials to create a more targeted system that could be authorized by a court. At the time, the agency had 72 hours to obtain a legal warrant, and Binney devised a method to computerize the system. “I had proposed that we automate the process of requesting a warrant and automate approval so we could manage a couple of million intercepts a day, rather than subvert the whole process.” But such a system would have required close coordination with the courts, and NSA officials weren’t interested in that, Binney says. Instead they continued to haul in data on a grand scale. Asked how many communications—”transactions,” in NSA’s lingo—the agency has intercepted since 9/11, Binney estimates the number at “between 15 and 20 trillion, the aggregate over 11 years.”
When Barack Obama took office, Binney hoped the new administration might be open to reforming the program to address his constitutional concerns. He and another former senior NSA analyst, J. Kirk Wiebe, tried to bring the idea of an automated warrant-approval system to the attention of the Department of Justice’s inspector general. They were given the brush-off. “They said, oh, OK, we can’t comment,” Binney says.
Sitting in a restaurant not far from NSA headquarters, the place where he spent nearly 40 years of his life, Binney held his thumb and forefinger close together. “We are, like, that far from a turnkey totalitarian state,” he says.
There is still one technology preventing untrammeled government access to private digital data: strong encryption. Anyone—from terrorists and weapons dealers to corporations, financial institutions, and ordinary email senders—can use it to seal their messages, plans, photos, and documents in hardened data shells. For years, one of the hardest shells has been the Advanced Encryption Standard, one of several algorithms used by much of the world to encrypt data. Available in three different strengths—128 bits, 192 bits, and 256 bits—it’s incorporated in most commercial email programs and web browsers and is considered so strong that the NSA has even approved its use for top-secret US government communications. Most experts say that a so-called brute-force computer attack on the algorithm—trying one combination after another to unlock the encryption—would likely take longer than the age of the universe. For a 128-bit cipher, the number of trial-and-error attempts would be 340 undecillion (1036).
Breaking into those complex mathematical shells like the AES is one of the key reasons for the construction going on in Bluffdale. That kind of cryptanalysis requires two major ingredients: super-fast computers to conduct brute-force attacks on encrypted messages and a massive number of those messages for the computers to analyze. The more messages from a given target, the more likely it is for the computers to detect telltale patterns, and Bluffdale will be able to hold a great many messages. “We questioned it one time,” says another source, a senior intelligence manager who was also involved with the planning. “Why were we building this NSA facility? And, boy, they rolled out all the old guys—the crypto guys.” According to the official, these experts told then-director of national intelligence Dennis Blair, “You’ve got to build this thing because we just don’t have the capability of doing the code-breaking.” It was a candid admission. In the long war between the code breakers and the code makers—the tens of thousands of cryptographers in the worldwide computer security industry—the code breakers were admitting defeat.
So the agency had one major ingredient—a massive data storage facility—under way. Meanwhile, across the country in Tennessee, the government was working in utmost secrecy on the other vital element: the most powerful computer the world has ever known.
The plan was launched in 2004 as a modern-day Manhattan Project. Dubbed the High Productivity Computing Systems program, its goal was to advance computer speed a thousandfold, creating a machine that could execute a quadrillion (1015) operations a second, known as a petaflop—the computer equivalent of breaking the land speed record. And as with the Manhattan Project, the venue chosen for the supercomputing program was the town of Oak Ridge in eastern Tennessee, a rural area where sharp ridges give way to low, scattered hills, and the southwestward-flowing Clinch River bends sharply to the southeast. About 25 miles from Knoxville, it is the “secret city” where uranium- 235 was extracted for the first atomic bomb. A sign near the exit read: what you see here, what you do here, what you hear here, when you leave here, let it stay here. Today, not far from where that sign stood, Oak Ridge is home to the Department of Energy’s Oak Ridge National Laboratory, and it’s engaged in a new secret war. But this time, instead of a bomb of almost unimaginable power, the weapon is a computer of almost unimaginable speed.
In 2004, as part of the supercomputing program, the Department of Energy established its Oak Ridge Leadership Computing Facility for multiple agencies to join forces on the project. But in reality there would be two tracks, one unclassified, in which all of the scientific work would be public, and another top-secret, in which the NSA could pursue its own computer covertly. “For our purposes, they had to create a separate facility,” says a former senior NSA computer expert who worked on the project and is still associated with the agency. (He is one of three sources who described the program.) It was an expensive undertaking, but one the NSA was desperate to launch.
Known as the Multiprogram Research Facility, or Building 5300, the $41 million, five-story, 214,000-square-foot structure was built on a plot of land on the lab’s East Campus and completed in 2006. Behind the brick walls and green-tinted windows, 318 scientists, computer engineers, and other staff work in secret on the cryptanalytic applications of high-speed computing and other classified projects. The supercomputer center was named in honor of George R. Cotter, the NSA’s now-retired chief scientist and head of its information technology program. Not that you’d know it. “There’s no sign on the door,” says the ex-NSA computer expert.
At the DOE’s unclassified center at Oak Ridge, work progressed at a furious pace, although it was a one-way street when it came to cooperation with the closemouthed people in Building 5300. Nevertheless, the unclassified team had its Cray XT4 supercomputer upgraded to a warehouse-sized XT5. Named Jaguar for its speed, it clocked in at 1.75 petaflops, officially becoming the world’s fastest computer in 2009.
Meanwhile, over in Building 5300, the NSA succeeded in building an even faster supercomputer. “They made a big breakthrough,” says another former senior intelligence official, who helped oversee the program. The NSA’s machine was likely similar to the unclassified Jaguar, but it was much faster out of the gate, modified specifically for cryptanalysis and targeted against one or more specific algorithms, like the AES. In other words, they were moving from the research and development phase to actually attacking extremely difficult encryption systems. The code-breaking effort was up and running.
The breakthrough was enormous, says the former official, and soon afterward the agency pulled the shade down tight on the project, even within the intelligence community and Congress. “Only the chairman and vice chairman and the two staff directors of each intelligence committee were told about it,” he says. The reason? “They were thinking that this computing breakthrough was going to give them the ability to crack current public encryption.”
In addition to giving the NSA access to a tremendous amount of Americans’ personal data, such an advance would also open a window on a trove of foreign secrets. While today most sensitive communications use the strongest encryption, much of the older data stored by the NSA, including a great deal of what will be transferred to Bluffdale once the center is complete, is encrypted with more vulnerable ciphers. “Remember,” says the former intelligence official, “a lot of foreign government stuff we’ve never been able to break is 128 or less. Break all that and you’ll find out a lot more of what you didn’t know—stuff we’ve already stored—so there’s an enormous amount of information still in there.”
The NSA believes it’s on the verge of breaking a key encryption algorithm—opening up hoards of data.
That, he notes, is where the value of Bluffdale, and its mountains of long-stored data, will come in. What can’t be broken today may be broken tomorrow. “Then you can see what they were saying in the past,” he says. “By extrapolating the way they did business, it gives us an indication of how they may do things now.” The danger, the former official says, is that it’s not only foreign government information that is locked in weaker algorithms, it’s also a great deal of personal domestic communications, such as Americans’ email intercepted by the NSA in the past decade.
But first the supercomputer must break the encryption, and to do that, speed is everything. The faster the computer, the faster it can break codes. The Data Encryption Standard, the 56-bit predecessor to the AES, debuted in 1976 and lasted about 25 years. The AES made its first appearance in 2001 and is expected to remain strong and durable for at least a decade. But if the NSA has secretly built a computer that is considerably faster than machines in the unclassified arena, then the agency has a chance of breaking the AES in a much shorter time. And with Bluffdale in operation, the NSA will have the luxury of storing an ever-expanding archive of intercepts until that breakthrough comes along.
But despite its progress, the agency has not finished building at Oak Ridge, nor is it satisfied with breaking the petaflop barrier. Its next goal is to reach exaflop speed, one quintillion (1018) operations a second, and eventually zettaflop (1021) and yottaflop.
These goals have considerable support in Congress. Last November a bipartisan group of 24 senators sent a letter to President Obama urging him to approve continued funding through 2013 for the Department of Energy’s exascale computing initiative (the NSA’s budget requests are classified). They cited the necessity to keep up with and surpass China and Japan. “The race is on to develop exascale computing capabilities,” the senators noted. The reason was clear: By late 2011 the Jaguar (now with a peak speed of 2.33 petaflops) ranked third behind Japan’s “K Computer,” with an impressive 10.51 petaflops, and the Chinese Tianhe-1A system, with 2.57 petaflops.
But the real competition will take place in the classified realm. To secretly develop the new exaflop (or higher) machine by 2018, the NSA has proposed constructing two connecting buildings, totaling 260,000 square feet, near its current facility on the East Campus of Oak Ridge. Called the Multiprogram Computational Data Center, the buildings will be low and wide like giant warehouses, a design necessary for the dozens of computer cabinets that will compose an exaflop-scale machine, possibly arranged in a cluster to minimize the distance between circuits. According to a presentation delivered to DOE employees in 2009, it will be an “unassuming facility with limited view from roads,” in keeping with the NSA’s desire for secrecy. And it will have an extraordinary appetite for electricity, eventually using about 200 megawatts, enough to power 200,000 homes. The computer will also produce a gargantuan amount of heat, requiring 60,000 tons of cooling equipment, the same amount that was needed to serve both of the World Trade Center towers.
In the meantime Cray is working on the next step for the NSA, funded in part by a $250 million contract with the Defense Advanced Research Projects Agency. It’s a massively parallel supercomputer called Cascade, a prototype of which is due at the end of 2012. Its development will run largely in parallel with the unclassified effort for the DOE and other partner agencies. That project, due in 2013, will upgrade the Jaguar XT5 into an XK6, codenamed Titan, upping its speed to 10 to 20 petaflops.
Yottabytes and exaflops, septillions and undecillions—the race for computing speed and data storage goes on. In his 1941 story “The Library of Babel,” Jorge Luis Borges imagined a collection of information where the entire world’s knowledge is stored but barely a single word is understood. In Bluffdale the NSA is constructing a library on a scale that even Borges might not have contemplated. And to hear the masters of the agency tell it, it’s only a matter of time until every word is illuminated.
James Bamford (washwriter@gmail.com) is the author of The Shadow Factory: The Ultra-Secret NSA from 9/11 to the Eavesdropping on America.
The spring air in the small, sand-dusted town has a soft haze to it, and clumps of green-gray sagebrush rustle in the breeze. Bluffdale sits in a bowl-shaped valley in the shadow of Utah’s Wasatch Range to the east and the Oquirrh Mountains to the west. It’s the heart of Mormon country, where religious pioneers first arrived more than 160 years ago.
They came to escape the rest of the world, to understand the mysterious words sent down from their god as revealed on buried golden plates, and to practice what has become known as “the principle,” marriage to multiple wives.
Today Bluffdale is home to one of the nation’s largest sects of polygamists, the Apostolic United Brethren, with upwards of 9,000 members. The brethren’s complex includes a chapel, a school, a sports field, and an archive. Membership has doubled since 1978—and the number of plural marriages has tripled—so the sect has recently been looking for ways to purchase more land and expand throughout the town.
But new pioneers have quietly begun moving into the area, secretive outsiders who say little and keep to themselves. Like the pious polygamists, they are focused on deciphering cryptic messages that only they have the power to understand. Just off Beef Hollow Road, less than a mile from brethren headquarters, thousands of hard-hatted construction workers in sweat-soaked T-shirts are laying the groundwork for the newcomers’ own temple and archive, a massive complex so large that it necessitated expanding the town’s boundaries.
Once built, it will be more than five times the size of the US Capitol.
Rather than Bibles, prophets, and worshippers, this temple will be filled with servers, computer intelligence experts, and armed guards. And instead of listening for words flowing down from heaven, these newcomers will be secretly capturing, storing, and analyzing vast quantities of words and images hurtling through the world’s telecommunications networks. In the little town of Bluffdale, Big Love and Big Brother have become uneasy neighbors.
The NSA has become the largest, most covert, and potentially most intrusive intelligence agency ever.
Under construction by contractors with top-secret clearances, the blandly named Utah Data Center is being built for the National Security Agency. A project of immense secrecy, it is the final piece in a complex puzzle assembled over the past decade. Its purpose: to intercept, decipher, analyze, and store vast swaths of the world’s communications as they zap down from satellites and zip through the underground and undersea cables of international, foreign, and domestic networks. The heavily fortified $2 billion center should be up and running in September 2013. Flowing through its servers and routers and stored in near-bottomless databases will be all forms of communication, including the complete contents of private emails, cell phone calls, and Google searches, as well as all sorts of personal data trails—parking receipts, travel itineraries, bookstore purchases, and other digital “pocket litter.”
It is, in some measure, the realization of the “total information awareness” program created during the first term of the Bush administration—an effort that was killed by Congress in 2003 after it caused an outcry over its potential for invading Americans’ privacy.
But “this is more than just a data center,” says one senior intelligence official who until recently was involved with the program. The mammoth Bluffdale center will have another important and far more secret role that until now has gone unrevealed. It is also critical, he says, for breaking codes. And code-breaking is crucial, because much of the data that the center will handle—financial information, stock transactions, business deals, foreign military and diplomatic secrets, legal documents, confidential personal communications—will be heavily encrypted. According to another top official also involved with the program, the NSA made an enormous breakthrough several years ago in its ability to cryptanalyze, or break, unfathomably complex encryption systems employed by not only governments around the world but also many average computer users in the US. The upshot, according to this official: “Everybody’s a target; everybody with communication is a target.”
For the NSA, overflowing with tens of billions of dollars in post-9/11 budget awards, the cryptanalysis breakthrough came at a time of explosive growth, in size as well as in power. Established as an arm of the Department of Defense following Pearl Harbor, with the primary purpose of preventing another surprise assault, the NSA suffered a series of humiliations in the post-Cold War years. Caught offguard by an escalating series of terrorist attacks—the first World Trade Center bombing, the blowing up of US embassies in East Africa, the attack on the USS Cole in Yemen, and finally the devastation of 9/11—some began questioning the agency’s very reason for being. In response, the NSA has quietly been reborn. And while there is little indication that its actual effectiveness has improved—after all, despite numerous pieces of evidence and intelligence-gathering opportunities, it missed the near-disastrous attempted attacks by the underwear bomber on a flight to Detroit in 2009 and by the car bomber in Times Square in 2010—there is no doubt that it has transformed itself into the largest, most covert, and potentially most intrusive intelligence agency ever created.
In the process—and for the first time since Watergate and the other scandals of the Nixon administration—the NSA has turned its surveillance apparatus on the US and its citizens. It has established listening posts throughout the nation to collect and sift through billions of email messages and phone calls, whether they originate within the country or overseas. It has created a supercomputer of almost unimaginable speed to look for patterns and unscramble codes. Finally, the agency has begun building a place to store all the trillions of words and thoughts and whispers captured in its electronic net. And, of course, it’s all being done in secret. To those on the inside, the old adage that NSA stands for Never Say Anything applies more than ever.
UTAH DATA CENTER
When construction is completed in 2013, the heavily fortified $2 billion facility in Bluffdale will encompass 1 million square feet.
1 Visitor control center
A $9.7 million facility for ensuring that only cleared personnel gain access.
2 Administration
Designated space for technical support and administrative personnel.3 Data halls
Four 25,000-square-foot facilities house rows and rows of servers.
4 Backup generators and fuel tanks
Can power the center for at least three days.
5 Water storage and pumping
Able to pump 1.7 million gallons of liquid per day.
6 Chiller plant
About 60,000 tons of cooling equipment to keep servers from overheating.
7 Power substation
An electrical substation to meet the center’s estimated 65-megawatt demand.
8 Security
Video surveillance, intrusion detection, and other protection will cost more than $10 million.
Source: U.S. Army Corps of Engineers Conceptual Site plan
A swath of freezing fog blanketed Salt Lake City on the morning of January 6, 2011, mixing with a weeklong coating of heavy gray smog. Red air alerts, warning people to stay indoors unless absolutely necessary, had become almost daily occurrences, and the temperature was in the bone-chilling twenties. “What I smell and taste is like coal smoke,” complained one local blogger that day. At the city’s international airport, many inbound flights were delayed or diverted while outbound regional jets were grounded. But among those making it through the icy mist was a figure whose gray suit and tie made him almost disappear into the background. He was tall and thin, with the physique of an aging basketball player and dark caterpillar eyebrows beneath a shock of matching hair. Accompanied by a retinue of bodyguards, the man was NSA deputy director Chris Inglis, the agency’s highest-ranking civilian and the person who ran its worldwide day-to-day operations.
A short time later, Inglis arrived in Bluffdale at the site of the future data center, a flat, unpaved runway on a little-used part of Camp Williams, a National Guard training site. There, in a white tent set up for the occasion, Inglis joined Harvey Davis, the agency’s associate director for installations and logistics, and Utah senator Orrin Hatch, along with a few generals and politicians in a surreal ceremony. Standing in an odd wooden sandbox and holding gold-painted shovels, they made awkward jabs at the sand and thus officially broke ground on what the local media had simply dubbed “the spy center.” Hoping for some details on what was about to be built, reporters turned to one of the invited guests, Lane Beattie of the Salt Lake Chamber of Commerce. Did he have any idea of the purpose behind the new facility in his backyard? “Absolutely not,” he said with a self-conscious half laugh. “Nor do I want them spying on me.”
For his part, Inglis simply engaged in a bit of double-talk, emphasizing the least threatening aspect of the center: “It’s a state-of-the-art facility designed to support the intelligence community in its mission to, in turn, enable and protect the nation’s cybersecurity.” While cybersecurity will certainly be among the areas focused on in Bluffdale, what is collected, how it’s collected, and what is done with the material are far more important issues. Battling hackers makes for a nice cover—it’s easy to explain, and who could be against it? Then the reporters turned to Hatch, who proudly described the center as “a great tribute to Utah,” then added, “I can’t tell you a lot about what they’re going to be doing, because it’s highly classified.”
And then there was this anomaly: Although this was supposedly the official ground-breaking for the nation’s largest and most expensive cybersecurity project, no one from the Department of Homeland Security, the agency responsible for protecting civilian networks from cyberattack, spoke from the lectern. In fact, the official who’d originally introduced the data center, at a press conference in Salt Lake City in October 2009, had nothing to do with cybersecurity. It was Glenn A. Gaffney, deputy director of national intelligence for collection, a man who had spent almost his entire career at the CIA. As head of collection for the intelligence community, he managed the country’s human and electronic spies.
Within days, the tent and sandbox and gold shovels would be gone and Inglis and the generals would be replaced by some 10,000 construction workers. “We’ve been asked not to talk about the project,” Rob Moore, president of Big-D Construction, one of the three major contractors working on the project, told a local reporter. The plans for the center show an extensive security system: an elaborate $10 million antiterrorism protection program, including a fence designed to stop a 15,000-pound vehicle traveling 50 miles per hour, closed-circuit cameras, a biometric identification system, a vehicle inspection facility, and a visitor-control center.
Inside, the facility will consist of four 25,000-square-foot halls filled with servers, complete with raised floor space for cables and storage. In addition, there will be more than 900,000 square feet for technical support and administration. The entire site will be self-sustaining, with fuel tanks large enough to power the backup generators for three days in an emergency, water storage with the capability of pumping 1.7 million gallons of liquid per day, as well as a sewage system and massive air-conditioning system to keep all those servers cool. Electricity will come from the center’s own substation built by Rocky Mountain Power to satisfy the 65-megawatt power demand. Such a mammoth amount of energy comes with a mammoth price tag—about $40 million a year, according to one estimate.
Given the facility’s scale and the fact that a terabyte of data can now be stored on a flash drive the size of a man’s pinky, the potential amount of information that could be housed in Bluffdale is truly staggering. But so is the exponential growth in the amount of intelligence data being produced every day by the eavesdropping sensors of the NSA and other intelligence agencies. As a result of this “expanding array of theater airborne and other sensor networks,” as a 2007 Department of Defense report puts it, the Pentagon is attempting to expand its worldwide communications network, known as the Global Information Grid, to handle yottabytes (1024 bytes) of data. (A yottabyte is a septillion bytes—so large that no one has yet coined a term for the next higher magnitude.)
It needs that capacity because, according to a recent report by Cisco, global Internet traffic will quadruple from 2010 to 2015, reaching 966 exabytes per year. (A million exabytes equal a yottabyte.) In terms of scale, Eric Schmidt, Google’s former CEO, once estimated that the total of all human knowledge created from the dawn of man to 2003 totaled 5 exabytes. And the data flow shows no sign of slowing. In 2011 more than 2 billion of the world’s 6.9 billion people were connected to the Internet. By 2015, market research firm IDC estimates, there will be 2.7 billion users. Thus, the NSA’s need for a 1-million-square-foot data storehouse. Should the agency ever fill the Utah center with a yottabyte of information, it would be equal to about 500 quintillion (500,000,000,000,000,000,000) pages of text.
The data stored in Bluffdale will naturally go far beyond the world’s billions of public web pages. The NSA is more interested in the so-called invisible web, also known as the deep web or deepnet—data beyond the reach of the public. This includes password-protected data, US and foreign government communications, and noncommercial file-sharing between trusted peers. “The deep web contains government reports, databases, and other sources of information of high value to DOD and the intelligence community,” according to a 2010 Defense Science Board report. “Alternative tools are needed to find and index data in the deep web … Stealing the classified secrets of a potential adversary is where the [intelligence] community is most comfortable.” With its new Utah Data Center, the NSA will at last have the technical capability to store, and rummage through, all those stolen secrets. The question, of course, is how the agency defines who is, and who is not, “a potential adversary.”
The NSA’S SPY NETWORK
Once it’s operational, the Utah Data Center will become, in effect, the NSA’s cloud. The center will be fed data collected by the agency’s eavesdropping satellites, overseas listening posts, and secret monitoring rooms in telecom facilities throughout the US. All that data will then be accessible to the NSA’s code breakers, data-miners, China analysts, counterterrorism specialists, and others working at its Fort Meade headquarters and around the world. Here’s how the data center appears to fit into the NSA’s global puzzle.—J.B.
1 Geostationary satellites
Four satellites positioned around the globe monitor frequencies carrying everything from walkie-talkies and cell phones in Libya to radar systems in North Korea. Onboard software acts as the first filter in the collection process, targeting only key regions, countries, cities, and phone numbers or email.
2 Aerospace Data Facility, Buckley Air Force Base, Colorado
Intelligence collected from the geostationary satellites, as well as signals from other spacecraft and overseas listening posts, is relayed to this facility outside Denver. About 850 NSA employees track the satellites, transmit target information, and download the intelligence haul.
3 NSA Georgia, Fort Gordon, Augusta, GeorgiaFocuses on intercepts from Europe, the Middle East, and North Africa. Codenamed Sweet Tea, the facility has been massively expanded and now consists of a 604,000-square-foot operations building for up to 4,000 intercept operators, analysts, and other specialists.
4 NSA Texas, Lackland Air Force Base, San AntonioFocuses on intercepts from Latin America and, since 9/11, the Middle East and Europe. Some 2,000 workers staff the operation. The NSA recently completed a $100 million renovation on a mega-data center here—a backup storage facility for the Utah Data Center.
5 NSA Hawaii, Oahu
Focuses on intercepts from Asia. Built to house an aircraft assembly plant during World War II, the 250,000-square-foot bunker is nicknamed the Hole. Like the other NSA operations centers, it has since been expanded: Its 2,700 employees now do their work aboveground from a new 234,000-square-foot facility.
6 Domestic listening posts
The NSA has long been free to eavesdrop on international satellite communications. But after 9/11, it installed taps in US telecom “switches,” gaining access to domestic traffic. An ex-NSA official says there are 10 to 20 such installations.
7 Overseas listening posts
According to a knowledgeable intelligence source, the NSA has installed taps on at least a dozen of the major overseas communications links, each capable of eavesdropping on information passing by at a high data rate.
8 Utah Data Center, Bluffdale, Utah
At a million square feet, this $2 billion digital storage facility outside Salt Lake City will be the centerpiece of the NSA’s cloud-based data strategy and essential in its plans for decrypting previously uncrackable documents.
9 Multiprogram Research Facility, Oak Ridge, Tennessee
Some 300 scientists and computer engineers with top security clearance toil away here, building the world’s fastest supercomputers and working on cryptanalytic applications and other secret projects.
Analysts here will access material stored at Bluffdale to prepare reports and recommendations that are sent to policymakers. To handle the increased data load, the NSA is also building an $896 million supercomputer center here.
Before yottabytes of data from the deep web and elsewhere can begin piling up inside the servers of the NSA’s new center, they must be collected. To better accomplish that, the agency has undergone the largest building boom in its history, including installing secret electronic monitoring rooms in major US telecom facilities. Controlled by the NSA, these highly secured spaces are where the agency taps into the US communications networks, a practice that came to light during the Bush years but was never acknowledged by the agency. The broad outlines of the so-called warrantless-wiretapping program have long been exposed—how the NSA secretly and illegally bypassed the Foreign Intelligence Surveillance Court, which was supposed to oversee and authorize highly targeted domestic eavesdropping; how the program allowed wholesale monitoring of millions of American phone calls and email. In the wake of the program’s exposure, Congress passed the FISA Amendments Act of 2008, which largely made the practices legal. Telecoms that had agreed to participate in the illegal activity were granted immunity from prosecution and lawsuits. What wasn’t revealed until now, however, was the enormity of this ongoing domestic spying program.
For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail. William Binney was a senior NSA crypto-mathematician largely responsible for automating the agency’s worldwide eavesdropping network. A tall man with strands of black hair across the front of his scalp and dark, determined eyes behind thick-rimmed glasses, the 68-year-old spent nearly four decades breaking codes and finding new ways to channel billions of private phone calls and email messages from around the world into the NSA’s bulging databases. As chief and one of the two cofounders of the agency’s Signals Intelligence Automation Research Center, Binney and his team designed much of the infrastructure that’s still likely used to intercept international and foreign communications.
He explains that the agency could have installed its tapping gear at the nation’s cable landing stations—the more than two dozen sites on the periphery of the US where fiber-optic cables come ashore. If it had taken that route, the NSA would have been able to limit its eavesdropping to just international communications, which at the time was all that was allowed under US law. Instead it chose to put the wiretapping rooms at key junction points throughout the country—large, windowless buildings known as switches—thus gaining access to not just international communications but also to most of the domestic traffic flowing through the US. The network of intercept stations goes far beyond the single room in an AT&T building in San Francisco exposed by a whistle-blower in 2006. “I think there’s 10 to 20 of them,” Binney says. “That’s not just San Francisco; they have them in the middle of the country and also on the East Coast.”
The eavesdropping on Americans doesn’t stop at the telecom switches. To capture satellite communications in and out of the US, the agency also monitors AT&T’s powerful earth stations, satellite receivers in locations that include Roaring Creek and Salt Creek. Tucked away on a back road in rural Catawissa, Pennsylvania, Roaring Creek’s three 105-foot dishes handle much of the country’s communications to and from Europe and the Middle East. And on an isolated stretch of land in remote Arbuckle, California, three similar dishes at the company’s Salt Creek station service the Pacific Rim and Asia.
The former NSA official held his thumb and forefinger close together: “We are that far from a turnkey totalitarian state.”
Binney left the NSA in late 2001, shortly after the agency launched its warrantless-wiretapping program. “They violated the Constitution setting it up,” he says bluntly. “But they didn’t care. They were going to do it anyway, and they were going to crucify anyone who stood in the way. When they started violating the Constitution, I couldn’t stay.” Binney says Stellar Wind was far larger than has been publicly disclosed and included not just eavesdropping on domestic phone calls but the inspection of domestic email. At the outset the program recorded 320 million calls a day, he says, which represented about 73 to 80 percent of the total volume of the agency’s worldwide intercepts. The haul only grew from there. According to Binney—who has maintained close contact with agency employees until a few years ago—the taps in the secret rooms dotting the country are actually powered by highly sophisticated software programs that conduct “deep packet inspection,” examining Internet traffic as it passes through the 10-gigabit-per-second cables at the speed of light.
The software, created by a company called Narus that’s now part of Boeing, is controlled remotely from NSA headquarters at Fort Meade in Maryland and searches US sources for target addresses, locations, countries, and phone numbers, as well as watch-listed names, keywords, and phrases in email. Any communication that arouses suspicion, especially those to or from the million or so people on agency watch lists, are automatically copied or recorded and then transmitted to the NSA.
The scope of surveillance expands from there, Binney says. Once a name is entered into the Narus database, all phone calls and other communications to and from that person are automatically routed to the NSA’s recorders. “Anybody you want, route to a recorder,” Binney says. “If your number’s in there? Routed and gets recorded.” He adds, “The Narus device allows you to take it all.” And when Bluffdale is completed, whatever is collected will be routed there for storage and analysis.
According to Binney, one of the deepest secrets of the Stellar Wind program—again, never confirmed until now—was that the NSA gained warrantless access to AT&T’s vast trove of domestic and international billing records, detailed information about who called whom in the US and around the world. As of 2007, AT&T had more than 2.8 trillion records housed in a database at its Florham Park, New Jersey, complex.
Verizon was also part of the program, Binney says, and that greatly expanded the volume of calls subject to the agency’s domestic eavesdropping. “That multiplies the call rate by at least a factor of five,” he says. “So you’re over a billion and a half calls a day.” (Spokespeople for Verizon and AT&T said their companies would not comment on matters of national security.)
After he left the NSA, Binney suggested a system for monitoring people’s communications according to how closely they are connected to an initial target. The further away from the target—say you’re just an acquaintance of a friend of the target—the less the surveillance. But the agency rejected the idea, and, given the massive new storage facility in Utah, Binney suspects that it now simply collects everything. “The whole idea was, how do you manage 20 terabytes of intercept a minute?” he says. “The way we proposed was to distinguish between things you want and things you don’t want.” Instead, he adds, “they’re storing everything they gather.” And the agency is gathering as much as it can.
Once the communications are intercepted and stored, the data-mining begins. “You can watch everybody all the time with data- mining,” Binney says. Everything a person does becomes charted on a graph, “financial transactions or travel or anything,” he says. Thus, as data like bookstore receipts, bank statements, and commuter toll records flow in, the NSA is able to paint a more and more detailed picture of someone’s life.
The NSA also has the ability to eavesdrop on phone calls directly and in real time. According to Adrienne J. Kinne, who worked both before and after 9/11 as a voice interceptor at the NSA facility in Georgia, in the wake of the World Trade Center attacks “basically all rules were thrown out the window, and they would use any excuse to justify a waiver to spy on Americans.” Even journalists calling home from overseas were included. “A lot of time you could tell they were calling their families,” she says, “incredibly intimate, personal conversations.” Kinne found the act of eavesdropping on innocent fellow citizens personally distressing. “It’s almost like going through and finding somebody’s diary,” she says.
In secret listening rooms nationwide, NSA software examines every email, phone call, and tweet as they zip by.
But there is, of course, reason for anyone to be distressed about the practice. Once the door is open for the government to spy on US citizens, there are often great temptations to abuse that power for political purposes, as when Richard Nixon eavesdropped on his political enemies during Watergate and ordered the NSA to spy on antiwar protesters. Those and other abuses prompted Congress to enact prohibitions in the mid-1970s against domestic spying.
Before he gave up and left the NSA, Binney tried to persuade officials to create a more targeted system that could be authorized by a court. At the time, the agency had 72 hours to obtain a legal warrant, and Binney devised a method to computerize the system. “I had proposed that we automate the process of requesting a warrant and automate approval so we could manage a couple of million intercepts a day, rather than subvert the whole process.” But such a system would have required close coordination with the courts, and NSA officials weren’t interested in that, Binney says. Instead they continued to haul in data on a grand scale. Asked how many communications—”transactions,” in NSA’s lingo—the agency has intercepted since 9/11, Binney estimates the number at “between 15 and 20 trillion, the aggregate over 11 years.”
When Barack Obama took office, Binney hoped the new administration might be open to reforming the program to address his constitutional concerns. He and another former senior NSA analyst, J. Kirk Wiebe, tried to bring the idea of an automated warrant-approval system to the attention of the Department of Justice’s inspector general. They were given the brush-off. “They said, oh, OK, we can’t comment,” Binney says.
Sitting in a restaurant not far from NSA headquarters, the place where he spent nearly 40 years of his life, Binney held his thumb and forefinger close together. “We are, like, that far from a turnkey totalitarian state,” he says.
There is still one technology preventing untrammeled government access to private digital data: strong encryption. Anyone—from terrorists and weapons dealers to corporations, financial institutions, and ordinary email senders—can use it to seal their messages, plans, photos, and documents in hardened data shells. For years, one of the hardest shells has been the Advanced Encryption Standard, one of several algorithms used by much of the world to encrypt data. Available in three different strengths—128 bits, 192 bits, and 256 bits—it’s incorporated in most commercial email programs and web browsers and is considered so strong that the NSA has even approved its use for top-secret US government communications. Most experts say that a so-called brute-force computer attack on the algorithm—trying one combination after another to unlock the encryption—would likely take longer than the age of the universe. For a 128-bit cipher, the number of trial-and-error attempts would be 340 undecillion (1036).
Breaking into those complex mathematical shells like the AES is one of the key reasons for the construction going on in Bluffdale. That kind of cryptanalysis requires two major ingredients: super-fast computers to conduct brute-force attacks on encrypted messages and a massive number of those messages for the computers to analyze. The more messages from a given target, the more likely it is for the computers to detect telltale patterns, and Bluffdale will be able to hold a great many messages. “We questioned it one time,” says another source, a senior intelligence manager who was also involved with the planning. “Why were we building this NSA facility? And, boy, they rolled out all the old guys—the crypto guys.” According to the official, these experts told then-director of national intelligence Dennis Blair, “You’ve got to build this thing because we just don’t have the capability of doing the code-breaking.” It was a candid admission. In the long war between the code breakers and the code makers—the tens of thousands of cryptographers in the worldwide computer security industry—the code breakers were admitting defeat.
So the agency had one major ingredient—a massive data storage facility—under way. Meanwhile, across the country in Tennessee, the government was working in utmost secrecy on the other vital element: the most powerful computer the world has ever known.
The plan was launched in 2004 as a modern-day Manhattan Project. Dubbed the High Productivity Computing Systems program, its goal was to advance computer speed a thousandfold, creating a machine that could execute a quadrillion (1015) operations a second, known as a petaflop—the computer equivalent of breaking the land speed record. And as with the Manhattan Project, the venue chosen for the supercomputing program was the town of Oak Ridge in eastern Tennessee, a rural area where sharp ridges give way to low, scattered hills, and the southwestward-flowing Clinch River bends sharply to the southeast. About 25 miles from Knoxville, it is the “secret city” where uranium- 235 was extracted for the first atomic bomb. A sign near the exit read: what you see here, what you do here, what you hear here, when you leave here, let it stay here. Today, not far from where that sign stood, Oak Ridge is home to the Department of Energy’s Oak Ridge National Laboratory, and it’s engaged in a new secret war. But this time, instead of a bomb of almost unimaginable power, the weapon is a computer of almost unimaginable speed.
In 2004, as part of the supercomputing program, the Department of Energy established its Oak Ridge Leadership Computing Facility for multiple agencies to join forces on the project. But in reality there would be two tracks, one unclassified, in which all of the scientific work would be public, and another top-secret, in which the NSA could pursue its own computer covertly. “For our purposes, they had to create a separate facility,” says a former senior NSA computer expert who worked on the project and is still associated with the agency. (He is one of three sources who described the program.) It was an expensive undertaking, but one the NSA was desperate to launch.
Known as the Multiprogram Research Facility, or Building 5300, the $41 million, five-story, 214,000-square-foot structure was built on a plot of land on the lab’s East Campus and completed in 2006. Behind the brick walls and green-tinted windows, 318 scientists, computer engineers, and other staff work in secret on the cryptanalytic applications of high-speed computing and other classified projects. The supercomputer center was named in honor of George R. Cotter, the NSA’s now-retired chief scientist and head of its information technology program. Not that you’d know it. “There’s no sign on the door,” says the ex-NSA computer expert.
At the DOE’s unclassified center at Oak Ridge, work progressed at a furious pace, although it was a one-way street when it came to cooperation with the closemouthed people in Building 5300. Nevertheless, the unclassified team had its Cray XT4 supercomputer upgraded to a warehouse-sized XT5. Named Jaguar for its speed, it clocked in at 1.75 petaflops, officially becoming the world’s fastest computer in 2009.
Meanwhile, over in Building 5300, the NSA succeeded in building an even faster supercomputer. “They made a big breakthrough,” says another former senior intelligence official, who helped oversee the program. The NSA’s machine was likely similar to the unclassified Jaguar, but it was much faster out of the gate, modified specifically for cryptanalysis and targeted against one or more specific algorithms, like the AES. In other words, they were moving from the research and development phase to actually attacking extremely difficult encryption systems. The code-breaking effort was up and running.
The breakthrough was enormous, says the former official, and soon afterward the agency pulled the shade down tight on the project, even within the intelligence community and Congress. “Only the chairman and vice chairman and the two staff directors of each intelligence committee were told about it,” he says. The reason? “They were thinking that this computing breakthrough was going to give them the ability to crack current public encryption.”
In addition to giving the NSA access to a tremendous amount of Americans’ personal data, such an advance would also open a window on a trove of foreign secrets. While today most sensitive communications use the strongest encryption, much of the older data stored by the NSA, including a great deal of what will be transferred to Bluffdale once the center is complete, is encrypted with more vulnerable ciphers. “Remember,” says the former intelligence official, “a lot of foreign government stuff we’ve never been able to break is 128 or less. Break all that and you’ll find out a lot more of what you didn’t know—stuff we’ve already stored—so there’s an enormous amount of information still in there.”
The NSA believes it’s on the verge of breaking a key encryption algorithm—opening up hoards of data.
That, he notes, is where the value of Bluffdale, and its mountains of long-stored data, will come in. What can’t be broken today may be broken tomorrow. “Then you can see what they were saying in the past,” he says. “By extrapolating the way they did business, it gives us an indication of how they may do things now.” The danger, the former official says, is that it’s not only foreign government information that is locked in weaker algorithms, it’s also a great deal of personal domestic communications, such as Americans’ email intercepted by the NSA in the past decade.
But first the supercomputer must break the encryption, and to do that, speed is everything. The faster the computer, the faster it can break codes. The Data Encryption Standard, the 56-bit predecessor to the AES, debuted in 1976 and lasted about 25 years. The AES made its first appearance in 2001 and is expected to remain strong and durable for at least a decade. But if the NSA has secretly built a computer that is considerably faster than machines in the unclassified arena, then the agency has a chance of breaking the AES in a much shorter time. And with Bluffdale in operation, the NSA will have the luxury of storing an ever-expanding archive of intercepts until that breakthrough comes along.
But despite its progress, the agency has not finished building at Oak Ridge, nor is it satisfied with breaking the petaflop barrier. Its next goal is to reach exaflop speed, one quintillion (1018) operations a second, and eventually zettaflop (1021) and yottaflop.
These goals have considerable support in Congress. Last November a bipartisan group of 24 senators sent a letter to President Obama urging him to approve continued funding through 2013 for the Department of Energy’s exascale computing initiative (the NSA’s budget requests are classified). They cited the necessity to keep up with and surpass China and Japan. “The race is on to develop exascale computing capabilities,” the senators noted. The reason was clear: By late 2011 the Jaguar (now with a peak speed of 2.33 petaflops) ranked third behind Japan’s “K Computer,” with an impressive 10.51 petaflops, and the Chinese Tianhe-1A system, with 2.57 petaflops.
But the real competition will take place in the classified realm. To secretly develop the new exaflop (or higher) machine by 2018, the NSA has proposed constructing two connecting buildings, totaling 260,000 square feet, near its current facility on the East Campus of Oak Ridge. Called the Multiprogram Computational Data Center, the buildings will be low and wide like giant warehouses, a design necessary for the dozens of computer cabinets that will compose an exaflop-scale machine, possibly arranged in a cluster to minimize the distance between circuits. According to a presentation delivered to DOE employees in 2009, it will be an “unassuming facility with limited view from roads,” in keeping with the NSA’s desire for secrecy. And it will have an extraordinary appetite for electricity, eventually using about 200 megawatts, enough to power 200,000 homes. The computer will also produce a gargantuan amount of heat, requiring 60,000 tons of cooling equipment, the same amount that was needed to serve both of the World Trade Center towers.
In the meantime Cray is working on the next step for the NSA, funded in part by a $250 million contract with the Defense Advanced Research Projects Agency. It’s a massively parallel supercomputer called Cascade, a prototype of which is due at the end of 2012. Its development will run largely in parallel with the unclassified effort for the DOE and other partner agencies. That project, due in 2013, will upgrade the Jaguar XT5 into an XK6, codenamed Titan, upping its speed to 10 to 20 petaflops.
Yottabytes and exaflops, septillions and undecillions—the race for computing speed and data storage goes on. In his 1941 story “The Library of Babel,” Jorge Luis Borges imagined a collection of information where the entire world’s knowledge is stored but barely a single word is understood. In Bluffdale the NSA is constructing a library on a scale that even Borges might not have contemplated. And to hear the masters of the agency tell it, it’s only a matter of time until every word is illuminated.
James Bamford (washwriter@gmail.com) is the author of The Shadow Factory: The Ultra-Secret NSA from 9/11 to the Eavesdropping on America.
Subscribe to:
Posts (Atom)