Tuesday, March 22, 2011
Open Networking Foundation Pursues New Standards
MOUNTAIN VIEW, Calif. — Acknowledging that so-called cloud computing will blur the distinctions between computers and networks, about two dozen big information technology companies plan to announce on Tuesday a new standards-setting group for computer networking.
The group, to be called the Open Networking Foundation, hopes to help standardize a set of technologies pioneered at Stanford and the University of California, Berkeley, and meant to make small and large networks programmable in much the same way that individual computers are.
The changes, if widely adopted, would have implications for global telecommunications networks and large corporate data centers, but also for small household networks. The benefits, proponents say, would be more flexible and secure networks that are less likely to suffer from congestion. Someday, they say, networks might even be less expensive to build and operate.
The new approach could allow for setting up on-demand “express lanes” for voice and data traffic that is time-sensitive. Or it might let big telecommunications companies, like Verizon or AT&T, use software to combine several fiber optic backbones temporarily for particularly heavy information loads and then have them automatically separate when a data rush hour is over.
For households, the new capabilities might let Internet service providers offer remote services like home security or energy control.
The foundation’s organizers also say the new technologies will offer ways to improve computer security and could possibly enhance individual privacy within the e-commerce and social networking markets. Those markets are the fastest-growing uses for computing and network resources.
While the new capabilities could be crucial to network engineers, for business users and consumers the changes might be no more noticeable than advances in plumbing, heating and air-conditioning. Everything might work better, but most users would probably not know — or care — why or how.
The members of the Open Networking Foundation will include Broadcom, Brocade, Ciena, Cisco, Citrix, Dell, Deutsche Telekom, Ericsson, Facebook, Force10, Google, Hewlett-Packard, I.B.M., Juniper, Marvell, Microsoft, NEC, Netgear, NTT, Riverbed Technology, Verizon, VMWare and Yahoo.
“This answers a question that the entire industry has had, and that is how do you provide owners and operators of large networks with the flexibility of control that they want in a standardized fashion,” said Nick McKeown, a professor of electrical engineering and computer science at Stanford, where his and colleagues’ work forms part of the technical underpinnings, called OpenFlow.
The effort is a departure from the traditional way the Internet works. As designed by military and academic experts in the 1960s, the Internet has been based on interconnected computers that send and receive packets of data, paying little heed to the content and making few distinctions among the various types of senders and receivers of information.
The intelligence in the original Internet was meant to reside largely at the end points of the network — the computers — while the specialized routing computers were relatively dumb post offices of various size, mainly confined to reading addresses and transferring packets of data to adjacent systems.
But these days, when cloud computing means a lot of the information is stored and processed on computers out on the network, there is growing need for more intelligent control systems to orchestrate the behavior of thousands of routing machines. It will make it possible, for example, for managers of large networks to program their network to prioritize certain types of data, perhaps to ensure quality of service or to add security to certain portions of a network.
The designers argue that because OpenFlow should open up hardware and software systems that control the flow of Internet data packets, systems that have been closed and proprietary, it will cause a new round of innovation focused principally upon the vast computing systems known as cloud computers.”This is a pragmatic solution,” said David Farber, a computer scientist at Carnegie Mellon who was one of the pioneers of data networking technology.
“The idea of moving intelligence to the end points of the network was one of the original design points of the Internet,” Mr. Farber said. But he noted that as the network evolved to offer sophisticated advanced services through centralized cloud computers, including the delivery of digital voice and video, it became less feasible to continue relying on decentralized network design.
Mr. Farber noted that there have been other research projects aimed at redesigning the Internet. For example, the National Science Foundation, in addition to supporting the OpenFlow initiative, has financed the Global Environment for Network Innovations, or GENI. Open Flow appears to have generated broad industry support, he said, but it must still prove itself in the market.
A number of networking companies, including Cisco, Hewlett-Packard and Juniper, have already produced prototype systems that support OpenFlow technology. Morever, at least one Silicon Valley start-up, Nicira Networks, is now testing OpenFlow products that are meant to enter the cloud computing market later this year.
“If you look at really big companies like Google and Amazon, they take really smart programmers and give them a problem like search or automating storage and then turn the crank and out pops a great system,” said Martin Casado, Nicira’s chief technologist and one of the members of the OpenFlow research project at Stanford.
But Mr. Casado noted that in the past the one part of the system that could not be programmed was the network. “This customizes the network to the applications that are actually being run.”
Saturday, March 19, 2011
'We're in a critical period for the internet' . . . Tim Wu
The Wu master
The internet as a model of free speech and access is coming to an end, says web expert Tim Wu
The internet is under threat. At risk is what's known as "net neutrality", or the principle of free access for each user to every online site, regardless of content. That's the view of the man who coined the above term, Tim Wu, whose new book, The Master Switch, was published yesterday. It argues the internet now runs the risk of not just political censorship – as seen in Libya and Egypt, and in the American reaction to WikiLeaks – but that of commercial censorship, too. Monopolies such as Google and Apple may soon decide to choose which parts of the internet to give us – or switch off – and in some cases have already started to do so.
"We are in a critical period for the internet," Tim Wu, the book's author, says. "What the internet is, is in flux." Wu looks, a colleague suggests, like a cleverer version of Keanu Reeves. In reality, he is a senior adviser to the Obama administration on, fittingly, the competition issues that concern internet and mobile industries. A position which, ironically, makes him a distant colleague of the officials waging war against WikiLeaks and Bradley Manning. An academic lawyer by trade – he has taught at Chicago, Columbia and Stanford – Wu has also long been a respected commentator on internet issues, and writes regularly for Slate magazine. The first to coin the term "net neutrality", Wu is sometimes mentioned in the same breath as social media experts Jay Rosen and Jeff Jarvis, and sceptics Evgeny Morozov, Nicholas Carr and Jaron Lanier. But unlike these five, whose work is mainly concerned with a discussion about the (de)merits of online activity, Wu's book perhaps places him in a critically different category. The Master Switch is less concerned with the rights and wrongs of the internet today, and more concerned with its long-term future.
"The internet is about 15 years into its cycle as an open medium," says Wu, "and at that moment in their cycle, most open media tend to turn to closed media." What Wu means is that the internet might be about to go the same way as the information services of the 20th century: the telephone, radio, cinema and television. "Internet is the descendant of these industries," Wu says, "a 15-year-old teenager." And if we want to know what kind of adult this teenager will become, "the clearest way is to look at its parents, and look what happened to them when they reached their 20s".
What happened, he argues, is that they went from being technologies used by lots of different individuals and companies, to ones controlled by just a few monopolies. He uses the example of AT&T, the great American telephone monopolist, "who went to the American people and said, 'We will be good, we will build the best telephone network in the world: give us the monopoly'." He points to the American film industry, and shows how it quickly was transformed from an industry that was relatively easy to enter, to one mainly controlled by a few Hollywood studios.
Wu's fear is that a similar consolidation of power may be about to happen to the internet. "When we talk about the internet," he says, "we're only talking about three or four companies . . . Amazon, Google, Apple, Facebook." These big four are so large that one or more of them could team up with a mobile network and end net neutrality (a term that Wu coined in 2003) by privileging that network's clients above all others. It's something that is already partly happening. Apple's iPhone was at first only available to AT&T customers in the US and O2 clients in the UK. And since Google teamed up with American phone firm Verizon last summer, it now at least has the option to do something similar. Wu won't comment on these deals directly, due to his position with the US government. But, he says, like AT&T in the 1910s, "Google has similarly promised the world: 'We will be a good company.' And we have essentially conferred it a dominance over the market, and over how we find our information, because we believe it will be good."
However, warns Wu, "the question everyone has is whether one day Google will have its Heart of Darkness moment." If it does, says Wu, what's at stake is the principle on which the internet was founded. At its inception, "the internet was typically a place where you could put up content without anybody's permission". But partnerships between the bigger web-related companies might squeeze out the smaller ones. Using the example of the online news industry, Wu suggests that if newspapers were to follow the example of Rupert Murdoch's new iPad-based "paper", The Daily, and "become exclusive partners with Apple, it may be easier for them to make money, but we may also end up with a media on the internet that is significantly more closed than it is now." This is because, he says, "You can imagine a future where blogs don't really have a meaningful future, because the content provided on a platform [such as Apple] doesn't create any room for anyone other than its exclusive media partners." So, Wu concludes: "The internet as a forum for speech, as a place where an individual with a talent can compete with a major newspaper – I'm suggesting that model may be passing."
But though the internet was a freer place in its younger days, I ask Wu, wasn't it only available to a privileged few? Big conglomerates may be growing ever powerful, but haven't they at least brought the internet to a much wider audience than the academics and techies of the early 90s? "It shouldn't be a trade-off," Wu replies. "There is some truth to the idea that companies are interested in consumers, and so they bring [the internet] to a broader marketplace – but it is still important to stand up for the original values of the internet." Wu sees these values "as fundamental to a free society", values that we should preserve even as the internet becomes "a mass consumption product. And I think it's possible. You don't have to throw those values out the window just because millions of people are using it."
Wu recommends protecting these values through the maintenance of something that in his book he calls the "separation principle". Just as journalists maintain "a separation of news and opinion", Wu argues, "the people who move and carry information should stay at some distance from the creators of content, because they have a natural conflict of interest." In other words, though Wu does not name names, companies such as Apple and Google should stay well away from mobile networks such as AT&T and Verizon.
He feels government and consumers have a dual responsibility to police this conflict. Legislation should prevent mergers between the carriers of internet content, and the producers of the content. Consumers should boycott any company that threatens net neutrality. Will it work? Wu is undecided. "My big question is whether, five years from now, the big four companies will be even more consolidated, with other companies mattering less and less – or whether the internet will have proved its truly radical nature, and a whole new cast of characters will have emerged . . . And I don't know the answer."
• The Master Switch: The Rise and Fall of Information Empires by Tim Wu is published by Atlantic Books, £19.99
Tuesday, March 15, 2011
This Data Isn't Dull. It Improves Lives.
GOVERNMENTS have learned a cheap new way to improve people’s lives. Here is the basic recipe:
Take data that you and I have already paid a government agency to collect, and post it online in a way that computer programmers can easily use. Then wait a few months. Voilà! The private sector gets busy, creating Web sites and smartphone apps that reformat the information in ways that are helpful to consumers, workers and companies.
Not surprisingly, San Francisco, with its proximity to Silicon Valley, has been a pioneer in these efforts. For some years, Bay Area transit systems had been tracking the locations of their trains and buses via onboard GPS. Then someone got the bright idea to post that information in real time. Thus the delightful app Routesy was born. Install it on a smartphone and the app can tell you that your bus is stuck in traffic and will be 10 minutes late — or it can help you realize that you are standing on the wrong street, dummy. It gives consumers a great new way to find out when and where the bus is coming, and all at minimal government expense.
Another example involves weather data produced by the National Oceanographic and Atmospheric Administration. The forecasts you find on the Weather Channel, or on the evening news or online, use the agency’s information. Again, the government produces and releases raw data, and the private sector transforms it into something useful for the public.
Several other departments in the Obama administration are looking to expand the use of such techniques. On data.gov, you will find huge amounts of downloadable data that had heretofore been inaccessible. As a sign of the importance that President Obama has attached to this approach, he put it on the government’s agenda on Jan. 21, 2009, his second day in office. (Disclosure: My book, “Nudge,” published in 2008, advocated this broad idea; Cass R. Sunstein, co-author of the book, is now administrator of the White House Office of Information and Regulatory Affairs.)
Now the administration is pushing to use this concept as a tool for regulation, and as a method of avoiding more heavy-handed rule making. The idea is that making things more transparent can immediately turn consumers into better shoppers and make markets work better. One might think that such an initiative would receive nearly universal support — after all, who could be against openness and transparency? But it turns out that some people are.
Two cases are under discussion right now.
First, the Department of Transportation is considering a new rule requiring airlines to make all of their prices public and immediately available online. The postings would include both ticket prices and the fees for “extras” like baggage, movies, food and beverages. The data would then be accessible to travel Web sites, and thus to all shoppers.
The airlines would retain the right to decide how and where to sell their products and services. But many of them are insisting that they should be able to decide where and how to display these extra fees. The issue is likely to grow in importance as airlines expand their lists of possible extras, from seats with more legroom to business-class meals served in coach.
Electronic disclosure of all fees can make it much easier for consumers to figure out what a trip really costs, and thus make markets more efficient, without requiring new rules and regulations. (As someone who once bought two tickets on a discount airline from London to Dublin for the advertised price of £1 each, then ended up paying hundreds of dollars for the privilege of bringing along two heavy suitcases, I acknowledge having a sore spot on this issue.)
Another initiative has been proposed by the Consumer Product Safety Commission. In 2008, Congress overwhelmingly passed and President George W. Bush signed legislation mandating an online database of reported safety issues in products, at saferproducts.gov. The Web site ran for a few months in a “soft launch” and went into full operation on Friday.
But a majority in the House of Representatives passed an amendment last month that might have stopped this initiative in its tracks. The amendment, sponsored by Representative Mike Pompeo, a Kansas Republican, would have prohibited the agency from spending any further money to start the site. One goal, of course, was to cut the budget, although proponents of the amendment also argue that the Web site might include information that is erroneous and damaging to the businesses that sell children’s products.
Yet several provisions in the final rules protect manufacturers from false or malicious statements. Consumers have to include identifying information and sign an affidavit testifying to the truth of their complaints. Furthermore, manufacturers will be able to see complaints before they are posted, and can then correct mistakes or add comments.
ALTHOUGH this amendment was passed in the name of deficit reduction, the requested money for the site is a puny $3 million a year. If we want to reduce the cost of government regulation, this is exactly the kind of effort we should be applauding and expanding.
Compared with the tiny costs, the benefits of this program could be enormous. Thirteen years ago, two of my dear friends experienced the nightmare that parents dread most. They were called at work by their child-care provider and told that their 18-month-old son had died in a crib accident. Imagine their anguish when they later learned that other children had died in this model of crib, and that still others had died in cribs with similar design. Yet there was no easy way for any parent or child-care provider to know that.
In a recent three-year span, some 265 children under the age of 5 died in accidents related to nursery products, the government has reported. If this program could reduce that number even slightly, the cost would seem amply justified.
Moving the government into the 21st century should be applauded. In a future column, I will explain how the release of some kinds of data can even help consumers better understand themselves.
Richard H. Thaler is a professor of economics and behavioral science at the Booth School of Business at the University of Chicago.
Wednesday, March 2, 2011
Halamka: Freeing the Data
Wednesday, March 2, 2011
I'm keynoting this year's Intersystems Global Conference on the topic of "Freeing the Data" from the transactional systems we use today such as Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Electronic Health Records (EHR), etc. As I've prepared my speech, I've given a lot of thought to the evolving data needs we have in our enterprises.
In healthcare and in many other industries, it's increasingly common for users to ask IT for tools and resources to look beyond the data we enter during the course of our daily work. For one patient, I know the diagnosis, but what treatments were given to the last 1000 similar patients. I know the sales today, but how do they vary over the week, the month, and the year? Can I predict future resource needs before they happen?
In the past, such analysis typically relied on structured data, exported from transactional systems into data marts using Extract/Transform/Load (ETL) utilities, followed by analysis with Online Analytical Processing (OLAP) or Business Intelligence (BI) tools.
In a world filled with highly scalable web search engines, increasingly capable natural language processing technologies, and practical examples of artificial intelligence/pattern recognition (think of IBM's Jeopardy-savvy Watson as a sophisticated data mining tool), there are novel approaches to freeing the data that go beyond a single database with pre-defined hypercube rollups. Here are my top 10 trends to watch as we increasingly free data from transactional systems.
1. Both structured and unstructured data will be important
In healthcare, the HITECH Act/Meaningful Use requires that clinicians document the smoking status of 50% of their patients. In the past, many EHRs did not have structured data elements to support this activity. Today's certified EHRs provided structured vocabularies and specific pulldowns/checkboxes for data entry, but what do we do about past data? Ideally, we'd use natural language processing, probability, and search to examine unstructured text in the patient record and figure out smoking status including the context of the word smoking such as "former", "active", "heavy", "never" etc.
Businesses will always have a combination of structured and unstructured data. Finding ways to leverage unstructured data will empower businesses to make the most of their information assets.
2. Inference is possible by parsing natural language
Watson on Jeopardy provided an important illustration of how natural language processing can really work. Watson does not understand the language and it is not conscious/sentient. Watson's programming enables it to assign probabilities to expressions. When asked "does he drink alcohol frequently?", finding the word "alcohol" associated with the word "excess" is more more likely to imply a drinking problem than finding "alcohol" associated with "to clean his skin before injecting his insulin". Next generation Natural Language Processing tools will provide the technology to assign probabilities and infer meaning from context.
3. Data mining needs to go beyond single databases owned by a single organization.
If I want to ask questions about patient treatment and outcomes, I may need to query data from hundreds of hospitals to achieve statistical significance. Each of those hospitals may have different IT systems with different data structures and vocabularies. How can a query a collection of heterogenous databases? Federation will possible by normalizing the queries through middleware. For example, data might be mapped to a common Resource Description Framework (RDF) exchange language using standardized SPARQL query tools. At Harvard, we've created a common web-based interface called SHRINE that queries all our hospital databases, providing aggregate de-identified answers to questions about diagnosis and treatment of millions of patients.
4. Non-obvious associations will be increasingly important
Sometimes, it is not enough to query multiple databases. Data needs to be linked external resources to produce novel information. For example, at Harvard, we've taken the address of each faculty member, examined every publication they have ever written, geo-encoded the location of every co-author, and created visualizations of productivity, impact, and influence based on the proximity of colleagues. We call this "social networking analysis"
5. The President's Council of Advisors on Science and Technology (PCAST) Healthcare IT report will offer several important directional themes to will accelerate "freeing the data".
The PCAST report suggests that we embrace the idea of universal exchange languages, metadata tagging with controlled vocabularies, privacy flagging, and search engine technology with probabilistic matching to transform transactional data sources into information, knowledge and wisdom. For example, imagine if all immunization data were normalized as it left transactional systems and pushed into state registries that were united by a federated search that included privacy protections. Suddenly every doctor could ensure that every person had up to date immunizations at every visit.
6. Ontologies and data models will be important to support analytics
Part of creating middleware solutions that enable federation of data sources requires that we generally know what data is important in healthcare and how data elements relate to each other. For example, it's important to know that an allergy has a substance, a severity, a reaction, an observer, and an onset data. Every EHR may implement allergies differently, but by using common detailed clinical model for data exchange and querying we can map heterogeneous data into comparable data.
7. Mapping free text to controlled vocabularies will be possible and should be done as close to the source of data as possible.
Every industry has its jargon. Most clinicians do not wake up every morning thinking about SNOMED-CT concepts of ICD-10 codes. One way to leverage unstructured data is to turn it into structured data as it is entered. If a clinician types "Allergy to Pencillin", it could become SNOMED-CT concept 294513009 for Pencillins. As more controlled vocabularies are introduced in medicine and other industries, transforming text into controlled concepts for later searching will be increasingly important. Ideally, this will be done as the data is entered, so it can be checked for accuracy. If not at entry, then transformations should be done as close to the source systems as possible to ensure data integrity. With every transformation and exchange of data from the original source, there is increasing risk of loss of meaning and context.
8. Linking identity among heterogenous databases will be required for healthcare reform and novel business applications.
If a patient is seen in multiple locations how can we combine their history together so they get the maximum benefit of alerts, reminders, and decision support? Among the hospitals I oversee, we have persistent linkage of all medical record numbers between hospitals - a master patient index. Surescripts/RxHub does a realtime probabilistic match on name/gender/date of birth for over 150 million people in real time. There are other interesting creative techniques such as those pioneered by Jeff Jonas for creating a unique hash of data for every person, then linking data based on that hash. For example John, Jon, Jonathan, and Johnny are reduced to one common root name John. The combination of John + lastname + date of birth is hashing using SHA-1. In this way, records about the person can be aggregated without ever disclosing who the person really is - it's just that hash that is used to find common records.
9. New tools will empower end users
All users, not just power users, want web-based or simple to use client server tools that allow data queries and visualizations without requiring a lot of expertise. The next generation of SQL Server and PowerPivot offer this kind of query power from the desktop. At BIDMC, we've created web-based parameterized queries in our Meaningful Use tools, we're implementing PowerPivot, and we're creating a powerful hospital-based visual query tool using I2B2 technologies.
10. Novel sources of data will be important
Today, patients and consumers are generating data from apps on smart phones, from wearable devices, and social networking sites. Novel approaches to creating knowledge and wisdom will source data from consumers as well as traditional corporate transactional systems.
Thus, as we all move toward "freeing the data" it will no longer be ufficient to use just structured transaction data entered by experts in a single organization, then mined by professional report writers. The speed of business and the need for enhanced quality and efficiency is pushing us toward near real time business intelligence and visualizations for all users. In a sense this mirrors the development of the web itself, evolving from expert HTML coders, to tools for content management for non-technical designated editors, to social networking where everyone is an author, publisher, and consumer.
"Freeing the data" is going to require new thinking about the way we approach application design and requirements. Just as security needs to be foundational, analytics need to be built in from the beginning.
I look forward to my keynote in a few weeks. Once I've delivered it, I'll post the presentation on my blog.
In healthcare and in many other industries, it's increasingly common for users to ask IT for tools and resources to look beyond the data we enter during the course of our daily work. For one patient, I know the diagnosis, but what treatments were given to the last 1000 similar patients. I know the sales today, but how do they vary over the week, the month, and the year? Can I predict future resource needs before they happen?
In the past, such analysis typically relied on structured data, exported from transactional systems into data marts using Extract/Transform/Load (ETL) utilities, followed by analysis with Online Analytical Processing (OLAP) or Business Intelligence (BI) tools.
In a world filled with highly scalable web search engines, increasingly capable natural language processing technologies, and practical examples of artificial intelligence/pattern recognition (think of IBM's Jeopardy-savvy Watson as a sophisticated data mining tool), there are novel approaches to freeing the data that go beyond a single database with pre-defined hypercube rollups. Here are my top 10 trends to watch as we increasingly free data from transactional systems.
1. Both structured and unstructured data will be important
In healthcare, the HITECH Act/Meaningful Use requires that clinicians document the smoking status of 50% of their patients. In the past, many EHRs did not have structured data elements to support this activity. Today's certified EHRs provided structured vocabularies and specific pulldowns/checkboxes for data entry, but what do we do about past data? Ideally, we'd use natural language processing, probability, and search to examine unstructured text in the patient record and figure out smoking status including the context of the word smoking such as "former", "active", "heavy", "never" etc.
Businesses will always have a combination of structured and unstructured data. Finding ways to leverage unstructured data will empower businesses to make the most of their information assets.
2. Inference is possible by parsing natural language
Watson on Jeopardy provided an important illustration of how natural language processing can really work. Watson does not understand the language and it is not conscious/sentient. Watson's programming enables it to assign probabilities to expressions. When asked "does he drink alcohol frequently?", finding the word "alcohol" associated with the word "excess" is more more likely to imply a drinking problem than finding "alcohol" associated with "to clean his skin before injecting his insulin". Next generation Natural Language Processing tools will provide the technology to assign probabilities and infer meaning from context.
3. Data mining needs to go beyond single databases owned by a single organization.
If I want to ask questions about patient treatment and outcomes, I may need to query data from hundreds of hospitals to achieve statistical significance. Each of those hospitals may have different IT systems with different data structures and vocabularies. How can a query a collection of heterogenous databases? Federation will possible by normalizing the queries through middleware. For example, data might be mapped to a common Resource Description Framework (RDF) exchange language using standardized SPARQL query tools. At Harvard, we've created a common web-based interface called SHRINE that queries all our hospital databases, providing aggregate de-identified answers to questions about diagnosis and treatment of millions of patients.
4. Non-obvious associations will be increasingly important
Sometimes, it is not enough to query multiple databases. Data needs to be linked external resources to produce novel information. For example, at Harvard, we've taken the address of each faculty member, examined every publication they have ever written, geo-encoded the location of every co-author, and created visualizations of productivity, impact, and influence based on the proximity of colleagues. We call this "social networking analysis"
5. The President's Council of Advisors on Science and Technology (PCAST) Healthcare IT report will offer several important directional themes to will accelerate "freeing the data".
The PCAST report suggests that we embrace the idea of universal exchange languages, metadata tagging with controlled vocabularies, privacy flagging, and search engine technology with probabilistic matching to transform transactional data sources into information, knowledge and wisdom. For example, imagine if all immunization data were normalized as it left transactional systems and pushed into state registries that were united by a federated search that included privacy protections. Suddenly every doctor could ensure that every person had up to date immunizations at every visit.
6. Ontologies and data models will be important to support analytics
Part of creating middleware solutions that enable federation of data sources requires that we generally know what data is important in healthcare and how data elements relate to each other. For example, it's important to know that an allergy has a substance, a severity, a reaction, an observer, and an onset data. Every EHR may implement allergies differently, but by using common detailed clinical model for data exchange and querying we can map heterogeneous data into comparable data.
7. Mapping free text to controlled vocabularies will be possible and should be done as close to the source of data as possible.
Every industry has its jargon. Most clinicians do not wake up every morning thinking about SNOMED-CT concepts of ICD-10 codes. One way to leverage unstructured data is to turn it into structured data as it is entered. If a clinician types "Allergy to Pencillin", it could become SNOMED-CT concept 294513009 for Pencillins. As more controlled vocabularies are introduced in medicine and other industries, transforming text into controlled concepts for later searching will be increasingly important. Ideally, this will be done as the data is entered, so it can be checked for accuracy. If not at entry, then transformations should be done as close to the source systems as possible to ensure data integrity. With every transformation and exchange of data from the original source, there is increasing risk of loss of meaning and context.
8. Linking identity among heterogenous databases will be required for healthcare reform and novel business applications.
If a patient is seen in multiple locations how can we combine their history together so they get the maximum benefit of alerts, reminders, and decision support? Among the hospitals I oversee, we have persistent linkage of all medical record numbers between hospitals - a master patient index. Surescripts/RxHub does a realtime probabilistic match on name/gender/date of birth for over 150 million people in real time. There are other interesting creative techniques such as those pioneered by Jeff Jonas for creating a unique hash of data for every person, then linking data based on that hash. For example John, Jon, Jonathan, and Johnny are reduced to one common root name John. The combination of John + lastname + date of birth is hashing using SHA-1. In this way, records about the person can be aggregated without ever disclosing who the person really is - it's just that hash that is used to find common records.
9. New tools will empower end users
All users, not just power users, want web-based or simple to use client server tools that allow data queries and visualizations without requiring a lot of expertise. The next generation of SQL Server and PowerPivot offer this kind of query power from the desktop. At BIDMC, we've created web-based parameterized queries in our Meaningful Use tools, we're implementing PowerPivot, and we're creating a powerful hospital-based visual query tool using I2B2 technologies.
10. Novel sources of data will be important
Today, patients and consumers are generating data from apps on smart phones, from wearable devices, and social networking sites. Novel approaches to creating knowledge and wisdom will source data from consumers as well as traditional corporate transactional systems.
Thus, as we all move toward "freeing the data" it will no longer be ufficient to use just structured transaction data entered by experts in a single organization, then mined by professional report writers. The speed of business and the need for enhanced quality and efficiency is pushing us toward near real time business intelligence and visualizations for all users. In a sense this mirrors the development of the web itself, evolving from expert HTML coders, to tools for content management for non-technical designated editors, to social networking where everyone is an author, publisher, and consumer.
"Freeing the data" is going to require new thinking about the way we approach application design and requirements. Just as security needs to be foundational, analytics need to be built in from the beginning.
I look forward to my keynote in a few weeks. Once I've delivered it, I'll post the presentation on my blog.
Tuesday, March 1, 2011
World Economic Forum Drives Health Data Initiative
The Global Health Data Charter calls for the use of technology to overcome worldwide gaps in health information collection, availability, privacy, and analysis.
By Nicole Lewis, Information Week, Feb 28, 2011
The World Economic Forum has launched the Global Health Data Charter, an initiative to advance global health through the management and collection of data. The charter aims to enable individuals and patients, health professionals, and policymakers to make more informed decisions through secure access to comprehensive health data.
Officials at the World Economic Forum in Geneva said at the charter's1 unveiling last week that accurate health data is not available across health systems operating in developed and developing countries, and that gaps in data can be overcome through the use of technology, which will be a main driver in the collection, analysis, and application of health information.
In an interview with InformationWeek, Olivier Raynaud, head of global health and healthcare industries at the World Economic Forum, said the charter is a foundation document that can be used by national and individual organizations and clinicians. Healthcare stakeholders, such as health research organizations, academia, providers, insurers, and nongovernmental organizations (NGOs), can play a role in and benefit from the capture, storage, sharing, and use of health data.
"Even though it is 2011, most health data is still captured with pen and paper. The cost of digital support for information is getting closer to zero and this immediately makes information easier to store, retrieve, share, and aggregate," Raynaud said. "There are large-scale programs taking place in the most challenging areas (such as the monitoring of pregnancies by BRAC3, a large NGO in Bangladesh, where midwifes were equipped with PDAs), which have shown that it is perfectly feasible and generates immediate and tangible results.
He also said many nations are at a tipping point as they transition from paper-based systems to capturing health data electronically, and the hope is that the charter will help foster and enable a data-based, digital health era that will address global disparities in health.
"Disadvantaged populations will gain more from improved health data management; similar to what has been seen with mobile communication, digital health information has the potential to enable a dramatic change in the pace of progress towards universal coverage and access to health," Raynaud said.
Sunday, February 27, 2011
It Takes a Network
The new frontline of modern warfare.
BY STANLEY A. MCCHRYSTAL | MARCH/APRIL 2011
From the outset of my command in Afghanistan, two or three times each week, accompanied by a few aides and often my Afghan counterparts, I would leave the International Security Assistance Force headquarters in Kabul and travel across Afghanistan -- from critical cities like Kandahar to the most remote outposts in violent border regions. Ideally, we left early, traveling light and small, normally using a combination of helicopters and fixed-wing aircraft, to meet with Afghans and their leaders and to connect with our troops on the ground: Brits and Marines rolling back the enemy in Helmand, Afghan National Army troops training in Mazar-e-Sharif, French Foreign Legionnaires patrolling in Kapisa.
But I was not alone: There were other combatants circling the battlefield. Mirroring our movements, competing with us, were insurgent leaders. Connected to, and often directly dispatched by, the Taliban's leadership in Pakistan, they moved through the same areas of Afghanistan. They made shows of public support for Taliban shadow governors, motivated tattered ranks, recruited new troops, distributed funds, reviewed tactics, and updated strategy. And when the sky above became too thick with our drones, their leaders used cell phones and the Internet to issue orders and rally their fighters. They aimed to keep dispersed insurgent cells motivated, strategically wired, and continually informed, all without a rigid -- or targetable -- chain of command.
While a deeply flawed insurgent force in many ways, the Taliban is a uniquely 21st-century threat. Enjoying the traditional insurgent advantage of living amid a population closely tied to them by history and culture, they also leverage sophisticated technology that connects remote valleys and severe mountains instantaneously -- and allows them to project their message worldwide, unhindered by time or filters. They are both deeply embedded in Afghanistan's complex society and impressively agile. And just like their allies in al Qaeda, this new Taliban is more network than army, more a community of interest than a corporate structure.
For the U.S. military that I spent my life in, this was not an easy insight to come by. It was only over the course of years, and with considerable frustrations, that we came to understand how the emerging networks of Islamist insurgents and terrorists are fundamentally different from any enemy the United States has previously known or faced.
In bitter, bloody fights in both Afghanistan and Iraq, it became clear to me and to many others that to defeat a networked enemy we had to become a network ourselves. We had to figure out a way to retain our traditional capabilities of professionalism, technology, and, when needed, overwhelming force, while achieving levels of knowledge, speed, precision, and unity of effort that only a network could provide. We needed to orchestrate a nuanced, population-centric campaign that comprised the ability to almost instantaneously swing a devastating hammer blow against an infiltrating insurgent force or wield a deft scalpel to capture or kill an enemy leader.
WHEN I FIRST WENT TO IRAQ in October 2003 to command a U.S. Joint Special Operations Task Force (JSOTF) that had been tailored down to a relatively small size in the months following the initial invasion, we found a growing threat from multiple sources -- but particularly from al Qaeda in Iraq (AQI). We began a review of our enemy, and of ourselves. Neither was easy to understand.
Like all too many military forces in history, we initially saw our enemy as we viewed ourselves. In a small base outside Baghdad, we started to diagram AQI on white dry-erase boards. Composed largely of foreign mujahideen and with an overall allegiance to Osama bin Laden but controlled inside Iraq by the Jordanian Abu Musab al-Zarqawi, AQI was responsible for an extremely violent campaign of attacks on coalition forces, the Iraqi government, and Iraqi Shiites. Its stated aim was to splinter the new Iraq and ultimately establish an Islamic caliphate. By habit, we started mapping the organization in a traditional military structure, with tiers and rows. At the top was Zarqawi, below him a cascade of lieutenants and foot soldiers.
But the closer we looked, the more the model didn't hold. Al Qaeda in Iraq's lieutenants did not wait for memos from their superiors, much less orders from bin Laden. Decisions were not centralized, but were made quickly and communicated laterally across the organization. Zarqawi's fighters were adapted to the areas they haunted, like Fallujah and Qaim in Iraq's western Anbar province, and yet through modern technology were closely linked to the rest of the province and country. Money, propaganda, and information flowed at alarming rates, allowing for powerful, nimble coordination. We would watch their tactics change (from rocket attacks to suicide bombings, for example) nearly simultaneously in disparate cities. It was a deadly choreography achieved with a constantly changing, often unrecognizable structure.
Over time, it became increasingly clear -- often from intercepted communications or the accounts of insurgents we had captured -- that our enemy was a constellation of fighters organized not by rank but on the basis of relationships and acquaintances, reputation and fame. Who became radicalized in the prisons of Egypt? Who trained together in the pre-9/11 camps in Afghanistan? Who is married to whose sister? Who is making a name for himself, and in doing so burnishing the al Qaeda brand?
All this allowed for flexibility and an impressive ability to grow and to sustain losses. The enemy does not convene promotion boards; the network is self-forming. We would watch a young Iraqi set up in a neighborhood and rise swiftly in importance: After achieving some tactical success, he would market himself, make connections, gain followers, and suddenly a new node of the network would be created and absorbed. The network's energy grew.
In warfare, you make decisions based on indicators. When facing the enemy, you estimate its tactical strength and intuit its planned strategy. This is much simpler when the enemy is a column advancing toward you in plain sight. Our problem in both the Iraq of 2003 and the Afghanistan of today is that indicators popped up everywhere, unevenly and unexpectedly, and often disappeared as quickly as they emerged, flickering in view for only a moment.
We realized we had to have the rapid ability to detect nuanced changes, whether the emergence of new personalities and alliances or sudden changes in tactics. And we had to process that new information in real time -- so we could act on it. A stream of hot cinders was falling everywhere around us, and we had to see them, catch those we could, and react instantly to those we had missed that were starting to set the ground on fire.
SHORTLY AFTER TAKING COMMAND of the JSOTF, I visited one of our teams in Mosul, the largest city in northern Iraq, which was at that time under the able command of then-Maj. Gen. David Petraeus and the troops of the 101st Airborne Division. Although Mosul was still less violent than some other areas of the country, it was clear that al Qaeda was organizing to aggressively contest control of the city -- and, from there, all of northern Iraq.
Our special operations force there was small: about 15 men, supported by a single intelligence analyst. They were set up in a corner of a larger base, operating quietly from a modest white trailer. Although they coordinated with the military forces and civilian (particularly intelligence) agencies on the base, operational security procedures and cultural habits limited the true synergy of their effort against AQI and the fight for the city that lay outside the base's gates.
Moreover, the few antennas that adorned the trailer's roof were unable to pump enough classified information between them and our task force headquarters (or other teams in Iraq) with any timeliness. It wasn't a marooned outpost, thanks to the remarkable team that manned the effort. But it felt like one.
http://www.foreignpolicy.com/files/fp_uploaded_images/110221_815-mcchrystal-hourglass.jpg
That night, on the plane back to Baghdad, I drew an hourglass on a yellow legal pad. The top half of the hourglass represented the team in Mosul; the other represented our task force HQ. They met at just one narrow point. At the top, our team in Mosul was accumulating knowledge and experience, yet lacked both the bandwidth and intelligence manpower to transmit, receive, or digest enough information either to effectively inform, or benefit from, its more robust task force headquarters. All across the country -- in Tikrit, Ramadi, Fallujah, Diyala -- we were waging similarly compartmentalized campaigns. It made our hard fight excruciatingly difficult, and potentially doomed.
The sketch from that evening -- early in a war against an enemy that would only grow more complex, capable, and vicious -- was the first step in what became one of the central missions in our effort: building the network. What was hazy then soon became our mantra: It takes a network to defeat a network.
But fashioning ourselves to counter our enemy's network was easier said than done, especially because it took time to learn what, exactly, made a network different. As we studied, experimented, and adjusted, it became apparent that an effective network involves much more than relaying data. A true network starts with robust communications connectivity, but also leverages physical and cultural proximity, shared purpose, established decision-making processes, personal relationships, and trust. Ultimately, a network is defined by how well it allows its members to see, decide, and effectively act. But transforming a traditional military structure into a truly flexible, empowered network is a difficult process.
Our first attempt at a network was to physically create one. We convinced the agencies partnered with the JSOTF to join us in a big tent at one of our bases so that we could share and process the intelligence in one location. Operators and analysts from multiple units and agencies sat side by side as we sought to fuse our intelligence and operations efforts -- and our cultures -- into a unified effort. This may seem obvious, but at the time it wasn't. Too often, intelligence would travel up the chain in organizational silos -- and return too slowly for those in the fight to take critical action.
It was clear, though, that in this fusion process we had created only a partial network: Each agency or operation had a representative in the tent, but that was not enough. The network needed to expand to include everyone relevant who was operating within the battlespace. Incomplete or unconnected networks can give the illusion of effectiveness, but are like finely crafted gears whose movement drives no other gears.
This insight allowed us to move closer to building a true network by connecting everyone who had a role -- no matter how small, geographically dispersed, or organizationally diverse they might have been -- in a successful counterterrorism operation. We called it, in our shorthand, F3EA: find, fix, finish, exploit, and analyze. The idea was to combine analysts who found the enemy (through intelligence, surveillance, and reconnaissance); drone operators who fixed the target; combat teams who finished the target by capturing or killing him; specialists who exploited the intelligence the raid yielded, such as cell phones, maps, and detainees; and the intelligence analysts who turned this raw information into usable knowledge. By doing this, we speeded up the cycle for a counterterrorism operation, gleaning valuable insights in hours, not days.
But it took a while to get there. The process started as a linear, relatively inefficient chain. Out of habit (and ignorance), each element gave the next group the minimum amount of information needed for it to be able to complete its task. Lacking sufficient shared purpose or situational awareness, each component contributed far less to the outcome than it could or should have.
This made us, in retrospect, painfully slow and uninformed. The linear process created what we called "blinks" -- time delays and missed junctures where information was lost or slowed when filtered down the line. In the early days of the effort, we had multiple experiences where information we captured could not be exploited, analyzed, or reacted to quickly enough -- giving enemy targets time to flee. A blink often meant a missed opportunity in an unforgiving fight.
The key was to reduce the blinks, and we did so by attempting to create a shared consciousness between each level of the counterterrorism teams.
We started by sharing information: Video streamed by the drones was sent to all the participants -- not just the reconnaissance and surveillance analysts controlling them. When an operation was set in motion, information was continuously communicated to and from the combat team, so that intelligence specialists miles away could alert the team on the ground about what they could expect to find of value at the scene and where it might be. Intelligence recovered on the spot was instantly pushed digitally from the target to analysts who could translate it into actionable data while the operators would still be clearing rooms and returning fire. This knowledge was immediately cycled back through the loop to our intelligence and surveillance forces following the results of the raid in real time.
The intelligence recovered on one target in, say, Mosul, might allow for another target to be found, fixed upon, and finished in Baghdad, or even Afghanistan. Sometimes, finding just one initial target could lead to remarkable results: The network sometimes completed this cycle three times in a single night in locations hundreds of miles apart -- all from the results of the first operation. As our operations in Iraq and Afghanistan intensified, the number of operations conducted each day increased tenfold, and both our precision and success rate also rose dramatically.
Although we got our message out differently than did our enemies, both organizations increasingly shared basic attributes that define an effective network. Decisions were decentralized and cut laterally across the organization. Traditional institutional boundaries fell away and diverse cultures meshed. The network expanded to include more groups, including unconventional actors. It valued competency above all else -- including rank. It sought a clear and evolving definition of the problem and constantly self-analyzed, revisiting its structure, aims, and processes, as well as those of the enemy. Most importantly, the network continually grew the capacity to inform itself.
From its birth in Iraq, both the actual network -- and the hard-earned appreciation for that organizational model -- increasingly expanded to Afghanistan, especially as our nation's focus turned toward that theater. When I became the commander there, we set about building a robust communications architecture and worked to establish relationships with key actors, moving frequently around the country to instill the shared consciousness and purpose necessary for a networked modern army. But that was only the first part of the task.
As we learned to build an effective network, we also learned that leading that network -- a diverse collection of organizations, personalities, and cultures -- is a daunting challenge in itself. That struggle remains a vital, untold chapter of the history of a global conflict that is still under way.
BY STANLEY A. MCCHRYSTAL | MARCH/APRIL 2011
From the outset of my command in Afghanistan, two or three times each week, accompanied by a few aides and often my Afghan counterparts, I would leave the International Security Assistance Force headquarters in Kabul and travel across Afghanistan -- from critical cities like Kandahar to the most remote outposts in violent border regions. Ideally, we left early, traveling light and small, normally using a combination of helicopters and fixed-wing aircraft, to meet with Afghans and their leaders and to connect with our troops on the ground: Brits and Marines rolling back the enemy in Helmand, Afghan National Army troops training in Mazar-e-Sharif, French Foreign Legionnaires patrolling in Kapisa.
But I was not alone: There were other combatants circling the battlefield. Mirroring our movements, competing with us, were insurgent leaders. Connected to, and often directly dispatched by, the Taliban's leadership in Pakistan, they moved through the same areas of Afghanistan. They made shows of public support for Taliban shadow governors, motivated tattered ranks, recruited new troops, distributed funds, reviewed tactics, and updated strategy. And when the sky above became too thick with our drones, their leaders used cell phones and the Internet to issue orders and rally their fighters. They aimed to keep dispersed insurgent cells motivated, strategically wired, and continually informed, all without a rigid -- or targetable -- chain of command.
While a deeply flawed insurgent force in many ways, the Taliban is a uniquely 21st-century threat. Enjoying the traditional insurgent advantage of living amid a population closely tied to them by history and culture, they also leverage sophisticated technology that connects remote valleys and severe mountains instantaneously -- and allows them to project their message worldwide, unhindered by time or filters. They are both deeply embedded in Afghanistan's complex society and impressively agile. And just like their allies in al Qaeda, this new Taliban is more network than army, more a community of interest than a corporate structure.
For the U.S. military that I spent my life in, this was not an easy insight to come by. It was only over the course of years, and with considerable frustrations, that we came to understand how the emerging networks of Islamist insurgents and terrorists are fundamentally different from any enemy the United States has previously known or faced.
In bitter, bloody fights in both Afghanistan and Iraq, it became clear to me and to many others that to defeat a networked enemy we had to become a network ourselves. We had to figure out a way to retain our traditional capabilities of professionalism, technology, and, when needed, overwhelming force, while achieving levels of knowledge, speed, precision, and unity of effort that only a network could provide. We needed to orchestrate a nuanced, population-centric campaign that comprised the ability to almost instantaneously swing a devastating hammer blow against an infiltrating insurgent force or wield a deft scalpel to capture or kill an enemy leader.
WHEN I FIRST WENT TO IRAQ in October 2003 to command a U.S. Joint Special Operations Task Force (JSOTF) that had been tailored down to a relatively small size in the months following the initial invasion, we found a growing threat from multiple sources -- but particularly from al Qaeda in Iraq (AQI). We began a review of our enemy, and of ourselves. Neither was easy to understand.
Like all too many military forces in history, we initially saw our enemy as we viewed ourselves. In a small base outside Baghdad, we started to diagram AQI on white dry-erase boards. Composed largely of foreign mujahideen and with an overall allegiance to Osama bin Laden but controlled inside Iraq by the Jordanian Abu Musab al-Zarqawi, AQI was responsible for an extremely violent campaign of attacks on coalition forces, the Iraqi government, and Iraqi Shiites. Its stated aim was to splinter the new Iraq and ultimately establish an Islamic caliphate. By habit, we started mapping the organization in a traditional military structure, with tiers and rows. At the top was Zarqawi, below him a cascade of lieutenants and foot soldiers.
But the closer we looked, the more the model didn't hold. Al Qaeda in Iraq's lieutenants did not wait for memos from their superiors, much less orders from bin Laden. Decisions were not centralized, but were made quickly and communicated laterally across the organization. Zarqawi's fighters were adapted to the areas they haunted, like Fallujah and Qaim in Iraq's western Anbar province, and yet through modern technology were closely linked to the rest of the province and country. Money, propaganda, and information flowed at alarming rates, allowing for powerful, nimble coordination. We would watch their tactics change (from rocket attacks to suicide bombings, for example) nearly simultaneously in disparate cities. It was a deadly choreography achieved with a constantly changing, often unrecognizable structure.
Over time, it became increasingly clear -- often from intercepted communications or the accounts of insurgents we had captured -- that our enemy was a constellation of fighters organized not by rank but on the basis of relationships and acquaintances, reputation and fame. Who became radicalized in the prisons of Egypt? Who trained together in the pre-9/11 camps in Afghanistan? Who is married to whose sister? Who is making a name for himself, and in doing so burnishing the al Qaeda brand?
All this allowed for flexibility and an impressive ability to grow and to sustain losses. The enemy does not convene promotion boards; the network is self-forming. We would watch a young Iraqi set up in a neighborhood and rise swiftly in importance: After achieving some tactical success, he would market himself, make connections, gain followers, and suddenly a new node of the network would be created and absorbed. The network's energy grew.
In warfare, you make decisions based on indicators. When facing the enemy, you estimate its tactical strength and intuit its planned strategy. This is much simpler when the enemy is a column advancing toward you in plain sight. Our problem in both the Iraq of 2003 and the Afghanistan of today is that indicators popped up everywhere, unevenly and unexpectedly, and often disappeared as quickly as they emerged, flickering in view for only a moment.
We realized we had to have the rapid ability to detect nuanced changes, whether the emergence of new personalities and alliances or sudden changes in tactics. And we had to process that new information in real time -- so we could act on it. A stream of hot cinders was falling everywhere around us, and we had to see them, catch those we could, and react instantly to those we had missed that were starting to set the ground on fire.
SHORTLY AFTER TAKING COMMAND of the JSOTF, I visited one of our teams in Mosul, the largest city in northern Iraq, which was at that time under the able command of then-Maj. Gen. David Petraeus and the troops of the 101st Airborne Division. Although Mosul was still less violent than some other areas of the country, it was clear that al Qaeda was organizing to aggressively contest control of the city -- and, from there, all of northern Iraq.
Our special operations force there was small: about 15 men, supported by a single intelligence analyst. They were set up in a corner of a larger base, operating quietly from a modest white trailer. Although they coordinated with the military forces and civilian (particularly intelligence) agencies on the base, operational security procedures and cultural habits limited the true synergy of their effort against AQI and the fight for the city that lay outside the base's gates.
Moreover, the few antennas that adorned the trailer's roof were unable to pump enough classified information between them and our task force headquarters (or other teams in Iraq) with any timeliness. It wasn't a marooned outpost, thanks to the remarkable team that manned the effort. But it felt like one.
http://www.foreignpolicy.com/files/fp_uploaded_images/110221_815-mcchrystal-hourglass.jpg
That night, on the plane back to Baghdad, I drew an hourglass on a yellow legal pad. The top half of the hourglass represented the team in Mosul; the other represented our task force HQ. They met at just one narrow point. At the top, our team in Mosul was accumulating knowledge and experience, yet lacked both the bandwidth and intelligence manpower to transmit, receive, or digest enough information either to effectively inform, or benefit from, its more robust task force headquarters. All across the country -- in Tikrit, Ramadi, Fallujah, Diyala -- we were waging similarly compartmentalized campaigns. It made our hard fight excruciatingly difficult, and potentially doomed.
The sketch from that evening -- early in a war against an enemy that would only grow more complex, capable, and vicious -- was the first step in what became one of the central missions in our effort: building the network. What was hazy then soon became our mantra: It takes a network to defeat a network.
But fashioning ourselves to counter our enemy's network was easier said than done, especially because it took time to learn what, exactly, made a network different. As we studied, experimented, and adjusted, it became apparent that an effective network involves much more than relaying data. A true network starts with robust communications connectivity, but also leverages physical and cultural proximity, shared purpose, established decision-making processes, personal relationships, and trust. Ultimately, a network is defined by how well it allows its members to see, decide, and effectively act. But transforming a traditional military structure into a truly flexible, empowered network is a difficult process.
Our first attempt at a network was to physically create one. We convinced the agencies partnered with the JSOTF to join us in a big tent at one of our bases so that we could share and process the intelligence in one location. Operators and analysts from multiple units and agencies sat side by side as we sought to fuse our intelligence and operations efforts -- and our cultures -- into a unified effort. This may seem obvious, but at the time it wasn't. Too often, intelligence would travel up the chain in organizational silos -- and return too slowly for those in the fight to take critical action.
It was clear, though, that in this fusion process we had created only a partial network: Each agency or operation had a representative in the tent, but that was not enough. The network needed to expand to include everyone relevant who was operating within the battlespace. Incomplete or unconnected networks can give the illusion of effectiveness, but are like finely crafted gears whose movement drives no other gears.
This insight allowed us to move closer to building a true network by connecting everyone who had a role -- no matter how small, geographically dispersed, or organizationally diverse they might have been -- in a successful counterterrorism operation. We called it, in our shorthand, F3EA: find, fix, finish, exploit, and analyze. The idea was to combine analysts who found the enemy (through intelligence, surveillance, and reconnaissance); drone operators who fixed the target; combat teams who finished the target by capturing or killing him; specialists who exploited the intelligence the raid yielded, such as cell phones, maps, and detainees; and the intelligence analysts who turned this raw information into usable knowledge. By doing this, we speeded up the cycle for a counterterrorism operation, gleaning valuable insights in hours, not days.
But it took a while to get there. The process started as a linear, relatively inefficient chain. Out of habit (and ignorance), each element gave the next group the minimum amount of information needed for it to be able to complete its task. Lacking sufficient shared purpose or situational awareness, each component contributed far less to the outcome than it could or should have.
This made us, in retrospect, painfully slow and uninformed. The linear process created what we called "blinks" -- time delays and missed junctures where information was lost or slowed when filtered down the line. In the early days of the effort, we had multiple experiences where information we captured could not be exploited, analyzed, or reacted to quickly enough -- giving enemy targets time to flee. A blink often meant a missed opportunity in an unforgiving fight.
The key was to reduce the blinks, and we did so by attempting to create a shared consciousness between each level of the counterterrorism teams.
We started by sharing information: Video streamed by the drones was sent to all the participants -- not just the reconnaissance and surveillance analysts controlling them. When an operation was set in motion, information was continuously communicated to and from the combat team, so that intelligence specialists miles away could alert the team on the ground about what they could expect to find of value at the scene and where it might be. Intelligence recovered on the spot was instantly pushed digitally from the target to analysts who could translate it into actionable data while the operators would still be clearing rooms and returning fire. This knowledge was immediately cycled back through the loop to our intelligence and surveillance forces following the results of the raid in real time.
The intelligence recovered on one target in, say, Mosul, might allow for another target to be found, fixed upon, and finished in Baghdad, or even Afghanistan. Sometimes, finding just one initial target could lead to remarkable results: The network sometimes completed this cycle three times in a single night in locations hundreds of miles apart -- all from the results of the first operation. As our operations in Iraq and Afghanistan intensified, the number of operations conducted each day increased tenfold, and both our precision and success rate also rose dramatically.
Although we got our message out differently than did our enemies, both organizations increasingly shared basic attributes that define an effective network. Decisions were decentralized and cut laterally across the organization. Traditional institutional boundaries fell away and diverse cultures meshed. The network expanded to include more groups, including unconventional actors. It valued competency above all else -- including rank. It sought a clear and evolving definition of the problem and constantly self-analyzed, revisiting its structure, aims, and processes, as well as those of the enemy. Most importantly, the network continually grew the capacity to inform itself.
From its birth in Iraq, both the actual network -- and the hard-earned appreciation for that organizational model -- increasingly expanded to Afghanistan, especially as our nation's focus turned toward that theater. When I became the commander there, we set about building a robust communications architecture and worked to establish relationships with key actors, moving frequently around the country to instill the shared consciousness and purpose necessary for a networked modern army. But that was only the first part of the task.
As we learned to build an effective network, we also learned that leading that network -- a diverse collection of organizations, personalities, and cultures -- is a daunting challenge in itself. That struggle remains a vital, untold chapter of the history of a global conflict that is still under way.
Social Networking's Newest Friend: Genomics
By Emily Singer, Technology Review, MIT, Feb 24, 2011
The first large-scale study to combine genome sequencing and social-network analysis solves a mysterious TB outbreak.
It was the baby's case that first caught people's attention: an infant in a medium-sized community in British Columbia that was diagnosed with tuberculosis in July 2006. When public health workers took a deeper look at the community's medical records, they found a number of additional cases suggestive of an outbreak. By December 2008, 41 cases had been identified, bumping up the region's annual incidence rate by a factor of 10.
Officials at the BC Centre for Disease Control (BCCDC) were faced with the question at the heart of any outbreak: what was the source? Had the bacteria that cause TB mutated to become more infectious? Or was there some change in the community that made the microbes more likely to spread?
The answer would be crucial in focusing public health efforts to stop it. Traditional methods for analyzing transmission patterns created only a hazy picture. Molecular analysis of specimens collected from patients suggested everyone was infected with the same strain. "Based on the information we had, we couldn't really figure out who was giving it to whom," says Patrick Tang, a medical microbiologist at the BCCDC.
So Tang and collaborators combined two tools to create a much clearer picture of the outbreak: social-network analysis, which has become increasingly common in tracking infectious disease over the last decade, and whole-genome sequencing—an analysis of the microbe's entire DNA sequence. The latter, which has been applied to outbreaks in only a few cases to date, allows much more precise tracing of infections than traditional molecular techniques, which look at only a few spots in the genome.
"For the first time, we can paint a really detailed picture of the relationships between people in the community and a really detailed picture of the relationships between the bacteria themselves," says Jennifer Gardy, head of the BCCDC's Genome Research Laboratory and lead author on the study. "We can reconstruct the path an organism took throughout a population."
Researchers sequenced the genomes of 36 bacterial samples collected from patients. They used specialized algorithms to compare individual mutations that arose in the microbes' DNA as they spread. The analysis, published today in the New England Journal of Medicine, revealed that there were actually two different lineages of the microbe, pointing to two different outbreaks spreading independently of one another. These findings suggested that an environmental factor lay at the heart of the outbreak, rather than a genetic one.
In addition to genome sequencing, researchers questioned patients about the people they lived with and worked with, as well as where they spent their time, creating a network diagram of potential interactions. "Instead of just getting a list of names, you're getting names, places, and behaviors, and you can paint a much more detailed picture of the underlying community structure," says Gardy. "Key people and places and certain behaviors that might be contributing to an outbreak's spread become much more apparent, and allow you to adjust your outbreak investigation in real-time as this new information becomes available."
The researchers could overlay the genetic data identifying individual mutations with information from the social network that pinpointed when different people might have interacted with each other. "We could identify super-spreaders of the disease," says Tang.
The researchers ultimately concluded that the outbreak was linked to an increase in crack cocaine use in the community. "That was the most likely trigger, reactivating latent disease and facilitating the spread of disease," says Tang. Using this information, public health agencies could focus their resources on the root of the problem and identify those at the highest risk for reactivation of TB, he says.
"The findings show that it is feasible to combine genetic data and social structure to give an idea of the transmission chain and to distinguish two outbreaks going on at the same time," says Joel Miller, a research fellow in the Center for Communicable Disease Dynamics at the Harvard School of Public Health.
Tang and others predict that this approach will become commonplace in the next few years. "With the cost of whole-genome sequencing coming down—it's a few hundred dollars per organism—a lot of people are interested in using it to address questions worldwide," says Tang. He says sequencing will be especially important in more complex cases, such as tracking the spread of antibiotic-resistant organisms around the world.
The main hurdle now is not the cost of sequencing, but rather the analysis tools, adds Tang. "The limitation for most people is how to make sense of the genomic data that is generated," he says.
The first large-scale study to combine genome sequencing and social-network analysis solves a mysterious TB outbreak.
It was the baby's case that first caught people's attention: an infant in a medium-sized community in British Columbia that was diagnosed with tuberculosis in July 2006. When public health workers took a deeper look at the community's medical records, they found a number of additional cases suggestive of an outbreak. By December 2008, 41 cases had been identified, bumping up the region's annual incidence rate by a factor of 10.
Officials at the BC Centre for Disease Control (BCCDC) were faced with the question at the heart of any outbreak: what was the source? Had the bacteria that cause TB mutated to become more infectious? Or was there some change in the community that made the microbes more likely to spread?
The answer would be crucial in focusing public health efforts to stop it. Traditional methods for analyzing transmission patterns created only a hazy picture. Molecular analysis of specimens collected from patients suggested everyone was infected with the same strain. "Based on the information we had, we couldn't really figure out who was giving it to whom," says Patrick Tang, a medical microbiologist at the BCCDC.
So Tang and collaborators combined two tools to create a much clearer picture of the outbreak: social-network analysis, which has become increasingly common in tracking infectious disease over the last decade, and whole-genome sequencing—an analysis of the microbe's entire DNA sequence. The latter, which has been applied to outbreaks in only a few cases to date, allows much more precise tracing of infections than traditional molecular techniques, which look at only a few spots in the genome.
"For the first time, we can paint a really detailed picture of the relationships between people in the community and a really detailed picture of the relationships between the bacteria themselves," says Jennifer Gardy, head of the BCCDC's Genome Research Laboratory and lead author on the study. "We can reconstruct the path an organism took throughout a population."
Researchers sequenced the genomes of 36 bacterial samples collected from patients. They used specialized algorithms to compare individual mutations that arose in the microbes' DNA as they spread. The analysis, published today in the New England Journal of Medicine, revealed that there were actually two different lineages of the microbe, pointing to two different outbreaks spreading independently of one another. These findings suggested that an environmental factor lay at the heart of the outbreak, rather than a genetic one.
In addition to genome sequencing, researchers questioned patients about the people they lived with and worked with, as well as where they spent their time, creating a network diagram of potential interactions. "Instead of just getting a list of names, you're getting names, places, and behaviors, and you can paint a much more detailed picture of the underlying community structure," says Gardy. "Key people and places and certain behaviors that might be contributing to an outbreak's spread become much more apparent, and allow you to adjust your outbreak investigation in real-time as this new information becomes available."
The researchers could overlay the genetic data identifying individual mutations with information from the social network that pinpointed when different people might have interacted with each other. "We could identify super-spreaders of the disease," says Tang.
The researchers ultimately concluded that the outbreak was linked to an increase in crack cocaine use in the community. "That was the most likely trigger, reactivating latent disease and facilitating the spread of disease," says Tang. Using this information, public health agencies could focus their resources on the root of the problem and identify those at the highest risk for reactivation of TB, he says.
"The findings show that it is feasible to combine genetic data and social structure to give an idea of the transmission chain and to distinguish two outbreaks going on at the same time," says Joel Miller, a research fellow in the Center for Communicable Disease Dynamics at the Harvard School of Public Health.
Tang and others predict that this approach will become commonplace in the next few years. "With the cost of whole-genome sequencing coming down—it's a few hundred dollars per organism—a lot of people are interested in using it to address questions worldwide," says Tang. He says sequencing will be especially important in more complex cases, such as tracking the spread of antibiotic-resistant organisms around the world.
The main hurdle now is not the cost of sequencing, but rather the analysis tools, adds Tang. "The limitation for most people is how to make sense of the genomic data that is generated," he says.
Treat the Patient, Not the CT Scan
By ABRAHAM VERGHESE, NYT, FEBRUARY 26, 2011
Palo Alto, Calif.
The other day as I walked through a wing of my hospital, it occurred to me that Watson, I.B.M.’s supercomputer, would be more at home here than he was on “Jeopardy!” Perhaps it’s good, I thought, that his next challenge, with the aid of the Columbia University Medical Center and the University of Maryland School of Medicine, will be to learn to diagnose illnesses and treat patients.
On our rounds of the wards, Watson would see lots of other computers with humans glued to them like piglets at a sow’s teats. We might visit a patient with a complex illness — one whose second liver transplant has failed, who has a fungal meningitis and now also has kidney failure and bleeding and is on a score of medications.
Watson might help me digest the sheer volume of data that is in the electronic medical record and might see trends in the data that speak of an impending disaster. And since Watson is constantly trolling the Web, he would perhaps bring to my attention a case report published the previous night in a Swedish journal describing a new interaction between two of the drugs my patient is taking.
Better still, if Watson could harness data from all the patients in our hospital and in every other hospital in America, we might be alerted to mini-epidemics taking shape. For example, Watson might recognize that the kidney failure in our patient is linked to kidney failure in a patient in Buffalo and another in San Antonio; all three patients, he might inform me, were taking a “natural” weight loss supplement that contained a Chinese herb, aristolochia, that has been associated with more than 100 cases of kidney failure.
In short, Watson would be a potent and clever companion as we made our rounds. But the complaints I hear from patients, family and friends are never about the dearth of technology but about its excesses. My own experience as a patient in an emergency room in another city helped me see this. My nurse would come in periodically to visit the computer work station in my cubicle, her back to me while she clicked and scrolled away. Over her shoulder she said, “On a scale of one to five how is your ...?”
The electronic record of my three-hour stay would have looked perfect, showing close monitoring, even though to me as a patient it lacked a human dimension. I don’t fault the nurse, because in my hospital, despite my best intentions, I too am spending too much time in front of the computer: the story of my patient’s many past admissions, the details of surgeries undergone, every consultant’s opinion, every drug given over every encounter, thousands of blood tests and so many CT scans, M.R.I.’s and ultrasound images reside in there.
This computer record creates what I call an “iPatient” — and this iPatient threatens to become the real focus of our attention, while the real patient in the bed often feels neglected, a mere placeholder for the virtual record.
Imaging the body has become so easy (and profitable, too, if you own the machine). When I was an intern some 30 years ago, about three million CT scans were performed annually in the United States; now the number is more like 80 million. Imaging tests are now responsible for half of the overall radiation Americans are exposed to, compared with about 15 percent in 1980.
With that radiation exposure comes increasing risk for cancer, but what worries me even more is that this ease of ordering a scan has caused doctors’ most basic skills in examining the body to atrophy. This loss is palpable when American medical trainees go to hospitals and clinics abroad with few resources: it can be quite humbling to see doctors in Africa and South America detect fluid around patients’ lungs not with X-rays but by percussing the chest with their fingers and listening with their stethoscopes.
Of course, we still teach medical students how to properly examine the body. In dedicated physical diagnosis courses in their first and second years, students learn on trained actors, who give them appropriate stories and responses, how to do a complete exam of the body’s systems (circulatory, respiratory, musculoskeletal and the rest). Faculty members stand by to assess that the required maneuvers are performed correctly.
But all that training can be undone the moment the students hit their clinical years. Then, they discover that the currency on the ward seems to be “throughput” — getting tests ordered and getting results, having procedures like colonoscopies done expeditiously, calling in specialists, arranging discharge. And the engine for all of that, indeed the place where the dialogue between doctors and nurses takes place, is the computer.
The consequence of losing both faith and skill in examining the body is that we miss simple things, and we order more tests and subject people to the dangers of radiation unnecessarily. Just a few weeks ago, I heard of a patient who arrived in an E.R. in extremis with seizures and breathing difficulties. After being stabilized and put on a breathing machine, she was taken for a CT scan of the chest, to rule out blood clots to the lung; but when the radiologist looked at the results, she turned out to have tumors in both breasts, along with the secondary spread of cancer all over the body.
In retrospect, though, her cancer should have been discovered long before the radiologist found it; before the emergency, the patient had been seen several times and at different places, for symptoms that were probably related to the cancer. I got to see the CT scan: the tumor masses in each breast were likely visible to the naked eye — and certainly to the hand. Yet they had never been noted.
Too frequently, I hear of (and in a study we are conducting, I am collecting) stories like that from all across the country. They represent a type of error that stems from not making use of basic bedside skills, not asking the patient to fully disrobe. It is a more subtle kind of error than operating on the wrong limb; indeed, this sort of mistake is not always recognized, and yet the consequences can be grave.
IN my experience, being skilled at examining the body has a salutary effect beyond finding important clues that lead to an early diagnosis. It is a ritual that remains important to the patient. Recently my ward team admitted an elderly woman who had been transferred from her nursing home in the night because of a change in her mental status. A CT of the head and all other tests were determined to be normal; the problem had been dehydration, and she was better, ready to go back. But as our team was about to enter the room, my intern warned me that the patient’s lawyer daughter was unhappy with the plan to return her mother to the nursing home, and was waiting impatiently to see me and contest the transfer.
After introducing myself to the patient and to her daughter, I did a thorough but quick neurologic exam. I put the patient through her paces: mental status, cranial nerves, motor and sensory function, used my reflex hammer and pointed out interesting things along the way to my interns and students. I then said to the daughter that her mother seemed back to normal. To our surprise, the daughter seemed comforted, and now had no objection to her mother’s return to the nursing home.
Later, our team discussed what had just happened. We all felt that the daughter witnessing the examination of the patient, that ritual, was the key to earning both their trusts. I find that patients from almost any culture have deep expectations of a ritual when a doctor sees them, and they are quick to perceive when he or she gives those procedures short shrift by, say, placing the stethoscope on top of the gown instead of the skin, doing a cursory prod of the belly and wrapping up in 30 seconds. Rituals are about transformation, the crossing of a threshold, and in the case of the bedside exam, the transformation is the cementing of the doctor-patient relationship, a way of saying: “I will see you through this illness. I will be with you through thick and thin.” It is paramount that doctors not forget the importance of this ritual.
An answer that might have been posed on “Jeopardy!” is, “An emergency treatment that is administered by ear.” I wonder if Watson would have known the question (though he will now, cybertroller that he is), which is, “What are words of comfort?”
Abraham Verghese, a professor at the Stanford University School of Medicine, is the author of the novel “Cutting for Stone.”
Palo Alto, Calif.
The other day as I walked through a wing of my hospital, it occurred to me that Watson, I.B.M.’s supercomputer, would be more at home here than he was on “Jeopardy!” Perhaps it’s good, I thought, that his next challenge, with the aid of the Columbia University Medical Center and the University of Maryland School of Medicine, will be to learn to diagnose illnesses and treat patients.
On our rounds of the wards, Watson would see lots of other computers with humans glued to them like piglets at a sow’s teats. We might visit a patient with a complex illness — one whose second liver transplant has failed, who has a fungal meningitis and now also has kidney failure and bleeding and is on a score of medications.
Watson might help me digest the sheer volume of data that is in the electronic medical record and might see trends in the data that speak of an impending disaster. And since Watson is constantly trolling the Web, he would perhaps bring to my attention a case report published the previous night in a Swedish journal describing a new interaction between two of the drugs my patient is taking.
Better still, if Watson could harness data from all the patients in our hospital and in every other hospital in America, we might be alerted to mini-epidemics taking shape. For example, Watson might recognize that the kidney failure in our patient is linked to kidney failure in a patient in Buffalo and another in San Antonio; all three patients, he might inform me, were taking a “natural” weight loss supplement that contained a Chinese herb, aristolochia, that has been associated with more than 100 cases of kidney failure.
In short, Watson would be a potent and clever companion as we made our rounds. But the complaints I hear from patients, family and friends are never about the dearth of technology but about its excesses. My own experience as a patient in an emergency room in another city helped me see this. My nurse would come in periodically to visit the computer work station in my cubicle, her back to me while she clicked and scrolled away. Over her shoulder she said, “On a scale of one to five how is your ...?”
The electronic record of my three-hour stay would have looked perfect, showing close monitoring, even though to me as a patient it lacked a human dimension. I don’t fault the nurse, because in my hospital, despite my best intentions, I too am spending too much time in front of the computer: the story of my patient’s many past admissions, the details of surgeries undergone, every consultant’s opinion, every drug given over every encounter, thousands of blood tests and so many CT scans, M.R.I.’s and ultrasound images reside in there.
This computer record creates what I call an “iPatient” — and this iPatient threatens to become the real focus of our attention, while the real patient in the bed often feels neglected, a mere placeholder for the virtual record.
Imaging the body has become so easy (and profitable, too, if you own the machine). When I was an intern some 30 years ago, about three million CT scans were performed annually in the United States; now the number is more like 80 million. Imaging tests are now responsible for half of the overall radiation Americans are exposed to, compared with about 15 percent in 1980.
With that radiation exposure comes increasing risk for cancer, but what worries me even more is that this ease of ordering a scan has caused doctors’ most basic skills in examining the body to atrophy. This loss is palpable when American medical trainees go to hospitals and clinics abroad with few resources: it can be quite humbling to see doctors in Africa and South America detect fluid around patients’ lungs not with X-rays but by percussing the chest with their fingers and listening with their stethoscopes.
Of course, we still teach medical students how to properly examine the body. In dedicated physical diagnosis courses in their first and second years, students learn on trained actors, who give them appropriate stories and responses, how to do a complete exam of the body’s systems (circulatory, respiratory, musculoskeletal and the rest). Faculty members stand by to assess that the required maneuvers are performed correctly.
But all that training can be undone the moment the students hit their clinical years. Then, they discover that the currency on the ward seems to be “throughput” — getting tests ordered and getting results, having procedures like colonoscopies done expeditiously, calling in specialists, arranging discharge. And the engine for all of that, indeed the place where the dialogue between doctors and nurses takes place, is the computer.
The consequence of losing both faith and skill in examining the body is that we miss simple things, and we order more tests and subject people to the dangers of radiation unnecessarily. Just a few weeks ago, I heard of a patient who arrived in an E.R. in extremis with seizures and breathing difficulties. After being stabilized and put on a breathing machine, she was taken for a CT scan of the chest, to rule out blood clots to the lung; but when the radiologist looked at the results, she turned out to have tumors in both breasts, along with the secondary spread of cancer all over the body.
In retrospect, though, her cancer should have been discovered long before the radiologist found it; before the emergency, the patient had been seen several times and at different places, for symptoms that were probably related to the cancer. I got to see the CT scan: the tumor masses in each breast were likely visible to the naked eye — and certainly to the hand. Yet they had never been noted.
Too frequently, I hear of (and in a study we are conducting, I am collecting) stories like that from all across the country. They represent a type of error that stems from not making use of basic bedside skills, not asking the patient to fully disrobe. It is a more subtle kind of error than operating on the wrong limb; indeed, this sort of mistake is not always recognized, and yet the consequences can be grave.
IN my experience, being skilled at examining the body has a salutary effect beyond finding important clues that lead to an early diagnosis. It is a ritual that remains important to the patient. Recently my ward team admitted an elderly woman who had been transferred from her nursing home in the night because of a change in her mental status. A CT of the head and all other tests were determined to be normal; the problem had been dehydration, and she was better, ready to go back. But as our team was about to enter the room, my intern warned me that the patient’s lawyer daughter was unhappy with the plan to return her mother to the nursing home, and was waiting impatiently to see me and contest the transfer.
After introducing myself to the patient and to her daughter, I did a thorough but quick neurologic exam. I put the patient through her paces: mental status, cranial nerves, motor and sensory function, used my reflex hammer and pointed out interesting things along the way to my interns and students. I then said to the daughter that her mother seemed back to normal. To our surprise, the daughter seemed comforted, and now had no objection to her mother’s return to the nursing home.
Later, our team discussed what had just happened. We all felt that the daughter witnessing the examination of the patient, that ritual, was the key to earning both their trusts. I find that patients from almost any culture have deep expectations of a ritual when a doctor sees them, and they are quick to perceive when he or she gives those procedures short shrift by, say, placing the stethoscope on top of the gown instead of the skin, doing a cursory prod of the belly and wrapping up in 30 seconds. Rituals are about transformation, the crossing of a threshold, and in the case of the bedside exam, the transformation is the cementing of the doctor-patient relationship, a way of saying: “I will see you through this illness. I will be with you through thick and thin.” It is paramount that doctors not forget the importance of this ritual.
An answer that might have been posed on “Jeopardy!” is, “An emergency treatment that is administered by ear.” I wonder if Watson would have known the question (though he will now, cybertroller that he is), which is, “What are words of comfort?”
Abraham Verghese, a professor at the Stanford University School of Medicine, is the author of the novel “Cutting for Stone.”
Friday, February 18, 2011
Emerging Semantic Web Technology Could Help Intelligence Analysts Spot New Terror Threats
In light of 9/11, the attempted Christmas Day bombing in 2009 and even last year’s WikiLeaks incident, it’s clear that the search and information-sharing process across government intelligence databases is flawed and missing an element that would potentially enable analysts to see threats and prevent future incidents.
Semantic technology is used increasingly to help organizations manage, integrate and gain intelligence from multiple streams of unstructured data and information. Semantic is unique in its ability to exceed the limits of other technologies and approach the automatic understanding of a text. While Semantic Web — a web of understood word meanings and connections — technology is quickly eclipsing first-generation, keyword-based index search systems and second-generation social media interaction, the transition is far from complete. Nowhere is this technology more useful than in the national intelligence space.
As a semantic technology professional, I think about how semantic technology could have aided in connecting the dots between the available information in the government intelligence community in the 2009 Christmas bomber case, and most recently, the highly publicized leak of classified government information about the war in Afghanistan.
As a former intelligence analyst, I know the frustration of lacking both complete information and computer systems capable of aiding the analysis process. Almost a decade after 9/11 and untold dollars later, the nation still struggles with effective intelligence sharing. An often mentioned issue is the lack of collaboration among intelligence teams on the analysis of incoming information from the multitude of existing databases.
The Los Angeles Times points to others:
“Lawmakers have been pushing for a capability to search across the government’s vast library of terrorism information, but intelligence officials say there are serious technical and policy hurdles. The databases are written in myriad computer languages; different legal standards are employed on how collected information can be used; and there is reluctance within some agencies to share data.”
The newspaper then makes the connection to 2009’s Christmas bomber threat:
“That makes it harder to connect disparate pieces of threat information, which is exactly what went wrong in the case of Umar Farouk Abdulmutallab, a Nigerian who on Christmas Day tried to blow up an airplane using explosives sewn into his underwear. The bomb failed to detonate, and a passenger jumped on him.”
Analysts must have a reason to collaborate. They must foresee or imagine how one or more evidence streams, often with many missing elements, overlap or fold into one another to form a complete picture. The reality is, even really good human analysts cannot juggle more than 50 to 60 data points — events, names, places, times, dates and the connections between them — at once.
But good technology that mimics the same approach has no such limitation. Allowing such a system to build the larger picture — to connect the dots — through trial and error, quickly and repeatedly with an analyst reviewing that picture for plausibility, internal consistency and impact, would be a more effective approach than adding a small army of new analysts to the problem.
A system that proactively and constantly builds and tests all the available evidence on a person, action, event, etc., is the current architecture of a Semantic Web. This approach is becoming prevalent in the private sector, and governments also are now taking advantage of the Semantic Web rather than a simple web of keywords.
To test this proposition, I used the timeline of known facts about the 2009 Christmas bomber as reported by The New York Times. Although this is a retrospective view, I wanted to know what I would have concluded over time, if I were an analyst and had good information sharing and robust analytical support, such as current Semantic Web technology can provide.
To begin, I took all the known facts and began to process them semantically. I used a semantic search and analysis system to analyze the content for people, places, things, facts, time and geography, but most importantly, for events. Such analysis answers: Who did what to whom when and where? Based on our established event timeline, in summer 2009, Abdulmutallab would have hit intelligence databases when Britain placed him on a watch/no-fly list after his student visa was rejected.
We can see right away that Abdulmutallab was known to have studied in 2004 and 2005 in Sanaa, Yemen; he has a direct connection to the radical Yemen cleric Anwar al-Awlaki; he’s loosely connected to al-Qaida because of his presence in Yemen; and he disappeared in September 2009. But the most important thread is that he was already on Britain’s watch/no entry list.
On the whole, perhaps this picture doesn’t portray a person who has planned a terrorist attack. But more connections come to light when we continue to build the picture of Abdulmutallab into fall 2009.
Through November 2009, several things become apparent. First, the number of connection points has risen significantly between Abdulmutallab, Yemen and al-Qaida relative to the previous summer. Second, the number of evidentiary warning signs around Abdulmutallab has grown to include his father, the United Nations and several U.S. agencies (e.g., the National Security Agency and National Counterterrorism Center). Third, there’s a lack of communication or information sharing among U.S. agencies.
Nonetheless, Abdulmutallab was placed on a terrorist watch list but not on the more restrictive no-fly list. This may not have been the case if analysts had a diagram that visualized the increased strength between him and al-Qaida, as well as the increase in additional connections of concern at this stage of this analysis.
In retrospect, we know that there was still more time. Adding the events from December 2009 in the examination makes the graph richer still:
Once again, as the connection between Abdulmutallab, Yemen and al-Qaida increases, more U.S. agencies take note, and now he has purchased airline tickets with a U.S. destination and didn’t check any baggage (a Transportation Security Administration warning signal since 9/11). As with most intelligence analysis, the strongest indicators come too late, so understanding how to fit them into the overall picture quickly is essential — in this case, the time it took Abdulmutallab to fly from Africa to the Netherlands and then to the United States. Semantic technology that can visualize the new input, can speed up analysts’ understanding.
Semantic Web technology can provide a window into how people, places, things and events come together into threats and opportunities. It’s impossible to expect analysts to manually “see” how anomalous and imperfect evidence streams fit together. And there is always more than one way that they fit together.
Let machines do what they’re good at. Namely when coupled with semantic understanding, measuring endless clues and hints, fitting, testing, removing and adding various puzzle pieces to see if the picture starts to make sense. Past a certain threshold, analysts can take over and do the work computers never will be able to do: apply human judgment and reasoning. Otherwise judgment and decision never arrive, connections are never made, and red flags are never raised.
The timeline of these past and recent events (9/11, the Christmas bomber and recent data leakage in Washington, D.C.) show a serious need to address the gaps in our country’s intelligence procedures and sharing processes. And this is where Semantic Web comes in.
Brooke Aker is CEO of Expert System USA. He writes and speaks on topics, such as competitive intelligence, knowledge management and predictive analytics.
You may use or reference this story with attribution and a link to
http://www.govtech.com/pcio/Semantic-Web-Could-Help-Spot-Terror-Threats-021111.html
Thursday, February 17, 2011
IBM Moving Watson Supercomputer Beyond 'Jeopardy' To Health-Care
By Shara Tibke, Dow Jones Newswires, Feb 16, 2011 10:28pm ET
After trouncing "Jeopardy!'s" best and brightest, International Business Machines Corp.'s (IBM) Watson supercomputer is on to a new challenge--health-care. IBM said it has reached a research agreement with Nuance Communications Inc. (NUAN), a provider of speech-recognition technology, to "explore, develop and commercialize" the Watson computing system's advanced analytics capabilities in the health-care industry. Columbia University Medical Center and the University of Maryland School of Medicine will be providing their medical expertise and research.
Watson, powered by 90 servers and 360 computer chips, was built over the past four years by a team of IBM researchers who set out to develop a machine that could quickly answer complex questions involving puns and wordplay.The room-sized system competed against former "Jeopardy!" champions Ken Jennings and Brad Rutter for three nights this week, finally winning the challenge Wednesday with a score of $77,147. Jennings finished with $24,000 and Rutter had $21,600. The victory nets Watson a total prize of $1 million, which IBM will be donating to charity. Jennings and Rutter get $300,000 and $200,000, respectively, with plans to donate half to charities.
After optimizing Watson for "Jeopardy!" play, IBM researchers are working to apply the system to business uses, such as helping physicians and nurses find answers within huge volumes of information. A doctor considering a patient's diagnosis could use Watson's analytics technology along with Nuance's voice and clinical language understanding offerings to rapidly consider all the related texts, reference materials, prior cases and latest knowledge in medical journals to gain information from more potential sources then previously possible, making the physician more confident in the patient's diagnosis, IBM said.
"Combining our analytics expertise with the experience and technology of Nuance, we can transform the way that health-care professionals accomplish everyday tasks by enabling them to work smarter and more efficiently," Dr. John E. Kelly III, senior vice president and director of IBM Research, said in a press release.
IBM and Nuance expect their first commercial offerings to be available in 18 to 24 months.
Katharine Frase, one of the IBM researchers working on business applications for Watson, said in an interview with Dow Jones Newswires that IBM is considering expanding Watson to other uses such as call centers, knowledge management and training of new employees in technical fields, financial sector applications and law--though she noted it is uncertain if there's a business model to support the law application.
"First we figure out the characteristics of uses that need the technology," Frase said. "Then we figure out a business model, how we deliver the service and who's the customer...Not every business problem needs Watson."
She said IBM has been fielding calls from clients, asking if Watson can help with their problems.IBM expects some customers will want to build versions of Watson behind their own firewalls, while others will want to access its capabilities as a cloud-delivered service. "There's a lot we don't know right now," Frase said. "We'll start with the medical realm and learn as we go along."
-By Shara Tibken, Dow Jones Newswires; 212-416-2189; shara.tibken@dowjones.com
Wednesday, February 16, 2011
Finance: Elusive information (FT today)
BY TOM BRAITHWAITE, FT, February 15 2011 22:25
It was Friday August 15 2008 and a senior official at the US Federal Reserve in Washington wrestled with a thorny problem: he wanted to know what was happening inside Lehman Brothers but was afraid to ask.
Pat Parkinson, now the Fed's top bank supervisor, was trying to find out which companies had derivatives contracts with Lehman as he gauged how severe the impact would be if the investment bank collapsed. But colleagues in New York told him that just requesting the data would be "a huge negative signal" for the bank's prospects and they were "very reluctant" to do anything that might "spook the market".
Lehman's implosion the following month was not the only recent instance where a calamitous lack of decent data has plagued financial markets. Others range from the European bank stress tests carried out last year, which officials admit relied on information "polluted by accounting", to the US stock market "flash crash" on May 6 that left the Securities and Exchange Commission floundering for answers.
But for the first time in decades there is a growing movement to rebuild the creaking data architecture that underpins modern finance.
Regulators' need to understand the crisis provided the impetus. When Lehman fell in September 2008, not only did institutions not know their rivals' exposure to Lehman, or to other problem areas such as subprime mortgages; they were sometimes unable to map their own with any speed. In the vortex, stock prices collapsed, liquidity dried up and investors and bankers ran scared.
"Everyone's going, 'what do I hold that is Lehman?'" says Mike Atkin, head of the Enterprise Data Management Council, a group of banks, information technology companies and regulators. "Wait a minute ... what is Lehman? Lehman isn't one entity – it's 10,000 entities. We don't know what our exposure is because we're not sure what Lehman is."
- Lowdown on the OFR ● The Office of Financial Research, set up by the DoddFrank financial reform act last year to improve the quality and analysis of US data, has wide-ranging powers enabling it to compel institutions to provide information. ● Lewis Alexander, interim head and former senior Citigroup economist, has "a few million" dollars from the Federal Reserve for staffing. By 2012 the OFR will be funded by a tax on big banks. ● Based in the Treasury, the OFR will have an independent chair who reports to Congress. The White House has approached candidates.
For years, individual statisticians, technology specialists and economists from regulators, financial institutions and academia had warned of the dangers. For years, they were dismissed as Jeremiahs and a root-and-branch reform of the data networks underpinning the financial system was rejected by the industry and regulators, which saw big costs and limited benefits.
John Geanakoplos, a Yale University professor, blames the Fed for not using data well, sometimes because of bureaucratic blockages, in one instance because officials balked at paying $400,000 for mortgage information, and sometimes because of a philosophical belief in self-correcting markets. In a recent presentation to the European Central Bank, he took aim at the "Greenspan-Bernanke doctrine" that "denied that there are bubbles, or that they could recognise one if they saw it".
In less damning terms, Keith Saxton, IBM's London-based global director of financial markets, points to the same problem. "Most of the data that the regulators and the central banks collect are what I call quite traditional," he says. "They have a view of 'The Bank' and everyone assumed that because that bank was healthy maybe there wasn't a problem with the system. It has turned out that the data they had about that bank weren't granular enough to be accurate: these guys can't get to the cash flow of an instrument."
But across the world, the evangelical geeks are gaining ground. In the US, Jack Reed, the Democratic senator from Rhode Island, took up the cause and managed to slip a new early-warning agency – the Office of Financial Research – into the mammoth Dodd-Frank financial reform bill that passed Congress last summer.
The OFR, which is being incubated in the Treasury before it gains independence, has begun work on standardising the components of every significant financial transaction. Eventually it will collect data, pull it all into a supercomputer and analyse the results, with the ultimate aim of spotting bubbles before they burst.
In a rare moment for Washington, an idea that was not proposed by the executive branch and without a big bloc of business support became law. A loose coalition of advocates managed to persuade one lawmaker, who sold the initiative to his colleagues in the face of scepticism from existing agencies. "Within the government there was a hesitancy to create an agency that was independent," says Mr Reed. The senator, an army veteran, says he wants the OFR to act like a "red team" – the military term for a group of soldiers charged with probing the weaknesses of their own comrades.
Words of change from previously sceptical bureaucrats are borne out by actions. The Fed does now buy the expensive granular mortgage data that Prof Geanakoplos highlighted. New rules will force more derivatives through clearing houses, allowing Mr Parkinson to grab data from fewer sources without alerting the market.
But the real data zealots think that regulators should go much further: the ultimate prize is a "dashboard" of the whole financial system. While anyone can view a snapshot of the stock market, the OFR and sister agencies in Europe and Asia would have the same visibility over darker parts of finance, allowing them to test various scenarios on Wall Street at the push of a button. The trouble is, at the moment, this is science fiction; it is impossible to create. "When something starts to go awry we go, 'what am I holding?' and I need to know it down to the loan level so I can run it against a scenario," says Mr Atkin.
"If Ford goes bankrupt what happens to the homes in Detroit? Well, I don't know how many homes in Detroit I've got in my mortgage-backed security because I can't unravel the bloody thing."
Francis Gross, head of external statistics at the Frankfurt-based ECB, points to the same issue: "The basic assumption is you have these vast pools of data that are quite homogeneous so that a spade is a spade – and that's where the problem comes."
The reason is that the building blocks of so-called "reference data" are not standard. As banks have grown by acquisition they have acquired thousands of legacy systems spread across different businesses – they do not have standard ways of recording data even within the group. So plotting relationships across the financial system is almost impossible. "The trading desk may book a transaction as Deutsche Bank and code that as 'DB'," says Lew Alexander, the Treasury official in charge of setting up the OFR. "When it gets to accounting, 'DB' may mean Dresdner Bank."
It is these non-standard "identifiers", for companies and for instruments, that Mr Gross and Mr Alexander are now trying to reconcile on both sides of the Atlantic. Unravelling garbled transactions costs the industry hundreds of millions of dollars a year in people and IT but during the boom years it always seemed too fiddly to fix.
One banker says: "We do a transaction and we get a lot of breaks and reconciliations ... Generally over 90 per cent of them are due to inconsistency of reference data, not due to misunderstanding between parties. If we all had the same reference data we would revolutionise how operations are done in our firms." He adds: "Logistically it's impossible to do unless it's imposed."
Mr Gross, who is leading calls for international standards of reference data, says regulators must be in the driving seat or it will be "ask[ing] cats to herd themselves".
The US now has the structure to start work, although even after Mr Reed managed to get the OFR through Congress, a handful of powerful Republicans continue to oppose it. Karl Rove, an adviser to former president George W. Bush, and Richard Shelby, the senior Republican on the Senate banking committee, both think it reeks of big government.
"I believe that the Democrats' new Office of Financial Research will not only fail to detect systemic threats and asset price bubbles in the future, it may threaten the civil liberties and privacy of Americans, waste billions of dollars of taxpayer resources and lull markets into the false belief that this new government power will protect the financial system from risk," says Mr Shelby.
Some Wall Street executives are worried about disclosing their trading positions to anyone, even regulators policing the system for systemic risk. Says the banker who has watched the OFR closely and supports it: "There is a concern about people giving up their positions ... There was a time in 1905 when people didn't report their income to the [Internal Revenue Service]. I think it will become: you do a transaction, you report it to the OFR."
From the other end of the political spectrum, liberal Democrats in Congress are worried the Treasury and the Fed are paying lip service to the OFR and will not give it the tools to be intrusive enough. They think scepticism among senior officials endures.
It is certainly a never-ending struggle. Forty years before Mr Parkinson grappled with Lehman, one of his antecedents extolled the virtues of technology for market supervision. In 1968, a year in which the New York Stock Exchange had to close for days at a time because paper records of trades were so out of hand, Manuel Cohen, chairman of the SEC, boasted that his agency was now using "its own computer" to monitor markets.
"We are able to provide a measure of protection to investors that theretofore had been virtually impossible due to budget and 'manpower' limitations," he said. "But our techniques in this area are not as fully developed as they will be."
..........................................
Information technology
How innovation has come to mean different things on different coasts.
In recent decades, a curious paradox has hung over the American economy. On the west coast, a gaggle of entrepreneurial companies, filled with some of the country's brightest brains, has been scrambling to track data in the smartest and most innovative way, writes Gillian Tett.
Companies such as Google, Amazon and Facebookare now able to monitor what consumers and businesses are doing around the world in real time. They can track everything from book purchases to friendship links and the consumption of breakfast cereal.
But on the east coast, another collection of highly talented brains has been delivering a very different form of innovation. Wall Street has produced a plethora of products and processes that has made the financial system more complex and (often) more opaque.
But while bankers have used cutting-edge computer technology to, say, develop ultra-fast automatic trading strategies, the data-handling innovations developed on the west coast have been slow to move east.
As recently as six years ago, traders in the credit default swaps market, for example, were still conducting deals by fax. Banks' back offices were not standardised and regulators could not collect data from them in anything resembling a timely manner.
Beyond banking, many other parts of the financial world went almost entirely untracked by regulators, who remain behind the technological curve.
The question that hangs over the Office of Financial Research, being set up as part of reforms to the sector, is whether these different west and east coast worlds can now meet – and apply Silicon Valley-style innovation to the financial system as a whole. Can the techniques that allow Facebook to aggregate data on online friends in a flash be used to track derivatives trades, say?
Optimists argue that the answer is yes, given the extraordinary strides in computing power that have already occurred. Officials linked to the OFR have started talking to companies such as IBMabout how to transplant innovations in the non-financial world into a coherent form of data collection in finance.
But pessimists retort that financial companies have little incentive to co-operate; after all, opacity has on the whole served Wall Street well, enabling traders to enjoy fat profits.
Either way, the really big question is whether the type of entrepreneurial, innovative drive that inspires Silicon Valley can be transplanted to the state sector.
"If you really wanted to revolutionise [data collection], you should ask somebody like Google to run it, and pay them properly," observes one senior banker, only partly in jest.
Right now, however, that prospect seems even harder to imagine than a world where the OFR starts to fly.
Copyright The Financial Times Limited 2011. Print a single copy of this article for personal use. Contact us if you wish to print more to distribu
Subscribe to:
Posts (Atom)