Who Are the Doctors Most Trusted by Doctors? Big Data Can Tell You.

Ki Mae Heussner, GigaOm, November 16, 2012

ZocDoc, Healthgrades, Vitals, Yelp and other sites can tell you what patients think of their doctors. But finding out in any aggregate way what doctors think of their peers has been much harder, if not near impossible, for patients — up until now.

By accessing information in government databases through FOIA (Freedom of Information Act) requests, healthcare innovators are now able to share connections between doctors that are based on millions of physician referrals — a valuable indicator of who doctors hold in esteem.

Last month, Fred Trotter, a self-identified “hacktivist,” revealed that he had obtained a dataset of Medicare physician referrals through a FOIA request and was making the initial data available to those who supported a Medstartr crowdfunding campaign meant to build out his “DocGraph” and make it freely available. This week, he announced that he not only blew past his $15,000 funding goal, but was launching a second campaign to integrate his current data with an additional dataset.

HealthTap, a Palo Alto-based startup that connects patients with an online network of 17,000 doctors, also this week launched a new feature based partly on Trotter’s data. Called “DOConnect,” it combines Trotter’s Medicare data with physician data from its own site and other sources to give patients a new window into their doctors’ networks.

“This isn’t just friendships and business connections. This is who doctors trust,” said HealthTap co-founder and CEO Ron Gutman. “If you could know who your doctor’s doctor is, if you knew who they would choose, this lets you see that for the first time.”

The new tool, which reflects 25 million doctor referral connections, enables patients to see how many doctors are linked to a particular doctor, as well as their locations. As patients search for new physicians and specialists, being able to see who their current doctors are linked with could help them decide who to visit.  
It also gives doctors an opportunity to build online networks that reflect their offline networks, Gutman said. In a post about his “DocGraph” project, Trotter said that his data wasn’t strictly a “referral” data set because, in some cases, doctors might be linked through a patient they both happened to see at the same time, not through an active referral. But Gutman emphasized that HealthTap’s DOConnect considered more than Medicare referrals in mapping connections between doctors.

In releasing the dataset, Trotter said his main goal was to create doctor-rating algorithms that “patients find useful and doctors find fair.” But he also hoped that academics, health policy wonks, entrepreneurs and others would use it to bring more transparency to health care overall.

Todd Park, the U.S. Chief Technology Officer, has frequently talked up the value of “setting data free” and has backed hackathons, “datapaloozas” and other open data initiatives to highlight the need for innovators to use government data for the public good — this is a great example of that vision and, hopefully, points to more similar projects in the future.

“Our goal is to empower the patient, make the system transparent and accountable, and release this d

Andrew McAfee : Let the Crowd Fix Your Product's Bugs

Andrew McAfee, Harvard Business Review Blog, November 6, 2012

I'm starting to come to the conclusion that of all the myths businesses and their leaders tell themselves, one of the most harmful is that they know where the expertise is. The more I learn about the results from crowdsourcing and open innovation efforts, the more I believe that the smart strategy is to expose your problems and challenges to as many people as possible and let them show you what they can do. Here's my most recent example of the power of this approach.

The online startup Kaggle assembles a diverse group of people from around the world to work on tough problems submitted by organizations. The company runs data science competitions, where the goal is to arrive at a better prediction than the submitting organization's starting 'baseline' prediction. Results from these contests are striking in a couple ways. For one thing, improvements over the baseline are usually substantial. In one case, Allstate submitted a dataset of vehicle characteristics and asked the Kaggle community to predict which of them would have later personal liability claims filed against them. The contest lasted approximately three months, and drew in more than 100 contestants. The winning prediction was more than 270% better than the insurance company's baseline.

Another interesting fact is that the majority of Kaggle contests are won by people who are marginal to the domain of the challenge — who, for example, made the best prediction about hospital readmission rates despite having no experience in health care — and so would not have been consulted as part of any traditional search for solutions. In many cases, these demonstrably capable and successful data scientists acquired their expertise in new and decidedly digital ways.

Between February and September of 2012 Kaggle hosted two competitions sponsored by the Hewlett Foundation about computer grading of student essays. Improvements in this area are important because essays are better at capturing student learning than multiple choice questions, but much more expensive to grade when human raters are used. So automatic grading of written answers would both improve the quality of testing and lower its cost. Kaggle and Hewlett worked with many education experts to set up the competitions, and as they were preparing to launch some of these people were worried.

The first contest was to consist of two rounds. Eleven established educational testing companies would compete against each other in the first, with members of Kaggle's community of data scientists invited to join in, individually or in teams, in the second. The experts were worried that the Kaggle crowd would simply not be competitive. After all, each of the testing companies had been working on automatic grading for some time, and had devoted substantial resources to the problem. Their hundreds of man years of accumulated experience and expertise seemed like an insurmountable advantage over a bunch of novices.

They needn't have worried. Many of the 'novices' drawn to the challenge outperformed all of the testing companies in the essay competition, and came closer to the consensus score of the human graders than did any of the humans themselves. The surprises continued when Kaggle investigated who the top performers were. In both competitions, none of the top three finishers had any previous significant experience with either essay grading or natural language processing. And in the second competition, none of the top three finishers had any formal training in artificial intelligence beyond a free online course offered by Stanford AI faculty and open to anyone in the world who wanted to take it. And people all over the world did, and learned a lot from it. The top three individual finishers were from, respectively, America, Slovenia, and Singapore.

Businesses certainly know where a lot of the relevant expertise is in any situation, but results like those from Kaggle show me that they certainly don't know where all of it is. As the open source software advocate Eric Raymond famously observed, with enough eyeballs all bugs are shallow. So why not expose your tough problems to as many eyeballs as possible?

Crowdsourcing Medical Treatments

Samahope Crowdsources Simple, Life-Saving Surgeries For The Poor

Ellen McGirt, Fast Company, November 2, 2012

Veteran social entrepreneur Leila Janah of Samasource recently co-launched a new project to crowdfund medical treatment for the very poor. Think of it as Kiva for surgery.


“I just started bawling,” Leila Janah is telling me about a trip she took to Sierra Leone earlier this year. "I’m usually pretty steely as a matter of course. But I’ve been crying a lot more lately.”

Janah, the founder of Samasource, a nonprofit organization that brings paid digital work to very poor women and youth, is no stranger to harsh realities. She studies them for a living.

But a sweet teenaged girl named Tiangay Kaiwo moved Janah to tears. Kaiwo was waiting for surgery at the government hospital in Bo to repair the extensive damage to her body after her teacher brutally raped her. Traditional tribal remedies involving herbs and a bath in boiling water exacerbated her condition. Kaiwo had been living with a painful rectovaginal fistula for over a year, but best as Janah could tell, the rapes began when she was 12 years old. Janah met dozens of girls and women with similar stories, all needing life-changing surgeries that nobody could afford. “Nobody even to hold their hands, to tell them that this wasn’t their fault,” Janah said.

When the slice of the market you are trying to corner is as troubled as the one that Janah is, then tears are clearly a rational response. But after tears, at least if you’re built like Janah, comes action.

Enter her latest project, Samahope, an experiment in crowdfunding medical treatment--like burn care--for the very poor. Think of it as Kiva for surgery. “There are millions of people who need corrective surgeries that we take for granted in the West,” she says. But the very poor often need types of care, like fistula surgery, that are now wholly unfamiliar and largely unnecessary in the developed world. (You can contribute to the development of the site via their Indiegogo campaign here.)

The site launched last month and has already funded a handful of surgeries; there are over 70 profiles on the site. If you believe in the premise of the League of Extraordinary Women, then the business case is clear: If you get one girl back on her feet, she can go to school. If she goes to school, she can get a job. Enough girls join the workforce and a country gets uplifted. But Janah sees another benefit. “They want to say what happened to them, to tell their own stories,” she says.

She has collected so many of these stories--of the poor and the embattled and their search for basic human rights through employment--that she’s writing a book. “I just interviewed a security guard at a hotel in Freetown," Janah said. "He grew up as a rebel and child soldier in the conflict--think about that for a minute--then forced into the diamond mines. His life was so full of conflict, a constant struggle to access basic human resources, that it’s impossible to wrap my mind around.” Recalling young Kaiwo, “For someone like her, being able to tell her story and help other girls not become a victim is a very powerful thing.”

Samasource, which last month closed a $7.5 million round of philanthropic funding led by The MasterCard Foundation, has become the darling of the tech crowd for its deft use of the Internet to match an excess capacity of potential workers with the jobs they need to live in dignity. “But the scale of the problems can seem so great compared to the resources you have to address them,” says Janah--thus, the crowdsourcing project. Unlike her peers in the for-proft tech world, she is not going to be able to turn to her staffers with breathless reports of sky-high valuations, rounds of venture funding, or promises of equity upside.
And yet, Janah is convinced that dignified work can resurrect even the most damaged lives, and that her own business case is sound. “We’ve gotten the microwork model on the agenda of a lot of foundations and government entities. We just have to prove it can scale.”

Janah recalls with fondness the “aha” moment when she knew that Samasource could actually be a business. But the slog of ramping up to achieve a massive goal is largely free of lightbulb moments. “Not a day goes by when I don’t doubt myself or question something,” she says. So instead, she takes the power of microwork and puts it to work for herself and her team. “I had to manage my own psychology around this. So, I’ve trained myself to pause and celebrate each step.”

She rattles off a list of things that sound more startup than do-good: Realign your expectations, hit your goals, stay close to the customer, stay connected to your mission. She ends up sounding more like a Zen master than elevator pitcher. “It’s about looking down and doing what’s in front of you. Truly savor it. Then do the next thing. My job is to make sure we’re all going in the right direction, and at the end of the year, the sum of those steps adds up to something really great.”

Artificial Intelligence, Powered by Many Humans

Crowdsourcing can create an artificial chat partner that's smarter than Siri-style personal assistants.

Tom Simonite, Technology Review, September 10, 2012

Personal assistants such as Apple's Siri may be useful, but they are still far from matching the smarts and conversational skills of a real person. Researchers at the University of Rochester have demonstrated a new, potentially better approach that creates a smart artificial chat partner from fleeting contributions from many crowdsourced workers.

Crowdsourcing typically involves posting simple tasks to a website such as Amazon Mechanical Turk, where Web users complete them for a reward of a few cents. The tasks are often simple, repetitive jobs that are easy for humans but tough for computers, such as categorizing images. Crowdsourcing has become a popular way for companies to handle such tasks, but some researchers, including the group at Rochester, believe it can also be used to take on more complex tasks.

When people talk to the new crowd-powered chat system, called Chorus, using an instant messaging window, they get an experience practically indistinguishable from chatting with a single real person. Yet behind the scenes, each response is the result of tens of people paid a few cents to perform small tasks: including suggesting possible replies and voting for the best suggestions submitted by other workers.

Tests where Chorus was asked for travel advice showed that it could be smarter than any one individual in the crowd, because around seven people were contributing to its responses at any one time. Helpers built this way might also be cheaper than paying a conventional one-on-one assistant. "It shows how a crowd-powered system that is relatively simple can do something that AI has struggled to do for decades," says Jeffrey Bigham, an assistant professor at the University of Rochester, and a member of the research team that created Chorus. Bigham jokes that Chorus is more likely to pass a Turing Test, which challenges an artificial intelligence system to fool someone into thinking it's human, than conventional chat software, although it may not meet most definitions of artificial intelligence.

In trials of the system, people asked Chorus for advice on restaurants to visit in Los Angeles and New York, and quickly received suggestions. Feedback such as "Hmm. That seems pricey," was quickly taken on board by the crowd, which came up with an alternative. AI systems such as Siri typically have difficulty following this kind of back-and-forth conversation, particularly in colloquial language.

Bigham worked with Rochester colleagues Walter Lasecki and Rachel Wesley, and Anand Kulkarni, the cofounder of crowdsourcing company MobileWorks (see "Human Workers, Managed by an Algorithm"). Their goal was to find a new way to increase the power of crowdsourcing, which is typically limited to simple, isolated tasks, such as adding labels to image files. "What we're really interested in is when a crowd as a collective can do better than even a high-quality individual," says Bigham, by combining work on many simple tasks into a coherent, complex whole.

Chorus does that with three simple types of task. First, any new chat updates from the human user are passed along to many crowd workers, who are asked to suggest a reply. Those suggestions are then voted on by crowd workers to determine the one that will be sent back.

A final mechanism creates a kind of working memory that ensures that Chorus's replies reflect the history of the conversation so far, crucial if it is to carry out long conversations—something that is a challenge for apps like Siri and even AI chatbots intended to showcase conversational skills.

For the working memory component, crowd members are asked to maintain a short running list of the eight most important snippets of information under discussion, to be used as a reference when workers suggest replies. This is important, as to allow for the natural turnover of crowdsourcing workers. "A single person may not be around for the duration of the conversation—they come and go, and some may contribute more than others," says Bigham.

Bigham says Chorus has the potential to be more than just a neat demonstration. "We definitely want to start embedding it into real systems," he says. "Perhaps you could help someone with cognitive impairment by having a crowd as a personal assistant."

Another possibility is to combine Chorus with another system previously developed at Rochester, which has crowd workers collaborate to steer a robot. "Could you create a robot this way that can drive around and interact intelligently with humans?" asks Bigham.

Michael Bernstein, an assistant professor at Stanford University who is currently doing research at Facebook, agrees that Chorus could lead to real-world applications (see "Adding Human Intelligence to Software").

"You could go from today where I call AT&T and speak with an individual, to a future where many people with different skills work together to act as a single incredibly intelligent tech support," says Bernstein. He says the Chorus software could become a true expert if it were able to direct incoming questions to members of the crowd with particular knowledge or skills.

However, Bernstein adds that it may be necessary to add more reviewing steps to Chorus in order to filter a crowd's suggestions to prevent it developing a split personality when faced with difficult questions. This is a familiar problem in applying crowdsourcing. Bigham's crowd-steered robot, for example, has been known to crash into obstacles dead ahead because half the crowd workers steering it wanted it to go left, and the other half wanted it to go right.