“If all You Have is a Hammer…” — How Useful is Humanitarian Crowdsourcing?

This article was originally published on MobileActive on 20 October 2010. At the time there was a lot of debate about the article because people misinterpreted it as a blanket dismissal of crowdsourcing, which it plainly was not. The MobileActive website is no longer… Active, but people continue to ask me about the article, so I am re-posting it here. I stand by most of my critique, particularly as many of the points were later confirmed by Ushahidi’s own evaluation of the Haiti project. I am not posting it to renew the discussion, but because I believe the points it makes about new technology still stand, particularly at the end of the article: For while the tools that we now have at our disposal are important, we have a responsibility to use them for the right tasks.

The original article was accompanied by an Editor’s Note: In this article, guest contributor Paul Currion looks at the potential for crowdsourcing data during large-scale humanitarian emergencies, as part of our “Deconstructing Mobile” series. Paul is an aid worker who has been working on the use of ICTs in large-scale emergencies for the last 10 years. He asks whether crowdsourcing adds significant value to responding to humanitarian emergencies, arguing that merely increasing the quantity of information in the wake of a large-scale emergency may be counterproductive. Instead, the humanitarian community needs clearly defined information that can help in making critical decisions in mounting their programmes in order to save lives and restore livelihoods. By taking a close look at the data collected via Ushahidi in the wake of the Haiti earthquake, he concludes that crowdsourced data from affected communities may not be useful for supporting the response to a large-scale disaster.

1. The Rise of Crowdsourcing in Emergencies

Ushahidi, the software platform for mapping incidents submitted by the crowd via SMS, email, Twitter or the web, has generated so many column inches of news coverage that the average person could be mistaken for thinking that it now plays a central role in coordinating crisis responses around the globe. At least this is what some articles say, such as Technology Review’s profile of David Kobia, Director of Technology Development for Ushahidi. For most people, both inside and outside the sector, who lack the expertise to dig any deeper, column inches translate into credibility. If everybody’s talking about Ushahidi, it must be doing a great job — right?

Maybe.

Ushahidi is the result of three important trends:

1. Increased availability and utility of spatial data;
2. Rapid growth of communication infrastructure, particularly mobile telephony; and
3. Convergence of networks based on that infrastructure on Internet access.

Given those trends, projects like Ushahidi may be inevitable rather than unexpected, but inevitability doesn’t give us any indication of how effective these projects are. Big claims are made about the way in which crowdsourcing is changing the way in which business is done in other sectors, and now attention has turned to the humanitarian sector. John Della Volpe’s short article in the Huffington Post is an example of such claims:

“If a handful of social entrepreneurs from Kenya could create an open-source “social mapping” platform that successfully tracks and sheds light on violence in Kenya, earthquake response in Chile and Haiti, and the oil spill in the Gulf — what else can we use it for?”

The key word in that sentence is “successfully”. There isn’t any evidence that Ushahidi “successfully” carried out these functions in these situations; only that an instance of the Ushahidi platform was set up. This is an extremely low bar to clear to achieve “success”, like claiming that a new business was successful because it had set up a website. There has lately been an unfounded belief that the transformative effects of the latest technology are positively inevitable and inevitably positive, simply by virtue of this technology’s existence.

2. What does Successful Crowdsourcing Look Like?

To be fair, it’s hard to know what would constitute “success” for crowdsourcing in emergencies. In the case of Ushahidi, we could look at how many reports are posted on any given instance — but that record is disappointing, and the number of submissions for each Ushahidi instance is exceedingly small in comparison to the size of the affected population — including Haiti, where Ushahidi received the most public praise for its contribution.

In any case, the number of reports posted is not in itself a useful measure of impact, since those reports might consist of recycled UN situation reports and links to the Washington Post’s “Your Earthquake Photos” feature. What we need to know is whether the service had a significant positive impact in helping communities affected by disaster. This is difficult to measure, even for experienced aid agencies whose work provides direct help. Perhaps the best we can do is ask a simple question: if the system worked exactly as promised, what added value would it deliver?

As Patrick Meier, a doctoral student and Director of Crisis Mapping and Strategic Partnerships for Ushahidi has explained, crowdsourcing would never be the only tool in the humanitarian information toolbox. That, of course, is correct and there is no doubt that crowdsourcing is useful for some activities — but is humanitarian response one of those activities?

A key question to ask is whether technology can improve information flow in humanitarian response. The answer is that it absolutely can, and that’s exactly what many people, including this author, have been working on for the last 10 years. However, it is a fallacy to think that if the quantity of information increases, the quality of information increases as well. This is pretty obviously false, and, in fact, the reverse might be true.

From an aid worker’s perspective, our bandwidth is extremely limited, both literally and metaphorically. Those working in emergency response — official or unofficial, paid or unpaid, community-based or institution-based, governmental or non-governmental — don’t need more information, they need better information. Specifically, they need clearly defined information which can help them to make critical decisions in mounting their programmes in order to save lives and restore livelihoods.

I wasn’t involved with the Haiti response, which made me think that perhaps my doubts about Ushahidi were unfounded and that perhaps the data they had gathered could be useful. In the course of discussions on Patrick Meier’s blog, I suggested that the best way for Ushahidi to show my position was wrong would be to present a use case to show how crowdsourced data could be used (as an example) by the Information Manager for the Water, Sanitation and Hygiene Coordination Cluster, a position which I filled in Bangladesh and Georgia. Two months later, I decided to try that experiment for myself.

3. In Which I Look At The Data Most Carefully

The only crowdsourced data I have is the Ushahidi dataset for Haiti, but since Haiti is claimed as a success, that seemed like to be a good place to start. I started by downloading and reading through the dataset — the complete log of all reports posted in Ushahidi. It was a mix of two datastreams:

• Material published on the web or received via email, such as UN sitreps, media reports, and blog updates, and
• Messages sent in by the public via the 4636 SMS shortcode established during the emergency.

I was struck by two observations:

• One of the claims made by the Ushahidi team is that its work should be considered an additional datastream for aid workers. However, the first datastream is simply duplicating information that aid workers are already likely to receive.
• The 4636 messages were a novel datastream, but also the outcome of specific conditions which may not hold in places other than Haiti. The fact that there is a shortcode does not guarantee results, as can be seen in the virtually empty Pakistan Ushahidi deployment.

I considered that perhaps the 4636 messages could demonstrate some added value. They fell into three broad categories: the first was information about the developing situation, the second was people looking for information about family or friends missing after the earthquake, and the third and by far the largest, was general requests for help.

I tried to imagine that I had been handed this dataset on my deployment to Haiti. The first thing I would have to do is to read through it, clean it up, and transcribe it into a useful format rather than just a blank list. This itself would be a massive undertaking that can only be done by somebody on the ground who knows what a useful format would be. Unfortunately, speaking from personal experience, people on the ground simply don’t have time for that, particularly if they are wrestling with other data such as NGO assessments or satellite images.

For the sake of argument, let’s say that I somehow have the time to clean up the data. I now have a dataset of messages regarding the first three weeks of the response. 95% of those messages are for shelter, water and food. I could have told you that those would be the main needs even before I arrived in position, so that doesn’t add any substantive value. On top of that, the data is up to 3 weeks old: I’d have to check each individual report just to find out just whether those people are still in the place that they were when they originally texted, and whether their needs have been met.

Again for the sake of argument, let’s say that I have a sufficient number of staff (as opposed to zero, which is the number of staff you usually have when you’re an information manager in the field) and they’ve checked every one of those requests. Now what? There are around 3000 individual “incidents” in the database, but most of those contain little to no detail about the people sending them. How many are included in the request, how many women, children and old people are there, what are their specific medical needs, exactly where they are located now — this is the vital information that aid agencies need to do their work, and it simply isn’t there.

Once again for the sake of argument, let’s say that all of those reports did contain that information — could I do something with it? If approximately 1.5 million people were affected by the disaster, those 3000 reports represent such a tiny fraction of the need that they can’t realistically be used as a basis for programming response activities. One of the reasons we need aid agencies is economies of scale: procuring food for large populations is better done by taking the population as a whole. Individual cases, while important for the media, are almost useless as the basis for making response decisions after a large-scale disaster.

There is also this very basic technical question: once we have this crowdsourced data, what do we do with? In the case of Ushahidi, it was put on a Google Maps mash-up — but this is largely pointless for two reasons. First, there’s a simple question of connectivity. Most aid workers and nearly all the population won’t have reliable access to the Internet, and where they do, won’t have time to browse through Google Maps. (It’s worth noting that this problem is becoming lessimportant as Internet connectivity, including the mobile web, improves globally — but also that the places and people prone to disasters tend to be the last to benefit from that connectivity.)

Second, from a functional perspective, the interface is rudimentary at best. The visual appeal of Ushahidi is similar to that of Powerpoint, casting an illusion of simplicity over what is, in fact, a complex situation. If I have 3000 text messages saying “I need food and water and shelter”, what added value is there from having those messages represented as a large circle on a map? The humanitarian community often lacks the capacity to analyse spatial data, but this map has almost no analytical capacity. The clustering of reports (where larger bubbles correspond to the places that most text messages refer to) may be a proxy for locations with the worst impact; but a pretty weak proxy derived from a self-selecting sample.

In the end, I was reduced to bouncing around the Ushahidi map, zooming in and out on individual reports — not something I would have time to do if I was actually in the field. Harsh as it sounds, my conclusion was that the data that crowdsourcing of this type is capable of collecting in a large-scale disaster response is operationally useless. The reason for this has nothing to do with Ushahidi, or the way that the system was implemented, but with the very nature of crowdsourcing itself.

4. Crowdsourcing Response or Digital Voluntourism?

One of the key definitions of “crowdsourcing” was provided by Jeff Howe in a Wired article that originally popularised the term: taking “a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” In the case of Haiti, part of the reason why people mistakenly thought crowdsourcing was successful, was because there were two different “crowds” being talked about.

The first was the global group of volunteers who came together to process the data that Ushahidi presented on its map. By all accounts, this was definitely a successful example of crowdsourcing as per Howe’s definition. We can all agree that this group put a lot of effort into their work. However, the end result wasn’t especially useful. Furthermore, most of those volunteers won’t show up for the next response — and in fact they didn’t for Pakistan.

The media coverage of Ushahidi focuses mainly on this first crowd — the group of volunteers working remotely. Yet, the second crowd is much more important: the affected community. Reading through the Ushahidi data was heartbreaking, indeed. But we already knew that people needed food, water, shelter, medical aid — plus a lot more things that they wouldn’t have been thinking of immediately as they stood in the ruins of their homes. In the Ushahidi model, this is the crowd that provides the actual data, the added value, but the question is whether crowdsourced data from affected communities could be useful from an operational perspective of organising the response to a large-scale disaster.

The data that this crowd can provide is unreliable for operational purposes for three reasons. First, you can’t know how many people will contribute their information, a self-selection bias that will skew an operational response. Second, the information that they do provide must be checked — not because affected populations may be lying, but because people in the immediate aftermath of a large-scale disaster do not necessarily know all that they specifically need or may not provide complete information. Third, the data is by nature extremely transitory, out-of-date as soon as it’s posted on the map.

Taken together, these three mean that aid agencies are going to have to carry out exactly the same needs assessments that they would have anyway — in which case, what use was that information in the first place?

5. Is Crowdsourcing Raising Expectations That Cannot be Met?

Many of the critiques that the crowdsourcing crowd defend against are questions about how to verify the accuracy of crowdsourced information, but I don’t think that’s the real problem. It’s the nature of an emergency that all information is provisional. The real question is whether it’s useful.

So to some extent those questions are a distraction from the real problems: how to engage with affected communities to help them respond to emergencies more effectively, and how to coordinate aid agencies to ensure and effective response. On the face of it, crowdsourcing looks like it can help to address those problems. In fact, the opposite may be true.

Disaster response on the scale of the Haiti earthquake or the Pakistan floods is not simply a question of aggregating individual experiences. Anecdotes about children being pulled from rubble by Search and Rescue teams are heart-warming and may help raise money for aid agencies but such stories are relatively incidental when the humanitarian need is clean water for 1 million people living in that rubble. Crowdsourced information — that is, information voluntarily submitted in an open call to the public — will not ever provide the sort of detail that aid agencies need to procure and supply essential services to entire populations.

That doesn’t mean that crowdsourcing is useless: based on the evidence from Haiti, Ushahidi did contribute to Search and Rescue (SAR). The reason for that is because SAR requires the receipt of a specific request for a specific service at a specific location to be delivered by a specific provider — the opposite of crowdsourcing. SAR is far from being a core component of most humanitarian responses, and benefits from a chain of command that makes responding much simpler. Since that same chain of command does not exist in the wider humanitarian community, ensuring any response to an individual 4636 message is almost impossible.

This in turn raises questions of accountability — is it wholly responsible to set up a shortcode system if there is no response capability behind it, or are we just raising the expectations of desperate people?

6. Could Crowdsourcing Add Value to Humanitarian Efforts?

Perhaps it could. However, the problem is that nobody who is promoting crowdsourcing currently has presented convincing arguments for that added value. To the extent that it’s a crowdsourcing tool, Ushahidi is not useful; to the extent that it’s useful, Ushahidi is not a crowdsourcing tool.

To their credit, this hasn’t gone unnoticed by at least some of the Ushahidi team, and there seems to be something of a retreat from crowdsourcing, described in this post by one of the developers, Chris Blow:

“One way to solve this: forget about crowdsourcing. Unless you want to do a huge outreach campaign, design your system to be used by just a few people. Start with the assumption that you are not going to get a single report from anyone who is not on your payroll. You can do a lot with just a few dedicated reporters who are pushing reports into the system, curating and aggregating sources.”

At least one of the Ushahidi team members now talks about “bounded crowdsourcing” which is a nonsensical concept. By definition, if you select the group doing the reporting, they’re not a crowd in the sense that Howe explained in his article. This may be an area where Ushahidi would be useful, since a selected (and presumably trained) group of reporters could deliver the sort of structured data with more consistent coverage that is actually useful — the opposite of what we saw in Haiti. Such an approach, however, is not crowdsourcing.

Crowdsourcing can be useful on the supply side: for example, one of the things that the humanitarian community does need is increased capacity to process data. One of the success stories in Haiti was the work of the OpenStreetMap (OSM) project, where spatial data derived from existing maps and satellite images was processed remotely to build up a far better digital map of Haiti than existed previously. However, this processing was carried out by the already existing OSM community rather than by the large and undefined crowd that Jeff Howe described.

Nevertheless this is something that the humanitarian community should explore, especially for data that has a long-term benefit for affected countries (such as core spatial data). To have available a recognised group of data processors who can do the legwork that is essential but time-consuming would be a real asset to the community — but there we’ve moved away from the crowd again.

7. A Small Conclusion

My critique of crowdsourcing — shared by other people working at the interface of humanitarian response and technology — is not that it is disruptive to business asusual. My critique is that it doesn’t work — not just that it doesn’t work given the constraints of the operational environment (which Ushahidi’s limited impact in past deployments shows to be largely true), but that even if the concept worked perfectly, it still wouldn’t offer sufficient value to warrant investing in.

Unfortunately, because Ushahidi rests its case almost entirely on the crowdsourcing concept, this article may be interpreted as an attack on Ushahidi and the people working on it. However, all of the questions I’ve raised here are not directed solely at Ushahidi (although I hope that there will be more debate about some of the points raised) but hopefully will become part of a wider and more informed debate about social media in general within the humanitarian community.

Resources are always scarce in the humanitarian sector, and the question of which technology to invest in is a critical one. We need more informed voices discussing these issues, based on concrete use cases because that’s the only way we can test the claims that are made about technology. For while the tools that we now have at our disposal are important, we have a responsibility to use them for the right tasks.