Beating Hindsight: Forecasting for Humanitarian Planning and Preparedness

Why is the future important?

All human endeavours are based on forecasting, i.e. attempts to narrow down possible future outcomes to a manageable range. We generally find forecasting extremely challenging due to psychological biases and cultural conditioning, but recent research has shown that better forecasting can be achieved through a particular combination of mental habits and skills, which some people already possess and most people can be trained in.

The humanitarian community has recently focused on improving leadership, but leadership needs to be supported by more rigorous and strategic thinking in order to be effective. Forecasting as a discipline for predicting the future based on past events and present insight is not pursued in a systematic and coherent way in the humanitarian sector. A small number of humanitarian organisations that pursue and promote evidence-based analysis have begun to move towards forecasting based on scenarios, but these organisations are in the minority.

As a result, although individual staff and units working within humanitarian organisations are asked to make forecasts all the time for the purposes of planning and response, this forecasting is almost solely estimation based on instinct, with very little attempt to improve the accuracy of estimates based on feedback. Despite this, these forecasts form the basis of planning and preparedness activities, which then form the basis for operations when an emergency strikes; forecasts are also required to support advocacy and fundraising activities.

This note explores a specific way in which the humanitarian community might improve its capacity in this area.

Why we struggle to see ahead

The challenges to forecasting in the humanitarian sector can be divided into two categories:

1. Outside

Humanitarian emergencies are often felt to be fundamentally unpredictable, due to a combination of three factors:

1. The perceived randomness and variable impact of natural disasters such as floods and earthquakes;
2. The complex nature of human-caused emergencies such as wars and the resulting refugee flows;
3. The unpredictability of wider political and economic systems in which humanitarian action is embedded.

These concerns are valid, but each of them can be addressed. In the first case, floods and earthquakes are not random, although they are unpredictable. They are more usefully predicted through technical means rather than forecasting, although forecasting can provide an analytical supplement to such early warning. The impact of such natural disasters is something that can be estimated, and this is one of the areas where forecasting can effectively combine with other human-led analysis to provide better estimates.

The second two points, complex emergencies and the wider political and economic systems in which they take place, are highly amenable to human prediction. Although algorithm-based prediction is increasingly accurate, a combination of human and computer analysis is likely to remain the most reliable approach to forecasting. The Good Judgement Project has repeatedly shown that it is possible to improve forecasts of the extremely complex situations that humanitarian action responds to.

2. Inside

Internal challenges are likely to be more difficult to resolve. Evidence provided by early warning projects has historically not lead to improved response due to the lack of political will inherent in a sector where incentives to respond are poorly aligned and decisions are not evidence-based. Early warning has been of limited success for a number of reasons, including:

1. They are necessarily limited to quite specific domains (e.g. famine early warning) and have frequently been vague in terms of predicting impact;
2. The sector has a high barrier to entry that external experts find difficult to breach, requiring a large investment in partnership work;
3. The community struggles to incorporate new conceptual models into its workflows, since those workflows have been developed in an ad hoc and reactive manner.

These challenges must all be addressed — particularly in communicating to the key stakeholders — but are not insurmountable. We propose to establish a forecasting tournament that would complement rather than compete with existing early warning systems; the latter would provide additional information which forecasters would draw on. A tournament would also offer an entry point for external experts (and non-experts) that would enable them to bring their expertise to bear on humanitarian problems. Forecasting can be pitched to humanitarian organisations as much as an exercise in capacity development as it is an exercise in prediction.

The third point is perhaps the most difficult to address. While the tournament approach is sufficiently novel and engaging that humanitarian actors are more likely to take an interest in it, poor organisational policies and accompanying lack of resources mean that evidence-based decision-making is still not the norm. The fact that improved forecasts will never predict the future with 100% certainty is likely to be offputting to senior decision-makers, regardless of the accuracy of their current forecasts.

This will need to be addressed by educating decision-makers in how to accept uncertainty, and to communicate that to other stakeholders (including the media). We will also need to experiment with ways of connecting forecasting directly to response, such as coordination agreements that set out clear actions once a forecast reaches a certain threshold — for example, a 90% probability of internal displacement might trigger pre-positioning of relief materials in the receiving area. Forecasting needs to be part of wider efforts such as this to move the sector towards evidence-based decision-making.

The Humanitarian Forecasting Tournament

The best available evidence suggests that forecasting tournaments are the most effective way to achieve better forecasts and improve forecasting skills. We could envision such a tournament managed by the Start Network, but developed with existing and new partners. Individual Start member agencies might wish to play a role in developing and managing the tournament, but participation would be open to staff of all agencies regardless.

The tournament itself will be a variation on the type of tournament run by the Good Judgement Project, but with a focus on topics relevant to the humanitarian sector (rather than wider political and economic developments) and on questions with a direct impact on humanitarian operations. Such forecasting seems to work best in a 6 -18 month timeframe, which is the perfect scale for decision-making the existing humanitarian programme cycle.

The first stage

will be to design such a tournament, to be tested with 4–5 Start member organisations. It is likely to be more effective if it focuses on a single response, and might be incorporated into Start’s plans for country-level consortia. We would aim to identify 4–5 national and international staff from within each participating member to take part in the initial tournament round, preferably a combination of field, regional and HQ staff.

Participants will receive full briefings about forecasting, particularly how to improve their own forecasting and work with others in group forecasting (which has been shown to improve accuracy in most cases). They will also receive ongoing support in the form of online resources to help improve their skills, as the tournament hopes to identify people with both an interest in and aptitude for forecasting who could then receive additional support to improve their skills.

The initial tournament will last 3 months (including platform development and participant training), based on 12 forecasting questions that could focus on areas such as:

• Projected population flows under displacement scenarios
• Damage estimates following major earthquakes
• Likely duration of long-term crises such as droughts
• Possible outcomes of conflict negotiations

The second stage

will expand:

1. The Tournaments to include a wider range of emergency responses, creating national tournaments in multiple countries either in crisis or at risk. National tournaments could be aggregated to address regional issues (e.g. the Syrian war or the European migrant crisis), and a linked global tournament could potentially address high-level trends such as aid flows and policy debates.
2. The Participants to include forecasters from outside the sector who would be interested to participate in the tournament as a way of contributing towards better responses. This expanded tournament will make it possible to test to what extent mixed forecasting groups (i.e. humanitarian professionals and non-professional forecasters) can achieve better results than solely humanitarian participants.
3. The Audience to include decision-makers within the sector who might benefit from better forecasts. This will require monitoring and evaluation to establish to what extent such forecasts improve the quality of decisions, and gain greater insight into what organisational processes are needed to embed this type of evidence-based analysis into the sector. This will also act as proof of concept for further investment in forecasting in the sector.

The third stage

will encourage humanitarian organisations not just to use existing forecasts provided by third-party organisations, but to promote forecasting within their own organisation. This should not be limited to data or policy units, but encouraged throughout the organisation to improve the quality of decision-making more generally; it will be especially important to incorporate into monitoring, evaluation and learning processes.

Positioning the Tournament

The tournament should be positioned as complementary to other future-oriented activities. The Start Network is involved in a number of such activities, including pilots based around insurance mechanisms and future roundtables that encourage encourage discussion about the long-term development of the sector. Partnerships should be encouraged with other platforms in this space, along three main axes:

• Platforms that collate primary and secondary data to provide situational awareness, e.g. ACAPS GEO. Launched in October 2013 by ACAPS, this provides an “easily accessible, updated snapshot of natural disasters and complex emergencies at a global level… combined with a more in-depth analysis” and has become a critical part of the humanitarian information ecosystem. ACAPS has used the GEO data to identify long-term trends and build scenarios in its Crisis Overview 2015: Humanitarian Trends and Risks for 2016. [pdf]
• Platforms that aggregate analytical material to support better decision-making, e.g. INFORM. This was launched in November 2014 by a coalition led by the Inter-Agency Standing Committee Task Team on Preparedness and Resilience and the European Commission. It is a “global, open-source risk assessment for humanitarian crises and disasters” that provides shared evidence from a range of partners, but does not provide any kind of forecast based on its rankings.
• Platforms that use a similar approach to forecast in related areas, e.g. the Early Warning Project. This was launched in December 2015 by the United States Holocaust Memorial Museum. It is based on a combination of statistical risk assessment and an expert opinion pool (i.e. a tournament approach). The EWP aims to forecast mass atrocities as part of wider early warning in order to prevent such atrocities from occurring. This gives it a more tightly defined and overtly political aim than a Humanitarian Forecasting tournament would have.

Conclusion: can forecasting save lives?

Forecasting offers a tremendous opportunity for the humanitarian community to increase the timeliness and efficiency of its responses, leading to improved outcomes for disaster-affected communities. Existing approaches to preparedness and planning — particularly in key areas such as logistics and security — could be improved by better forecasting, and operational responses in fast-changing environments could be made more flexible.

Better forecasting can lead to better policy and planning; not just because it gives us more information about possible future developments, but because the type of thinking required for better forecasting encourages more rigorous thinking in general. Better forecasting will benefit specific organisations and the overall sector, contributing to the gradual improvement in their use of evidence and quality of analysis. Forecasting skills are possible to identify and can be developed using a tournament-style approach, leading to improved capacity within the sector on a more sustainable basis.

Although this approach may only result in incremental improvements in the effectiveness and efficiency of humanitarian action, even relatively minor improvements will result in more lives saved, more livelihoods preserved or restored, and a humanitarian sector that is better prepared to meet its responsibilities.