[This post is based on a presentation given by Cheryl Brown, for Ideas to Impact, at the Innovating in Development Learning Event in February 2017.]
Is it a fair game?
Before we talk about the risks of prizes, let’s warm ourselves up with a game. Imagine we’re travelling on a train together. I remember I’ve got a pack of cards in my handbag and ask you if you fancy playing a game to pass the time. After some lively rounds of snap, I suggest that you pick a card at random and if you get the Jack of Hearts, I’ll give you £60. But if you get any of the other 51 cards you’ll have to give me £1. As it happens you have a pound coin in your pocket – you owe it to someone you’re on your way to meet today, but…£60. Then you remember me saying earlier that I’m hopeless at the statistics side of evaluation. So, if we play, who’s taking the biggest risk? Me or you? Is it a fair game? How do you decide whether to play or not?
How should we tackle risk assessment when using prizes, specifically innovation inducement prizes for development? Some prizes are about recognising achievements and might have no money attached to them (e.g. Building Healthier Lives ) but Ideas to Impact is focusing on inducement prizes. These are intended to motivate people (often quite specific communities) to put in additional effort, to get specific results in return for the chance to win some cash. To win, they’re going to have to make a change, do something that the prize team wants them to do.
Testing inducement prizes
Inducement prizes in development aren’t new – a campaign to encourage teenagers to get vaccinated might use a prize draw to win a mobile phone, alongside other activities, to increase participation. Ideas to Impact is looking at innovation inducement prizes; can we incentivise these specific communities to use new or improved products, processes, technologies or services (that are at least new to them) in order to accelerate social change and poverty reduction? So back to vaccinating teenagers, you could run a two stage innovation inducement prize that asks Health Authorities to develop (stage 1) and run (stage 2) innovative campaigns that dramatically increase the number of teenagers being vaccinated in a fixed time period. The ones that meet the criteria and get the best results, win a prize; the other entrants get nothing. And if nobody achieves the goal, then the cash awards aren’t made.
So Ideas to Impact is testing whether (and under which circumstances) these kinds of prizes – innovation inducement prizes – can catalyse changes necessary to make progress in some tricky areas of development.
Can they prompt community based organisations in Nepal to speed up scaling up and expanding community driven Climate Change Adaptation?
In Kenya, can they encourage people to work with some of the poorest groups in society when developing climate change information services, so the information is actually based on what people like very small-scale farmers need and want, and we get these farmers actually using the services to make decisions?
Can they motivate local governments in Ghana to produce and start implementing good quality liquid waste management plans?
But hold on…
Using inducement prizes in developing countries…making change happen quicker than it would have done if you’d left things alone. That sounds risky. Influencing people to behave in ways they weren’t planning to beforehand, getting local governments and community based organisations to compete against each other and rewarding only some of the people who get results…that sounds a bit, well, dodgy .
If you’re going to use innovation inducement prizes for development, how do you predict what’s going to happen? How do you check that you’re catalysing the right change, and how do you pick up in advance where things might go awry? We’re at the point in the programme where the inducement prizes are starting to kick in – in Ghana, for example, local authorities who made it through to Stage 2 of the Sanitation Challenge for Ghana prize are now starting to implement their new liquid waste management plans. We’ll know more in a couple of years about the outcomes. But what we can share with you now is how Ideas to Impact has been tackling risk so far including: risk assessments, independent review panels, in-country research, and a pesky Evaluation Team that asks questions like “Why do you think this is going to happen if you do that? What does ‘use’ mean? What evidence have you got for those assumptions?”.
Let’s start with risk assessments: Through a variety of processes, brainstorming, document review, expert interviews and drawing on their own experience, the Prize Teams identified risks in each category for their prize . They then assessed each risk’s probability and potential impact using a rating scale of 0 to 5. If a risk got 3 out of 5 for probability and 4 out of 5 for impact, it had an overall score of 12 and the Prize Team would think about how they could reduce the probability of it occurring and bring the overall score down to one that is acceptable.The risks are checked regularly and new ones added, or existing ones revised.
For evaluators, the changes made each time are where the stories lie in the prizes. Sometimes the risk score goes up: E.g. one risk of prize design for the Sanitation Challenge for Ghana was making the timeline too short: doing this may not give your participants enough time to develop solutions, but by the time you know if this is the case (when the solutions are submitted), it’s too late. In March 2016, this risk was rated Green but by September 2016 it had been upgraded to Yellow because, as a result of political events and the need for a detailed end line evaluation for applicants in Stage 2, the implementation stage of the prize had been reduced to 18 months. The Prize Team has responded by bringing in support to the local government applicants through learning and practice workshops to increase the likelihood that participants can meet the deadline.
Risk assessment covers risks relating to achieving what the prizes set out to achieve. But there are two other important areas of risk that come up less often through this process: 1) risks taken by society (unintended consequences) and 2) risks taken by solvers (e.g. community groups in Nepal). The Dreampipe Challenge I was focused on incentivising attention on new financing arrangements to address non-revenue water (NRW): the risk assessment identified that the prize might be successfully awarded and yet low-income populations might still not benefit from NRW reduction strategies. The response from the team in this case was to introduce selection criteria that required contractual arrangements to include pro-poor provisions and reinvestment a substantial portion of the benefits from NRW reduction to benefit poor customers.
To bring in unintended consequences and other stakeholder’s perspectives, Ideas to Impact has used Independent Review Panels (experts who give the prizes’ design a really good grilling) to help the Prize Teams at the design stage. The reviewers for the Climate Information Prize, for example, asked:
If making use of climate information was not within the capacity of certain groups in society, could the prize inadvertently cement inequitable social relations?
How would the prize distinguish between apparent public-private-poor ‘triple wins’ and the reality of more powerful actors’ interests overriding the poorest in the distribution of costs and benefits?
In-country research trips were undertaken by the Prize Teams during the design phase to find out from stakeholders on the ground what they had missed. For example, the Sanitation Challenge for Ghana research made it clear to the Prize Team that they would have to make it clear in consultations that the Prize would not run like a performance grant, i.e. the rewards would not be allocated to all local government authorities that “performed well”.
Some of these risks we can think about it advance and prepare for, or have some “risk identifiers” – indicators that help us sense if the risk is happening - but some others we suspect may emerge as we go along. We need to spot them quickly, learn, adapt and share the lessons with other people.
The Evaluation Team is supporting Ideas to Impact to capture and share this learning about risks and ethics of prizes for development. There are several questions we are exploring in this area , and the two we’re currently focusing on are who’s taking the risk and how do we make risk assessment more objective?
Who’s taking the risk?
Donors, managers and partners of the prize still bear some financial risk even if the prize doesn’t make an award and the reputational risk of being associated with a prize is probably greater than with other funding approaches due to the increased media activity that is a key element of prizes.
Prize solvers are the people who have to put in their own personal or organisational resources into the work leading up to making an application so the bulk of the donor’s financial risk is transferred to them. Innovation inducement prizes are not subject to specific prize regulation, so solvers also have to accept the risk of mismanagement: prizes rewarding the “wrong” solvers or their Intellectual Property being stolen.
Communities or the wider society where the prizes make change happen bear the risk of unintended negative consequence but do they have enough information to make an informed choice about the risks involved? Do they even have a choice? Who has the right to decide about risk on behalf of other people?
How do we make risk assessment more objective?
Definitions of risk vary, from being the possibility that something unpleasant will happen to a definition that considers a risk to be a measurable uncertainty with a known probability . Risk scores for Ideas to Impact were based on informed judgement, with Prize Managers weighing up the information available to them about each risk before rating its probability and impact on scales of 0 to 5. Can we strengthen this and make it less subjective? Are there tools we could use to aid calculations of impact and probability? How could we use monitoring data to update these calculations? And where do you draw the line? If it is possible to calculate the probability, is it worth the extra effort when probability is not the same as reality – if you picked one card from a full pack 52 times, you might not get the Jack of Hearts, although you should in theory.
Ideas to Impact is keen to learn from other people who have been thinking about these questions. Social marketing, is one such sector that innovation prizes for development could draw on for guidance in understanding and managing the ethics of influencing behaviour for social change. For example, Professor Lynne Eagle’s Review of Ethics in Social Marketing strikes a chord with prizes:
“Interventions that are based on rewarding people financially, or by the provision of goods may also raise ethical concerns, given that rewarding people for adopting new, socially desirable behaviours is perceived as being akin to bribery. In addition, it, raises the question of whether the behaviour will be maintained in the long term.”
The draft code of conduct Eagle presents at the end of the report could also be a useful reference for those working in innovation prizes for development.
When we share our short discussion paper later this year detailing more of the risks we’ve identified before and during the prizes’ implementation, we hope you’ll join in with the conversation .
Cheryl Brown is the Evaluation & Learning Coordinator on Ideas to Impact and Itad Ltd, an innovative monitoring and evaluation consultancy.
Further reading on balancing the risks and benefits of inducement prizes - Getting the balance right for Inducement Prizes for Development by Jonathan Slater