By Salisha Chandra (@salisha_ww) Founding Member of KUAPO
Yes I know, those are a lot of long words beginning with the letter “c” in one sentence. But it is an important sentence and one that I hope pushes those who support conservation efforts to look deeper and further, beyond the rhetoric and the magniloquence that you maybe assailed with, to real, actual and measurable impacts.
Any conservation organisation worth its salt should have in place an evaluation framework that allows it to, as unbiasedly as possible, measure its progress towards the goals that it has set for itself. Whether the organisation is field-based or advocacy oriented, it would have adopted specific activities and outputs that it believes will result in the ultimate outcomes or goals. In the case of an advocacy-based strategy for instance: how do we know that it is actually effective – does it reach the right people, do those people have the power to make decisions and are they incentivised to make those decisions? How do we know that it is the conservation groups activities or outputs that have led to change? Would that change have happened anyway? To put it bluntly – “correlation is not causation” and that’s the the crux of today’s blog post.
At the heart of every evaluation framework is the “assumption” that if X had not done Y then Z would not have happened. To put it in less abstract terms, we could say for instance “if KUAPO had not marched on the streets on January 22nd, 2013 then the government would never have passed the wildlife act.” This is the counterfactual – a conditional statement that assumes what the outcome would have been had the activity we mentioned not taken place. In this case, the evaluation framework being used is a simple pre and post comparison that analyses outcomes prior to and subsequent to the introduction of the program. Unfortunately this assumption does not hold in the majority of cases.
So let’s examine this sample statement a little further – KUAPO marched and petitioned the government in the first quarter of 2013 . The government passed the Wildlife Act in December 2013. Did anything else change during that time period? Were there other conservation organisations lobbying for the same thing? Was the new Cabinet Secretary mandated to ensure that this bill finally passed through parliament? In other words, can we really and definitively say that the only reason the bill was passed is because KUAPO marched and if we hadn’t, the bill would still be pending? Of course not! In order to do this, we would need to statistically account for every other factor that could possibly affect the outcome. While monitoring changes in outcomes overtime is valuable, it does not allow us to determine conclusively whether – or by how much – a particular program contributed to that change because there are other time varying factors existing that are affecting the same outcome. And this leads us to the “counterfeit counterfactual” which basically means that the pre-program outcome is never a good estimate of the counterfactual (what would have happened were there no program).
Let’s take another example – the hunting debate – we talked about this in our previous blog post but bring it up again here because it is worth reiterating. Proponents of hunting always fall back on this statement -“Look at Kenya since it banned sport hunting, it has lost 70% of its wildlife”. Here the counterfactual is that if Kenya had not banned sport hunting, we would not have lost 70% of our wildlife. This naive argument suffers from the same issues we explained regarding KUAPO’s marches and the passing of the bill – the existence of time varying factors other than the banning of hunting that led to this decline e.g., population growth, price of ivory, poverty levels, climate change etc – the list is almost endless.
So you get the point right? Conservation and the counterfeit counterfactual share a long history and as supporters of conservation – we urge you to look beyond these superficial statements and ask yourselves – what is this program really doing? what is its true impact? and was it solely responsible for the change as it claims?
P.S. KUAPO is working towards building a framework based on our strategy that will allow us to measure what impact we are having. Stay tuned!