Choosing the best approach to ensure effective evaluation

Corruption is complex, entrenched, and clandestine. Evaluating anti-corruption interventions is therefore not easy. Key challenges include the difficulty of accurately measuring corruption and changes in corruption; the time it takes to achieve an impact; the difficulty of proving causation and contribution; and the likelihood of unintended consequences and backlashes.

This Issue aims to provide theme-specific advice for anyone commissioning or carrying out an evaluation that either exclusively or in part assesses anti-corruption interventions. The target audience is individuals familiar with evaluations but with limited expertise in corruption and anti-corruption measures. It may also prove insightful for anti-corruption experts with limited evaluation knowledge. The focus is on interventions that aim to reduce corruption in society – not corruption risk management – and, in particular, those that are donor-funded. However, most insights are generally applicable.

The paper is intended as a complement to evaluation handbooks. The headline messages will be obvious to most readers. Carefully gathering and making use of data is, for example, something every evaluator knows how to do. However, they may not know the specific data challenges with regard to corruption and anti-corruption initiatives – and how they can be overcome. Part One provides an introduction to corruption and anti-corruption measures; Part Two explores the implications for evaluations.

Since it is not intended as a stand-alone handbook, the Issue does not cover all of the messages emphasised in comprehensive evaluation handbooks. For a more overarching introduction to evaluations, see Sida’s 2020 Evaluation Handbook or UNODC’s 2017 Evaluation Handbook.

Part 1: Understanding corruption and anti-corruption efforts

In order to appreciate the theme-specific implications for evaluations, we must first understand the field. The U4 Issue Understanding corruption and how to curb it: A synthesis of latest thinking identifies ten main insights into corruption and the approaches to counter it. These can be re-summarised into the following six points:

1.1 There are many forms of corruption – the scale and types of corruption, as well as the actors involved and the level of harm done, vary by context.

Corruption is employed by a multitude of actors in a multitude of circumstances and fora – from a politician receiving kickbacks from companies that have been awarded large contracts, to a teacher eliciting sex in return for good grades. Common forms of corruption include bribery, trading in influence, embezzlement, fraud, kickbacks, extortion, nepotism, clientelism, abuse of discretion, and graft. Determining which forms of corruption are most harmful is context specific. In some cases, many small acts of corruption are more harmful than one big act. In other cases, the opposite is true. The prevalence of each form of corruption varies both across and within countries – including by region, sector, and institution.

Corruption takes place within the political realm, the public sector, and the private sector – as well as intersections between the three. Corruption can also be trans-national and enabled by external factors such as tax havens and foreign policy objectives, including regional security, trade relations, and diplomacy. The ways in which external factors can enable corruption is clearly illustrated by Chayes’ case study of the structure of corruption in Azerbaijan.

In some contexts, corruption is an anomaly – the act of a few individuals. However, in many contexts, corruption is widespread. In its extreme, corruption is an integrated part of the socio-political and economic system. For a visual depiction of how complex and entrenched a system of corruption can be, see the Justice and Legitimacy Program’s systems map of the dynamics driving police and judicial corruption in northern Uganda.

1.2 There are many anti-corruption tools, the details of which matter

There are numerous types of anti-corruption interventions. They can be global, regional, national, subnational, sectoral, or institution specific. They can be stand-alone or integrated into a larger programme. They can be explicit or implicit; internal or external; direct or indirect. Many are not specifically designed with the prevention of corruption in mind, but rather for broader goals such as good governance or civil society strengthening.

Examples of common interventions that are funded in anti-corruption programmes include:

• Building the capacity of the police and judiciary
• Developing an e-procurement system
• Training investigative journalists
• Establishing, training, and funding an anti-corruption agency (ACA)
• Running anti-corruption advocacy campaigns
• Creating a community monitoring mechanism for a large health programme

Several studies suggest that many conventional anti-corruption tools are ineffective.1902a8163e23 However, a specific tool cannot be definitively labelled as being effective or ineffective – the details matter. Anti-corruption agencies, for example, vary in mandate, manpower, and degree of independence. Some have investigative and prosecutorial powers, while others do not. Some are spearheaded by strong individuals willing and able to indict politicians and their families, while others are not. These variations are key determinates of the likely impact that an ACA can have.07cd017386d1 The context in which the tool is implemented – including the political settlement – also matters.

1.3 A unique combination of approaches, tools, and actors is needed to address the (often) multiple reinforcing drivers of corruption in a given context

Corruption is a complex phenomenon, often caused and sustained by multiple interlinked factors. These factors can be categorised into four drivers: principal–agent/institutional problems; collective action problems; justifying norms, values, and pressures; and the short-term functionality of corruption. These drivers are broad categories, and each requires further unpacking. A principal–agent problem could, for example, stem from a lack of transparency. This lack of transparency could, in turn, arise from the absence of an operational freedom of information law.

Anti-corruption interventions need to address the underlying drivers of corruption, not just the symptoms. In the past, the focus has largely been on ‘technical’ top-down reforms designed to address the principal–agent drivers of corruption.9544273a8f06 However, this approach is often insufficient. It can lead to a situation where, for example, income and asset declaration rules are put in place, but few people comply with them as they do not feel compelled to do so.

A problem-focused approach to understanding and addressing corruption requires exploring all the relevant stakeholders and drivers, as well as the wider socio-political and economic context. In any given context, the strength and combination of drivers, as well as the actors involved, will vary. Even when two countries, regions, sectors, or institutions appear to have the same corruption problem, the exact cause of it, as well as what solutions are feasible, will be circumstance specific. Effective tools that can be employed for such an exercise include Wedel’s Mapping Method and Marquette and Peiffer’s Corruption Functionality Framework.

When multiple drivers of a corrupt act are present, a multi-pronged approach is likely needed. Small, shallow, and isolated anti-corruption interventions are less likely to contribute to a sustained impact.

1.4 When corruption is systemic, anti-corruption efforts need to take a systems approach

In some contexts, corruption is not only widespread, but also deeply entrenched. This phenomenon is often referred to as ‘systemic corruption’ or ‘systems of corruption’. The U4 Glossary defines systemic corruption as: ‘A situation when corruption is an integral part of a state’s economic, social and political system, and where most people have no alternatives to dealing with corrupt officials.’ Jackson, Tobin and Eggertb3a5ef0dd278 similarly state that: ‘Systemic corruption exists when a corrupt act recurs consistently and is connected to other corrupt acts through an underlying system that enables and encourages the corruption.’

While our understanding of how to address systemic corruption is still limited, it is clear that a different approach is likely needed. When corruption is systemic, anti-corruption interventions need to target the underlying drivers and the system itself, rather than symptoms or corrupt individuals. Convicting a powerful individual of corruption can be effective, if that sends a clear (and accurate) signal that the ‘rules of the game’ are changing. However, often the individuals that replace those that have been fired, transferred, convicted, etc due to corruption are (or become) corrupt themselves. Moreover, not everyone that engages in corruption is in favour of corruption. Many wish that the system would change but engage in corruption because of pressure or necessity. Construction contractors may, for example, risk bankruptcy and physical attack by their peers if they refuse to participate in well-established bid-rigging protocol.

When corruption is systemic, it can act like an equilibrium that is difficult to pierce. Anti-corruption efforts are most likely to be successful if they are (collectively) sufficient to bring about a new equilibrium. If they do not, the marginal effects of each intervention may diminish over time as vested interests find ways of undermining them and returning to the original equilibrium.f07eafe32204 There can be value in interventions that ‘lay the foundation’ for future reform efforts in contexts where the political settlement is not ready for a major anti-corruption drive. However, these interventions should be based on a strong theory of change and be in line with latest (and local) thinking on what interventions make sense given the context.8c70c15a1928 If not, they may do more harm than good.

For an overview of latest corruption theories and thinking, see Jackson.bafd9c4a92c9

1.5 Anti-corruption interventions need to be flexible, politically responsive, and designed with potential backlashes in mind

Corruption is not a disease or deviation, but the historical baseline. Curbing corruption is therefore a complex, never-ending process. Progress is neither linear nor one-way. It is messy and uncertain. Even incremental improvements and smaller reforms can be difficult to bring about and sustain – especially within the timeframes of most development assistance projects.

Corruption and anti-corruption interventions are both political. Powerful and influential individuals and groups often have a vested interest in maintaining the status quo. The potential for anti-corruption interventions to encounter pushback and backlashes, or to bring about negative unintended consequences, is therefore ever present. Indeed, Fisman and Golden11419346592e go as far as to state that ‘every anticorruption or antifraud program elicits a strategic response by those who orchestrated and benefited from wrongdoing in the first place.’

The following tables shows examples of potential negative unintended consequences and backlashes.

The likelihood of unintended consequences and backlashes can be mitigated by designing anti-corruption interventions based on a strong theory of change and a clear understanding of the political economy and political settlement. Key questions that need to be answered include: ‘Which powerful stakeholders are for and against reform and why?’ and ‘How might the system “push back” against efforts at reform?’.b5f581233198

Many academics and practitioners believe that anti-corruption interventions should target corruption problems that are feasible to address, given the political settlement.46f3d42f2bc9 Approaches such as taking advantage of windows of opportunity, building political will and trust, changing expectations, collaborating and coordinating with others, proactively working to maintain momentum, and continuously monitoring and adjusting interventions also show promise.6e4f6641f072

Finally, the level and genuineness of stakeholders’ commitment to anti-corruption efforts should not be taken at face value. Politicians may, for example, need to appear to be making progress on good governance in order to appease voters or receive international aid.

1.6 There are limits to what anti-corruption interventions alone can achieve, as well as the role that donor agencies can play, so successful anti-corruption efforts may require a broader approach

There are limits to what even the perfectly designed and implemented anti-corruption interventions can achieve. Training investigative journalists to uncover corruption, for example, can lead to an increase in acts of corruption being brought to light. However, whether this will lead to an increase in arrests, prosecutions, and convictions depends on a range of factors beyond the control of that intervention.

Similarly, there are limits to the role that external actors can play. Countries that have significantly reduced their corruption levels have primarily done so due to internal forces. Klitgaard18333e32cf66 finds that ‘providers of development assistance can contribute resources, knowledge, convening power and leverage that may help recipient countries reform.’ However, political will and bottom-up pressure are essential. Masonb3dce77dffe3 believes that the role of external pressure may also be enhanced if countries employ a whole-of-government approach, rather than limit anti-corruption measures to development assistance.

Part 2: Implications for anti-corruption evaluations

The nature of corruption and the current approach to anti-corruption initiatives have a number of implications for evaluations – from the way data are gathered to the way sustainability is assessed. The remainder of this Issue aims to explain these theme-specific challenges and offer potential ways to overcome them. While the general messages will be familiar to most readers, the specific issues relating to evaluating anti-corruption interventions may not.

In addition to these guidelines, evaluations should adhere to evolving standards of best practice. This entails, amongst other things, carrying out evaluations that are:

• Timely, useful, and usedceeb555ff5da
• Clear, focused, and tailored
• Genuinely142593e7b846 participatory3e135e0832d0
• Gender-, human rights-, and sustainability-responsive60e18006e70a
• In adherence with quality and ethical standards, as well as aid effectiveness principles45ab288bd54e

Many evaluations are commissioned and organised in accordance with the six OECD/DAC evaluation criteria. The following table summarises the applicability of the paper’s main messages to each of these. It can be used to guide readers to the sub-sections most relevant for their needs.

2.1 Consider the need for a complexity-responsive evaluation methodology

As explored in Part One, corruption and measures to curb it are often complex. Bamberger, Vaessen, and Raimondof67d07224fc9 identify five, often interlinked, dimensions of potential complexity in development evaluations: 1) the intervention; 2) the institutions and stakeholders involved in the intervention; 3) the effects of the intervention – including the causal change processes; 4) the context in which the intervention is implemented; and 5) the evaluation itself.

The extent to which complexity needs to be considered when evaluating anti-corruption interventions will vary. An intervention may be more complex in some dimensions than in others. A checklist for assessing the level and areas of complexity has been developed by Bamberger, Vaessen, and Raimondo.49d4fed21c6b The following table adapts this checklist in order to highlight areas of potential high complexity.

When a complexity-responsive evaluation is deemed necessary, evaluators may need to complement established evaluation approaches with emerging methods – such as methods from complexity science or innovative data-collection techniques.0e916dd83113 Several potential methods are already becoming more mainstream. These methods include outcome harvesting, qualitative comparative analysis, process tracing, contribution analysis, social network analysis, systems mapping, and appreciative enquiry.

For an overview of complex collection and analysis methodologies that may be relevant when evaluating anti-corruption interventions, see INTRAC.c3f844fc5e44 Similarly, for a framework for determining which impact evaluation methods are most appropriate when evaluating the complex, see section 2.9.

It may also prove useful to consider evaluation criteria beyond those defined by OECD/DAC, as shown in Box 1.

2.2 Carefully gather and make use of data

Evaluators of anti-corruption interventions often face four data challenges: getting accurate data; getting intervention-specific data; getting sufficient data; and securing data in an ethically responsible and safe manner.

Corruption is clandestine and illegal. It is therefore difficult to measure fully and directly.14ff439bb09a As a result, evaluators must often rely on imperfect estimates such as proxy indicators, perception surveys, and expert opinion. At best, these sources of data provide a realistic estimate of corruption and its consequences. However, their findings can also be misleading. For example, a crackdown on corruption can result in the public becoming more aware of the extent of corruption. This can lead to a worsening in perception-based corruption survey results and, potentially, public apathy towards corruption.

The available data may also not be intervention specific. Many interventions’ logframes and monitoring systems rely in part on secondary data. These data are often too aggregated to be attributable to the intervention itself or to capture small changes. Global indexes that can be disaggregated by type of corruption, sector, or institution are generally preferable to global indexes with no basis for disaggregation. However, even these are often beyond the intervention’s sphere of influence, as further explained in section 2.4 on attribution. Locally generated data, such as administrative statistics and target surveys, are more likely to reflect a specific intervention’s effects.d4f90250cea4

While programme-specific indicators are the best gauge of an intervention’s performance, not all anti-corruption interventions have sufficient baseline and monitoring data.45111b67d01d This is particularly common when the anti-corruption intervention is implicit or mainstreamed. Because anti-corruption efforts are political in nature, the actors behind the intervention may choose a clandestine approach and decide not to rigorously document results. As Robillard and Robillard519aecd4bd78 point out, making an anti-corruption intervention explicit ‘makes it easy for people in power to applaud these initiatives in public – and to avoid them, or even undermine them, in private. By the time the project reports are written, the systems that facilitate corruption will have shifted, adapted, and survived.’

The political nature of corruption also means that stakeholders may be unwilling to participate in interviews or other data-gathering methods. Even if they do, they may hold back useful information for fear of sanctions or retaliation.ab320f81b4b3 Such concerns are justified, as whistleblowers, journalists, and reformists alike have lost their jobs, been arrested, had their reputations destroyed, been threatened, and – in some cases – even killed.1349176ccb1a Gaining access to, and eliciting honest answers from, the individuals facilitating and engaging in corruption can also be both difficult and ethically complex.

A final challenge is that there is no universally agreed definition of corruption.20f158cacf88 While studies have found that people’s view of what corruption comprises is relatively similar across societies,e182d5bb8ed8 there are differences in opinion. Small informal payments may not always be viewed as a bribe. Similarly, corruption that benefits the community or family can be considered morally legitimate.47ced524716d

In response to these challenges, the commissioners of anti-corruption evaluations should:

• Determine the evaluability of the intervention at the outsetea4e978d2c35
• Select a diverse evaluation team that brings together evaluation, political economy, thematic, and local expertise, as well as access to stakeholders
• Consider any biases, pressures, or repercussions that they or the evaluation team may face

Similarly, evaluators should:

• Prioritise the safety and comfort of evaluation participants
• Carefully choose their words to ensure clarity and avoid unnecessary controversy
• Use data-gathering methods that allow for more open and honest input (eg anonymous surveys, informal talks, vignettes, etc)
• Tailor the choice of data-collection approach to the type of corruption being discussed and the stakeholder being questioned3c84a1a841f4
• Gather data from a range of stakeholders with different perspectives
• Base their analysis on primary and secondary data that are attributable to the intervention
• Use a mixed-method approach and triangulate findings
• Consider the quality of the various data sources599309851517

Evaluators should also take into account standard data-collection issues, such as how to overcome social desirability bias and discern perceptions from reality.84ed47ecb09f The team should also disaggregate data by gender, poverty, and other relevant socioeconomic and demographic characteristics as corruption, and measures to counter it, can impact – directly and indirectly – groups in different ways.ce6d023d8f59

For general advice on how to address data constraints, including missing or difficult to collect data, see chapter five of the third edition (2020) of Bamberger and Mabry’s book RealWorld evaluation: Working under budget, time, data, and political constraints. For an assessment of the strengths and weakness of potential corruption measurement tools see Hart;1e9f4b3f8bae Trapnell, Jenkins and Chêne;9fcffa755b5b and Trapnell.6e7dbb5bf19a For further details on data generation tools and approaches that are well suited to sensitive topics such as corruption, see Scharbatke-Church,a4279bd5d5dc as well as the section on methods in Jackson and Köbis,d08f8b41355c pages 26–29 of Scharbatke-Church with Barnard-Webster,74e33379dcc8 and sections 3.1.2 and 3.1.3 of the Council of Europe.3d53852eaa4f For insights on how to measure changes in social norms, see Scharbatke-Church and Kothari.d1ac79c2b635 See also Box 2 for lessons learned from studying corruption in the criminal justice system.

2.3 Employ a realistic barometer of intervention success – even if this means questioning the logframe

As discussed in section 2.2, anti-corruption interventions do not always have clearly stated goals, objectives, and indicators with accompanying baseline and monitoring data. Even when they do, these indicators are not always a fair barometer of success. Some anti-corruption interventions have logframes with indicators outside their sphere of control and influence.5d1129ff38d7 A country’s score or rank on the Transparency International Corruption Perceptions Index is, for example, not an accurate gauge of the impact of establishing and building the capacity of an anti-corruption agency, regardless of how effective the agency may be. Less ambitious indicators – such as the number of prosecutions – are also often dependent on a range of factors beyond an intervention’s sphere of control or influence.

When assessing effectiveness and impact, evaluators must keep in mind that even small changes in corruption levels are difficult to bring about and sustain. The impact of most interventions will be modest in scale9fb3eb9b0ae3 and a project cycle is often too short a time period to bring about significant and lasting change.

An assessment of effectiveness and impact also needs to consider the context. It is, for example, more difficult to reduce corruption in fragile states and to address corruption that is 8a33ce5b8dc2 In contexts where anti-corruption efforts are being actively undermined, preventing an increase in corruption or holding on to past gains may in itself be considered a success.

2.4 When attribution cannot be determined, aim to demonstrate contribution

Demonstrating the causal link between a specific anti-corruption intervention and concrete outcomes and impacts can prove difficult – particularly at the higher level.66d099108fd6 First, multiple donors and interventions may be supporting the same target unit (eg the judiciary). Second, change is often brought about by a multitude of past and present forces. Third, there are often data limitations.In many cases, it is therefore appropriate to aim to demonstrate contribution rather than attribution.

Imagine, for example, that leaders of a corruption scheme are found out, prosecuted, and convicted. This ‘impact’ is potentially the result of training investigative journalists (who first brought the case to light); getting a freedom of information law passed (so that the journalists were able to access evidence); providing the police, prosecutors, and judges with capacity building (so that a solid case could be built and fairly tried); a newly elected prime minister wishing to demonstrate that she is tough on crime (ensuring the necessary political will); and corruption awareness campaigns (creating mounting pressure to act).

There are four approaches to establishing a causal inference: counterfactual frameworks; regularity frameworks; configurational frameworks; and generative frameworks.4ac1fff2c66c Counterfactual methods, such as randomised control trials, are often touted as the preferred approach. However, the value of other approaches is increasingly recognised. Specific methods include RAPID Outcome Assessment (ROA), episode studies, and contribution analysis. The first two methods begin with the change itself and then map the process backwards to determine causal factors. This can be an effective way to ensure that the importance of the intervention is not overestimated. Triangulation can also help overcome attribution and contribution challenges.3789ae398165

2.5 Assess the extent to which there is a problem–strategy match

The OECD/DAC criteria relevance does more than determine whether an intervention is in line with local and global needs, priorities, and policies. It aids in recognising the likelihood of effectiveness, impact, and sustainability. The OECD guidance states that the term ‘responds to’ in the definition of relevance includes whether ‘the objectives and design of the intervention are sensitive to the economic, environmental, equity, social, political economy and capacity conditions in which it takes place.’

An intervention can address the ‘right’ problem, but in the ‘wrong’ way. By analysing whether there is a problem–strategy match, evaluators can get an indication of how likely an anti-corruption intervention is to achieve results and be sustainable, as well as avoid negative unintended consequences (a topic explored in section 2.7). Unfortunately, anti-corruption interventions often lack an explicit and clear theory of change.6096e3f4739c However, when they do, evaluators can compare this theory of change to the situational analysis, including any recent updates. They can also work with stakeholders to retroactively create a theory of change.

A key question to explore is whether the intervention was appropriate, given the context. Even when the same corruption problem appears in two different contexts, the same solution may not work in both, as the underlying drivers and wider conditions will vary. Hanna et al.,7107bdc46409 for example, find that community-level monitoring can be an effective anti-corruption tool, but only when the community is able to punish corruption. Such insights have implications both for maximising results and doing no harm. As Johnston states, ‘What might seem to be a good reform idea in country A may well be impossible in B, irrelevant in C, and downright harmful in D’.e928efcfd004

Another important question is whether the intervention, either on its own or in combination with other efforts, is sufficient to address the problem – and, ideally, pierce the corruption equilibrium. Why? Because small, shallow, and isolated anti-corruption interventions are less likely to contribute to significant changes and more likely to have their effect diminished over time.

2.6 Determine whether the intervention is in line with latest and local thinking on corruption and anti-corruption efforts

Relevance, effectiveness, impact, and sustainability are in part determined by whether the intervention was (and remains) designed and implemented in line with latest and local thinking on corruption and anti-corruption measures. Three decades of anti-corruption efforts have not delivered the anticipated level of results.9952f3c3392c There are many justifiable reasons for this – including the difficulty of combating corruption. However, the experience to date can in some degree be attributable to weaknesses in conventional anti-corruption approaches.

Part One of this Issue summarises many of the key lessons learned. The reader is particularly encouraged to familiarise themselves with sub-sections 1.3, 1.4, and 1.5.

Other anti-corruption resources that can be drawn on to determine whether interventions are likely to succeed include: Jackson 2020; World Bank 2020; Mungiu-Pippidi 2017; Rocha Menocal et al. 2015; Johnsøn, Taxell, and Zaum 2012; Hanna et al. 2011; and Anti-Corruption Evidence (ACE) Research Consortium 2019.

Some of these publications focus exclusively on the overarching principles – such as the need to address all drivers of corruption, and not just principal–agent problems. Others assess the evidence base for specific anti-corruption tools. While evidence on ‘what works’ is limited, it is growing. For example, Johnsøn, Taxell, and Zaume939ae5d667c and Rocha Menocal et al.af130a874987 find that public financial management can be an effective anti-corruption tool. Similarly, although universally applicable formulas can never be created, evidence is emerging on potentially complementary approaches. For example, Mungiu-Pippidi and Dadašov74578bd91c50 find that anti-corruption legislation is more effective when there is freedom of the press and independence of the media.

While best practice is one important source of knowledge, it is not the only source. Local knowledge on what has worked, what has not worked, what is feasible, and why, is equally insightful. See section 4 of the Problem-Driven Iterative Adaptation (PDIA) toolkit by the Building State Capability Program for details.

2.7 Explore potential unintended consequences and the likelihood of a backlash

As discussed in section 1.5, the likelihood of anti-corruption interventions bringing about unintended consequences and backlashes being experienced is high. Unintended consequences can be positive or negative. Similarly, a backlash can be immediate or years in the making. The Indonesian Corruption Eradication Commission (KPK), for example, successfully staved off efforts to undermine its credibility, independence, and effectiveness for over 20 years. However, in 2019 the House of Representatives quickly pushed through legislation that significantly reduced the agency’s autonomy.04019afd3999

Evaluators should proactively seek to identify unintended consequences and backlashes that have already transpired or are likely to happen in the future. Potential ways of doing so include:

• Assessing the extent to which there is a problem–strategy match (see section 2.5)
• Speaking to third party experts with a deep understanding of the local context and ‘how things work’ – eg grassroots leaders, anthropologists, etc.
• Selecting evaluation tools designed to uncover unexpected results – eg Most Significant Change (MSC) stories and tracer studies
• Exploring the current political settlement, including whether there is sufficient political will to sustain reform efforts (see section 2.8)

For additional guidance on how to incorporate a focus on unintended consequences into an evaluation, see Bamberger, Tarsilla and Hess-Biber.0bba903f4fc6

2.8 Draw on all of these insights, and more, to assess sustainability

According to the OECD/DAC revised evaluation criteria, an assessment of sustainability ‘[i]ncludes an examination of the financial, economic, social, environmental, and institutional capacities of the systems needed to sustain net benefits over time. [… It involves] analyses of resilience, risks, and potential trade-offs.’

Like all interventions, an analysis of the sustainability of an anti-corruption intervention entails asking questions such as ‘Does the intervention have sufficient funds to continue?’ and ‘Are the stakeholders involved in the intervention interested in continuing to be active?’ Many community monitoring groups, for example, become inactive once funding for the programme that supported them comes to an end.

However, there is also another level of questions that need to be answered when assessing how likely results are to be sustained in the long term. For example:

• Did the intervention address the underlying drivers of corruption?
• Has the gap between the formal and informal rules of the game been closed?d1b093755016
• Is the intervention backed by powerful and influential stakeholders?58e57f52cec0
• Does the political settlement support these reforms?d901da8bb04c
• Has the intervention (in combination with other reform efforts) succeeded in shifting the landscape to a new corruption equilibrium?5fc7e9c1add1

For further details, see section 2.5 on the need for a problem–strategy match and section 2.7 on backlashes. For additional guidance on how to incorporate sustainability questions into evaluations see Chelimsky.d56513f47bc3

2.9 Determine impact and ‘what works’, using appropriate anti-corruption methods

Additional studies that rigorously assess impact can help fill the anti-corruption evidence gap, as there is insufficient academic research on ‘what works and why’.9c1a416a4192 If an evaluation is going to assess impact, it should do so thoroughly. A decade ago, few evaluations of anti-corruption interventions focused on impact. Those that did generally made use of insufficiently rigorous methodologies.4e0740c21fa9 As background research for this Issue, we reviewed 37 anti-corruption and good governance evaluations conducted between 2014 and 2020. While 16 of the 37 evaluations claim to assess impact, very few do. Most of the methodologies utilised are also not suitable for measuring impact. These findings are in line with the results of the industry-wide consultation survey carried out as part of the OECD/DAC criteria revision process: only 34% of respondents were satisfied with the way the ‘impact’ criterion is implemented – compared to a satisfaction rate of over 75% for the ‘relevance’ and ‘effectiveness’ criteria.5a17bbfb0df3

When evaluating the impact of an intervention, it is important to be issues-driven, rather than methods-driven.29ded08e6f84 The evaluation approach, methodology, and techniques should be guided by the overall purpose and scope of the evaluation; the questions that need to be answered; the intervention’s attributes; the context; the availability of data; and the allotted budget, time, and human resources. With regards to the field-specific considerations, Wathne110a494dc24b states: ‘Rather than a binary analysis of their impact – as effective or not – we need to understand whether, why, to what extent, under what circumstances, in which contexts, in which combinations, and for whom anti-corruption efforts have a direct or indirect impact on corruption levels and, ultimately, on development outcomes.’

This statement acknowledges that:

• What works in one situation, may not work in another.
• There are multiple pathways to bringing about a reduction in corruption.
• A specific anti-corruption tool is unlikely to be sufficient on its own. A multi-pronged approach is generally needed. Using logic terminology, the implied impact question becomes: ‘Which anti-corruption tools are “necessary but not sufficient” vs. “neither sufficient nor necessary”?’
• Quantifying the impact of anti-corruption interventions is challenging, given the difficulty of measuring corruption and changes in corruptiona6c2e86ac18c as well as determining an intervention’s level of contribution.
• Partial successes and limited results are often all that are feasible within a given budget, time frame, or context.
• Determining causality is difficult.69952165df42
• Progress is not linear. Long periods with no results can be followed by rapid change, and setbacks are likely. Gauging the likely impact trajectory over time may therefore be as important as documenting the level of impact at a given point in time.0971983a1a76
• Interventions can be multi-pronged and bring about multiple results; bring about different results for different stakeholders;6267d3563e69 and lead to unanticipated results and consequences.
• The details of a specific anti-corruption tool matter. As explored in section 1.2, anti-corruption agencies vary considerably in terms of their mandate, manpower, degree of independence, and the context in which they operate. This variation goes a long way to explaining why some ACAs have been more effective than others.

Evaluators need to design an impact evaluation that considers the above realities of corruption and anti-corruption measures. The ability of each evaluation method to gauge impact in the context of complexity can be assessed along six dimensions:e45e2777368e

• Attribution: Documenting changes and attributing them to the intervention
• Explanation: Showing how the intervention worked and how it affected change
• Multiple causal pathways: Exploring multiple changes and/or the confluence of factors impacting change
• Nature of causal change: Assessing change that is uncertain, non-linear, and emergent
• Emergence: Accommodating both the intervention adapting over time and the wider context changing over time
• Scope of effects: Capturing all effects, including unintended outcomes

A number of evaluation handbooks have clear statements regarding which methods are best when determining impact and ‘what works’. Many of these handbooks favour quantitative methods. The World Bank/IDB 2016 handbook, Impact Evaluation in Practice, for example, includes five chapters on quantitative design approaches. Reference to qualitative approaches is limited to a sub-section on complementary approaches, where they are referred to as valuable tools for supplementing findings. Similarly, 3ie’s ‘Impact evaluation glossary’ defines an impact evaluation as having ‘either an experimental or quasi-experimental design.’ The glossary goes on to define a small ‘n’ impact evaluation as ‘the set of best available methods when “n” is too small to apply statistical approaches to constructing a counterfactual.’ However, qualitative approaches are not just a means of supplementing and triangulating quantitative findings. They are important approaches in their own right. Qualitative approaches are, for example, generally better placed to explore unintended consequences.96d42ab85eb1

For an overview of qualitative methods for assessing impact, see the DfID’s 2012 Working Paper. In it, the authors show how and when theory-based, case-based, and participatory approaches can be the preferred choice for an impact evaluation. For an overview of quantitative methods, see Johnsøn and Søreide1161c6f01f8f and Garcia.b1345a77d8bc

The international development network Bond has developed an open-access tool to help evaluators choose the right method. Examples of when various methods may be appropriate are also provided in Box 3. In most cases, a genuinely mixed methods approach is best0f7d294d0041 – a topic explored further in Box 4.

Two common challenges when evaluating anti-corruption interventions are the sample size and approach. Anti-corruption interventions can have a target unit ranging from one (eg increasing the integrity of an agency) to millions (eg running a national campaign to raise awareness of corruption). However, the ‘n size’ is often small. Interventions also often consist of multiple activities and contribute to multiple results. Assessing impact is still possible. Johnsøn and Søreidebf6089f0c21a propose three approaches to evaluating anti-corruption interventions with a small target unit and/or multiple components: employ theory-based evaluation tools; use a formative (real-time) evaluation; or break an intervention into its constituent parts and evaluate each work stream separately. In so doing, it is important to keep in mind that the overall impact of an intervention can be more than the sum of its parts. The process of reassembling must therefore capture interactions between the work streams.9804183af03e

The rise of big data, machine learning, and artificial intelligence may also open up new opportunities for evaluating impact. Governments are increasingly adopting e-government and open data projects. These data could potentially be analysed over time and across jurisdictions. See Berliner and Dupuy;17f3b8928a48 White;02e1b67ec925 van de Braak, Choenni and Bamberger;398c29803d14 and Leouze, Areias and Jackson38cbd77256f6 for details.

2.10 Assess effectiveness and impact at the appropriate level

In addition to deciding the most appropriate design and method, the level of results to be evaluated needs to be clear. Is it, for example, sufficient to determine changes in capacity or attitudes? Or should the evaluation aim to determine changes in practice, such as asset declarations being filled out and reviewed, or an increase in corruption convictions? Alternatively, should the evaluation determine whether and by how much the level of corruption has been reduced and whether this has caused a significant change in the lives of beneficiaries? An anti-corruption intervention in the education sector may, for example, aim to contribute to increasing students’ learning by ensuring that funds allocated for books, desks, and classrooms are properly spent and that teachers regularly show up and teach the full curriculum.

Alternatively (or additionally), the evaluation can focus on the extent to which underlying drivers of corruption have been addressed – whether, for example, incentives and social norms have changed. This distinction builds on material produced by the author, together with David Jackson, that suggests that anti-corruption interventions which target systems of corruption can theoretically have an impact at three levels: reducing or eliminating a specific corrupt practice; weakening the larger system of corruption; and contributing to nurturing deeper systems of accountability in a society.

The final two levels of impact are in line with current callsd59b5dc27258 for an evaluation approach that is appropriate for assessing transformational change and systems change. As Patton6e989bab7c70 says: ‘Increasing the number of beds in shelters for the homeless constitutes increased impact. Changing the housing system so that no one is homeless is transformational.’ Burnsbfbe082e7389 similarly points out that ‘If our aim is to create sustainable change, we have to show more than that an intervention created a change. We have to show that the system dynamic changed. [… It is therefore necessary to] focus our impact assessment on how the system dynamic has changed, not on what changes have taken place within the system dynamic.’

The revised OECD/DAC impact criteria acknowledge this necessity. A suggested evaluation question in the OECD guidance is therefore: ‘Is the intervention transformative – does it create enduring changes in norms – including gender norms – and systems, whether intended or not?’ The guidance goes on to state the following with regards to transformation: ‘The definition defines transformational change as “holistic and enduring changes in systems or norms”. Transformational change can be thought of as addressing root causes, or systemic drivers of poverty, inequalities, exclusion and environmental damage, and is recognised in the 2030 Agenda as necessary to achieving the sustainable development goals.’

A focus on transformational changes entails assessing significance, depth of change, and scale of change. For further details see Feinstein,4ba90be379d2 the Independent Evaluation Group at the World Bankf83ef2f84cdb and the Blue Marble Evaluation.

Filling the anti-corruption evidence gap – the benefits of effective evaluations

The findings from the 2021 Transparency International Corruption Perception Index are sobering: the global average score (43 out of 100 points) remains unchanged for the tenth consecutive year. While this aggregate figure masks local successes, the findings confirm that corruption remains widespread.

Evaluations can help policymakers and practitioners to better design and implement anti-corruption strategies and interventions, but only if the evaluations are both useful and used. This is by no means a revolutionary conclusion. Indeed, most monitoring and evaluation handbooks emphasise the importance of a system that facilitates both accountability and learning. Yet, in practice, the focus is often still on accountability. There is also still a tendency to highlight and share successes, while glossing over challenges and failures.

We need to raise the evaluation bar. This requires planning for evaluations from the outset so that there are sufficient data to carry out a quality evaluation; designing and undertaking evaluations using methods and tools that are tailored to studying the problem and the intervention (in this case, corruption and the measures to curb it); and, lastly, openly sharing and actively disseminating the findings from evaluations in order to facilitate learning and improve future policies and practice.