Hubbry Logo
Impact evaluationImpact evaluationMain
Open search
Impact evaluation
Community hub
Impact evaluation
logo
7 pages, 0 posts
0 subscribers
Be the first to start a discussion here.
Be the first to start a discussion here.
Impact evaluation
Impact evaluation
Impact evaluation
View on Wikipedia
from Wikipedia

Impact evaluation assesses the extent to which an intervention (such as a project, program or policy) is causally responsible for observed outcomes, intended and unintended.[1] In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants' well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, "a comparison between what actually happened and what would have happened in the absence of the intervention."[2] Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.[3]

Impact evaluation helps people answer key questions for evidence-based policy making: what works, what doesn't, where, why and for how much? It has received increasing attention in policy making in recent years in the context of both developed and developing countries.[4] It is an important component of the armory of evaluation tools and approaches and integral to global efforts to improve the effectiveness of aid delivery and public spending more generally in improving living standards. Originally more oriented towards evaluation of social sector programs in developing countries, notably conditional cash transfers, impact evaluation is now being increasingly applied in other areas such as agriculture, energy and transport.

Counterfactual evaluation designs

[edit]

Counterfactual analysis enables evaluators to attribute cause and effect between interventions and outcomes. The 'counterfactual' measures what would have happened to beneficiaries in the absence of the intervention, and impact is estimated by comparing counterfactual outcomes to those observed under the intervention. The key challenge in impact evaluation is that the counterfactual cannot be directly observed and must be approximated with reference to a comparison group. There are a range of accepted approaches to determining an appropriate comparison group for counterfactual analysis, using either prospective (ex ante) or retrospective (ex post) evaluation design. Prospective evaluations begin during the design phase of the intervention, involving collection of baseline and end-line data from intervention beneficiaries (the 'treatment group') and non-beneficiaries (the 'comparison group'); they may involve selection of individuals or communities into treatment and comparison groups. Retrospective evaluations are usually conducted after the implementation phase and may exploit existing survey data, although the best evaluations will collect data as close to baseline as possible, to ensure comparability of intervention and comparison groups.

There are five key principles relating to internal validity (study design) and external validity (generalizability) which rigorous impact evaluations should address: confounding factors, selection bias, spillover effects, contamination, and impact heterogeneity.[5]

  • Confounding occurs where certain factors, typically relating to socioeconomic status, are correlated with exposure to the intervention and, independent of exposure, are causally related to the outcome of interest. Confounding factors are therefore alternate explanations for an observed (possibly spurious) relationship between intervention and outcome.
  • Selection bias, a special case of confounding, occurs where intervention participants are non-randomly drawn from the beneficiary population, and the criteria determining selection are correlated with outcomes. Unobserved factors, which are associated with access to or participation in the intervention, and are causally related to the outcome of interest, may lead to a spurious relationship between intervention and outcome if unaccounted for. Self-selection occurs where, for example, more able or organized individuals or communities, who are more likely to have better outcomes of interest, are also more likely to participate in the intervention. Endogenous program selection occurs where individuals or communities are chosen to participate because they are seen to be more likely to benefit from the intervention. Ignoring confounding factors can lead to a problem of omitted variable bias. In the special case of selection bias, the endogeneity of the selection variables can cause simultaneity bias.
  • Spillover (referred to as contagion in the case of experimental evaluations) occurs when members of the comparison (control) group are affected by the intervention.
  • Contamination occurs when members of treatment and/or comparison groups have access to another intervention which also affects the outcome of interest.
  • Impact heterogeneity refers to differences in impact due by beneficiary type and context. High quality impact evaluations will assess the extent to which different groups (e.g., the disadvantaged) benefit from an intervention as well as the potential effect of context on impact. The degree that results are generalizable will determine the applicability of lessons learned for interventions in other contexts.

Impact evaluation designs are identified by the type of methods used to generate the counterfactual and can be broadly classified into three categories – experimental, quasi-experimental and non-experimental designs – that vary in feasibility, cost, involvement during design or after implementation phase of the intervention, and degree of selection bias. White (2006)[6] and Ravallion (2008)[7] discuss alternate Impact Evaluation approaches.

Experimental approaches

[edit]
Further information: Experimental design

Under experimental evaluations the treatment and comparison groups are selected randomly and isolated both from the intervention, as well as any interventions which may affect the outcome of interest. These evaluation designs are referred to as randomized control trials (RCTs). In experimental evaluations the comparison group is called a control group. When randomization is implemented over a sufficiently large sample with no contagion by the intervention, the only difference between treatment and control groups on average is that the latter does not receive the intervention. Random sample surveys, in which the sample for the evaluation is chosen randomly, should not be confused with experimental evaluation designs, which require the random assignment of the treatment.

The experimental approach is often held up as the 'gold standard' of evaluation. It is the only evaluation design which can conclusively account for selection bias in demonstrating a causal relationship between intervention and outcomes. Randomization and isolation from interventions might not be practicable in the realm of social policy and may be ethically difficult to defend,[8][9] although there may be opportunities to use natural experiments. Bamberger and White (2007)[10] highlight some of the limitations to applying RCTs to development interventions. Methodological critiques have been made by Scriven (2008)[11] on account of the biases introduced since social interventions cannot be fully blinded, and Deaton (2009)[12] has pointed out that in practice analysis of RCTs falls back on the regression-based approaches they seek to avoid and so are subject to the same potential biases. Other problems include the often heterogeneous and changing contexts of interventions, logistical and practical challenges, difficulties with monitoring service delivery, access to the intervention by the comparison group and changes in selection criteria and/or intervention over time. Thus, it is estimated that RCTs are only applicable to 5 percent of development finance.[10]

Randomised control trials (RCTs)

[edit]

RCTs are studies used to measure the effectiveness of a new intervention. They are unlikely to prove causality on their own, however randomisation reduces bias while providing a tool for examining cause-effect relationships.[13] RCTs rely on random assignment, meaning that that evaluation almost always has to be designed ex ante, as it is rare that the natural assignment of a project would be on a random basis.[14] When designing an RCT, there are five key questions that need to be asked: What treatment is being tested, how many treatment arms will there be, what will be the unit of assignment, how large of a sample is needed, how will the test be randomised.[14] A well conducted RCT will yield a credible estimate regarding the average treatment effect within one specific population or unit of assignment.[15] A drawback of RCTs is 'the transportation problem', outlining that what works within one population does not necessarily work within another population, meaning that the average treatment effect is not applicable across differing units of assignment.[15]

Natural experiments

[edit]

Natural experiments are used because these methods relax the inherent tension uncontrolled field and controlled laboratory data collection approaches.[16] Natural experiments leverage events outside the researchers' and subjects' control to address several threats to internal validity, minimising the chance of confounding elements, while sacrificing a few of the features of field data, such as more natural ranges of treatment effects and the presence of organically formed context.[16] A main problem with natural experiments is the issue of replicability. Laboratory work, when properly described and repeated, should be able to produce similar results. Due to the uniqueness of natural experiments, replication is often limited to analysis of alternate data from a similar event.[16]

Non-experimental approaches

[edit]

Quasi-experimental design

[edit]

Quasi-experimental approaches can remove bias arising from selection on observables and, where panel data are available, time invariant unobservables. Quasi-experimental methods include matching, differencing, instrumental variables and the pipeline approach; they are usually carried out by multivariate regression analysis.

If selection characteristics are known and observed, they can be controlled for to remove the bias. Matching involves comparing program participants with non-participants based on observed selection characteristics. Propensity score matching (PSM) uses a statistical model to calculate the probability of participating on the basis of a set of observable characteristics and matches participants and non-participants with similar probability scores. Regression discontinuity design exploits a decision rule as to who does and does not get the intervention to compare outcomes for those just either side of this cut-off.

Difference in differences or double differences, which use data collected at baseline and end-line for intervention and comparison groups, can be used to account for selection bias under the assumption that unobservable factors determining selection are fixed over time (time invariant). Difference in differences can also be applied to multiple time points and when an intervention is incrementally introduced in phases.[17]

Instrumental variables estimation accounts for selection bias by modelling participation using factors ('instruments') that are correlated with selection but not the outcome, thus isolating the aspects of program participation which can be treated as exogenous.

The pipeline approach (stepped-wedge design) uses beneficiaries already chosen to participate in a project at a later stage as the comparison group. The assumption is that as they have been selected to receive the intervention in the future they are similar to the treatment group, and therefore comparable in terms of outcome variables of interest. However, in practice, it cannot be guaranteed that treatment and comparison groups are comparable and some method of matching will need to be applied to verify comparability.

Non-experimental design

[edit]

Non-experimental impact evaluations are so-called because they do not involve a comparison group that does not have access to the intervention. The method used in non-experimental evaluation is to compare intervention groups before and after implementation of the intervention. Intervention interrupted time-series (ITS) evaluations require multiple data points on treated individuals before and after the intervention, while before versus after (or pre-test post-test) designs simply require a single data point before and after. Post-test analyses include data after the intervention from the intervention group only. Non-experimental designs are the weakest evaluation design, because to show a causal relationship between intervention and outcomes convincingly, the evaluation must demonstrate that any likely alternate explanations for the outcomes are irrelevant. However, there remain applications to which this design is relevant, for example, in calculating time-savings from an intervention which improves access to amenities. In addition, there may be cases where non-experimental designs are the only feasible impact evaluation design, such as universally implemented programmes or national policy reforms in which no isolated comparison groups are likely to exist.

Biases in estimating programme effects

[edit]

Randomized field experiments are the strongest research designs for assessing program impact. This particular research design is said to generally be the design of choice when it is feasible as it allows for a fair and accurate estimate of the program's actual effects (Rossi, Lipsey & Freeman, 2004).

With that said, randomized field experiments are not always feasible to carry out and in these situations there are alternative research designs that are at the disposal of an evaluator. The main problem though is that regardless of which design an evaluator chooses, they are prone to a common problem: Regardless of how well thought through or well implemented the design is, each design is subject to yielding biased estimates of the program effects. These biases play the role of exaggerating or diminishing program effects. Not only that, but the direction the bias may take cannot usually be known in advance (Rossi et al., 2004). These biases affect the interest of the stakeholder. Furthermore, it is possible that program participants are disadvantaged if the bias is in such a way that it contributes to making an ineffective or harmful program seem effective. There is also the possibility that a bias can make an effective program seem ineffective or even as far as harmful. This could possibly make the accomplishments of program seem small or even insignificant therefore forcing the personnel and even cause the program's sponsors to reduce or eliminate the funding for the program (Rossi et al., 2004).

It is safe to say that if an inadequate design yields bias, the stakeholders who are largely responsible for the funding of the program will be the ones most concerned; the results of the evaluation help the stakeholders decide whether or not to continue funding the program because the final decision lies with the funders and the sponsors. Not only are the stakeholders mostly concerned, but those taking part in the program or those the program is intended to positively affect will be affected by the design chosen and the outcome rendered by that chosen design. Therefore, the evaluator's concern is to minimize the amount of bias in the estimation of program effects (Rossi et al., 2004).

Biases are normally visible in two situations: when the measurement of the outcome with program exposure or the estimate of what the outcome would have been without the program exposure is higher or lower than the corresponding "true" value (p267). Unfortunately, not all forms of bias that may compromise impact assessment are obvious (Rossi et al., 2004).

The most common form of impact evaluation design is comparing two groups of individuals or other units, an intervention group that receives the program and a control group that does not. The estimate of program effect is then based on the difference between the groups on a suitable outcome measure (Rossi et al., 2004). The random assignment of individuals to program and control groups allows for making the assumption of continuing equivalence. Group comparisons that have not been formed through randomization are known as non-equivalent comparison designs (Rossi et al., 2004).

Selection bias

[edit]

When there is an absence of the assumption of equivalence, the difference in outcome between the groups that would have occurred regardless creates a form of bias in the estimate of program effects. This is known as selection bias (Rossi et al., 2004). It creates a threat to the validity of the program effect estimate in any impact assessment using a non-equivalent group comparison design and appears in situations where some process responsible for influences that are not fully known selects which individuals will be in which group instead of the assignment to groups being determined by pure chance (Rossi et al., 2004). This may be because of participant self-selection, or it may be because of program placement (placement bias).[18]

Selection bias can occur through natural or deliberate processes that cause a loss of outcome data for members of the intervention and control groups that have already been formed. This is known as attrition and it can come about in two ways (Rossi et al., 2004): targets drop out of the intervention or control group cannot be reached or targets refuse to co-operate in outcome measurement. Differential attrition is assumed when attrition occurs as a result of something either than explicit chance process (Rossi et al., 2004). This means that "those individuals that were from the intervention group whose outcome data are missing cannot be assumed to have the same outcome-relevant characteristics as those from the control group whose outcome data are missing" (Rossi et al., 2004, p271). However, random assignment designs are not safe from selection bias which is induced by attrition (Rossi et al., 2004).

Other forms of bias

[edit]

There are other factors that can be responsible for bias in the results of an impact assessment. These generally have to do with events or experiences other than receiving the program that occur during the intervention. These biases include secular trends, interfering events and maturation (Rossi et al., 2004).

[edit]

Secular trends can be defined as being relatively long-term trends in the community, region or country. These are also termed secular drift and may produce changes that enhance or mask the apparent effects of an intervention(Rossi et al., 2004). For example, when a community's birth rate is declining, a program to reduce fertility may appear effective because of bias stemming from that downward trend (Rossi et al., 2004, p273).

Interfering events

[edit]

Interfering events are similar to secular trends; in this case it is the short-term events that can produce changes that may introduce bias into estimates of program effect, such as a power outage disrupting communications or hampering the delivery of food supplements may interfere with a nutrition program (Rossi et al., 2004, p273).

Maturation

[edit]

Impact evaluation needs to accommodate the fact that natural maturational and developmental processes can produce considerable change independently of the program. Including these changes in the estimates of program effects would result in bias estimates. An example of this form of bias would be a program to improve preventative health practices among adults may seem ineffective because health generally declines with age (Rossi et al., 2004, p273).

"Careful maintenance of comparable circumstances for program and control groups between random assignment and outcome measurement should prevent bias from the influence of other differential experiences or events on the groups. If either of these conditions is absent from the design, there is potential for bias in the estimates of program effect" (Rossi et al., 2004, p274).

Estimation methods

[edit]

Estimation methods broadly follow evaluation designs. Different designs require different estimation methods to measure changes in well-being from the counterfactual. In experimental and quasi-experimental evaluation, the estimated impact of the intervention is calculated as the difference in mean outcomes between the treatment group (those receiving the intervention) and the control or comparison group (those who don't). This method is also called randomized control trials (RCT). According to an interview with Jim Rough, former representative of the American Evaluation Association, in the magazine D+C Development and Cooperation, this method does not work for complex, multilayer matters. The single difference estimator compares mean outcomes at end-line and is valid where treatment and control groups have the same outcome values at baseline. The difference-in-difference (or double difference) estimator calculates the difference in the change in the outcome over time for treatment and comparison groups, thus utilizing data collected at baseline for both groups and a second round of data collected at end-line, after implementation of the intervention, which may be years later.[19]

Impact Evaluations which have to compare average outcomes in the treatment group, irrespective of beneficiary participation (also referred to as 'compliance' or 'adherence'), to outcomes in the comparison group are referred to as intention-to-treat (ITT) analyses. Impact Evaluations which compare outcomes among beneficiaries who comply or adhere to the intervention in the treatment group to outcomes in the control group are referred to as treatment-on-the-treated (TOT) analyses. ITT therefore provides a lower-bound estimate of impact, but is arguably of greater policy relevance than TOT in the analysis of voluntary programs.[20]...yes

Debates

[edit]

While there is agreement on the importance of impact evaluation, and a consensus is emerging around the use of counterfactual evaluation methods, there has also been widespread debate in recent years on both the definition of impact evaluation and the use of appropriate methods (see White 2009[21] for an overview).

Definitions

[edit]

The International Initiative for Impact Evaluation (3ie) defines rigorous impact evaluations as: "analyses that measure the net change in outcomes for a particular group of people that can be attributed to a specific program using the best methodology available, feasible and appropriate to the evaluation question that is being investigated and to the specific context".[22]

According to the World Bank's DIME Initiative, "Impact evaluations compare the outcomes of a program against a counterfactual that shows what would have happened to beneficiaries without the program. Unlike other forms of evaluation, they permit the attribution of observed changes in outcomes to the program being evaluated by following experimental and quasi-experimental designs".[23]

Similarly, according to the US Environmental Protection Agency impact evaluation is a form of evaluation that assesses the net effect of a program by comparing program outcomes with an estimate of what would have happened in the absence of a program.[24]

According to the World Bank's Independent Evaluation Group (IEG), impact evaluation is the systematic identification of the effects positive or negative, intended or not on individual households, institutions, and the environment caused by a given development activity such as a program or project.[25]

Impact evaluation has been defined differently over the past few decades.[6] Other interpretations of impact evaluation include:

  • An evaluation which looks at the impact of an intervention on final welfare outcomes, rather than only at project outputs, or a process evaluation which focuses on implementation;
  • An evaluation carried out some time (five to ten years) after the intervention has been completed so as to allow time for impact to appear; and
  • An evaluation considering all interventions within a given sector or geographical area.

Other authors make a distinction between "impact evaluation" and "impact assessment." "Impact evaluation" uses empirical techniques to estimate the effects of interventions and their statistical significance, whereas "impact assessment" includes a broader set of methods, including structural simulations and other approaches that cannot test for statistical significance.[18]

Common definitions of 'impact' used in evaluation generally refer to the totality of longer-term consequences associated with an intervention on quality-of-life outcomes. For example, the Organization for Economic Cooperation and Development's Development Assistance Committee (OECD-DAC) defines impact as the "positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended".[26] A number of international agencies have also adopted this definition of impact. For example, UNICEF defines impact as "The longer term results of a program – technical, economic, socio-cultural, institutional, environmental or other – whether intended or unintended. The intended impact should correspond to the program goal."[27] Similarly, Evaluationwiki.org defines impact evaluation as an evaluation that looks beyond the immediate results of policies, instruction, or services to identify longer-term as well as unintended program effects.[28]

Technically, an evaluation could be conducted to assess 'impact' as defined here without reference to a counterfactual. However, much of the existing literature (e.g. NONIE Guidelines on Impact Evaluation[29]) adopts the OECD-DAC definition of impact while referring to the techniques used to attribute impact to an intervention as necessarily based on counterfactual analysis.

What is missing from the term 'impact' evaluation is the way 'impact' shows up long-term. For instance, most Monitoring and Evaluation 'logical framework' plans have inputs-outputs-outcomes and... impacts. While the first three appear during the project duration itself, impact takes far longer to take place. For instance, in a 5-year agricultural project, seeds are inputs, farmers trained in using them our outputs, changes in crop yields as a result of the seeds being planted properly is an outcome and families being more sustainably food secure over time is an impact. Such post-project impact evaluations are very rare. They are also called ex-post evaluations or we are coining the term sustained impact evaluations. While hundreds of thousands of documents call for them, rarely do donors have the funding flexibility - or interest - to return to see how sustained, and durable our interventions remained after project close out, after resources were withdrawn. There are many lessons to be learned for design, implementation, M&E and how to foster country-ownership.

Methodological debates

[edit]

There is intensive debate in academic circles around the appropriate methodologies for impact evaluation, between proponents of experimental methods on the one hand and proponents of more general methodologies on the other. William Easterly has referred to this as 'The Civil War in Development economics' Archived 2010-02-06 at the Wayback Machine. Proponents of experimental designs, sometimes referred to as 'randomistas',[8] argue randomization is the only means to ensure unobservable selection bias is accounted for, and that building up the flimsy experimental evidence base should be developed as a matter of priority.[30] In contrast, others argue that randomized assignment is seldom appropriate to development interventions and even when it is, experiments provide us with information on the results of a specific intervention applied to a specific context, and little of external relevance.[31] There has been criticism from evaluation bodies and others that some donors and academics overemphasize favoured methods for impact evaluation,[32] and that this may in fact hinder learning and accountability.[33] In addition, there has been a debate around the appropriate role for qualitative methods within impact evaluations.[34][35]

Theory-based impact evaluation

[edit]

While knowledge of effectiveness is vital, it is also important to understand the reasons for effectiveness and the circumstances under which results are likely to be replicated. In contrast with 'black box' impact evaluation approaches, which only report mean differences in outcomes between treatment and comparison groups, theory-based impact evaluation involves mapping out the causal chain from inputs to outcomes and impact and testing the underlying assumptions.[36][29] Most interventions within the realm of public policy are of a voluntary, rather than coercive (legally required) nature. In addition, interventions are often active rather than passive, requiring a greater rather than lesser degree of participation among beneficiaries and therefore behavior change as a pre-requisite for effectiveness. Public policy will therefore be successful to the extent that people are incentivized to change their behaviour favourably. A theory-based approach enables policy-makers to understand the reasons for differing levels of program participation (referred to as 'compliance' or 'adherence') and the processes determining behavior change. Theory-Based approaches use both quantitative and qualitative data collection, and the latter can be particularly useful in understanding the reasons for compliance and therefore whether and how the intervention may be replicated in other settings. Methods of qualitative data collection include focus groups, in-depth interviews, participatory rural appraisal (PRA) and field visits, as well as reading of anthropological and political literature.

White (2009b)[36] advocates more widespread application of a theory-based approach to impact evaluation as a means to improve policy relevance of impact evaluations, outlining six key principles of the theory-based approach:

  1. Map out the causal chain (program theory) which explains how the intervention is expected to lead to the intended outcomes, and collect data to test the underlying assumptions of the causal links.
  2. Understand context, including the social, political and economic setting of the intervention.
  3. Anticipate heterogeneity to help in identifying sub-groups and adjusting the sample size to account for the levels of disaggregation to be used in the analysis.
  4. Rigorous evaluation of impact using a credible counterfactual (as discussed above).
  5. Rigorous factual analysis of links in the causal chain.
  6. Use mixed methods (a combination of quantitative and qualitative methods).

Examples

[edit]

While experimental impact evaluation methodologies have been used to assess nutrition and water and sanitation interventions in developing countries since the 1980s, the first, and best known, application of experimental methods to a large-scale development program is the evaluation of the Conditional Cash Transfer (CCT) program Progresa (now called Oportunidades) in Mexico, which examined a range of development outcomes, including schooling, immunization rates and child work.[37][38] CCT programs have since been implemented by a number of governments in Latin America and elsewhere, and a report released by the World Bank in February 2009 examines the impact of CCTs across twenty countries.[39]

More recently, impact evaluation has been applied to a range of interventions across social and productive sectors. 3ie has launched an online database of impact evaluations[permanent dead link] covering studies conducted in low- and middle income countries. Other organisations publishing Impact Evaluations include Innovations for Poverty Action, the World Bank's DIME Initiative and NONIE. The IEG of the World Bank has systematically assessed and summarized the experience of ten impact evaluation of development programs in various sectors carried out over the past 20 years.[40]

Organizations promoting impact evaluation of development interventions

[edit]

In 2006, the Evaluation Gap Working Group[41] argued for a major gap in the evidence on development interventions, and in particular for an independent body to be set up to plug the gap by funding and advocating for rigorous impact evaluation in low- and middle-income countries. The International Initiative for Impact Evaluation (3ie) was set up in response to this report. 3ie seeks to improve the lives of poor people in low- and middle-income countries by providing, and summarizing, evidence of what works, when, why and for how much. 3ie operates a grant program, financing impact studies in low- and middle-income countries and synthetic reviews of existing evidence updated as new evidence appears, and supports quality impact evaluation through its quality assurance services.

Another initiative devoted to the evaluation of impacts is the Committee on Sustainability Assessment (COSA). COSA is a non-profit global consortium of institutions, sustained in partnership with the International Institute for Sustainable Development (IISD) Sustainable Commodity Initiative, the United Nations Conference on Trade and Development (UNCTAD), and the United Nations International Trade Centre (ITC). COSA is developing and applying an independent measurement tool to analyze the distinct social, environmental and economic impacts of agricultural practices, and in particular those associated with the implementation of specific sustainability programs (Organic, Fairtrade etc.). The focus of the initiative is to establish global indicators and measurement tools which farmers, policy-makers, and industry can use to understand and improve their sustainability with different crops or agricultural sectors. COSA aims to facilitate this by enabling them to accurately calculate the relative costs and benefits of becoming involved in any given sustainability initiative.

A number of additional organizations have been established to promote impact evaluation globally, including Innovations for Poverty Action, the World Bank's Strategic Impact Evaluation Fund (SIEF), the World Bank's Development Impact Evaluation (DIME) Initiative, the Institutional Learning and Change (ILAC) Initiative of the CGIAR, and the Network of Networks on Impact Evaluation (NONIE).

Systematic reviews of impact evidence

[edit]

A range of organizations are working to coordinate the production of systematic reviews. Systematic reviews aim to bridge the research-policy divide by assessing the range of existing evidence on a particular topic, and presenting the information in an accessible format. Like rigorous impact evaluations, they are developed from a study Protocol which sets out a priori the criteria for study inclusion, search and methods of synthesis. Systematic reviews involve five key steps: determination of interventions, populations, outcomes and study designs to be included; searches to identify published and unpublished literature, and application of study inclusion criteria (relating to interventions, populations, outcomes and study design), as set out in study Protocol; coding of information from studies; presentation of quantitative estimates on intervention effectiveness using forest plots and, where interventions are determined as appropriately homogeneous, calculation of a pooled summary estimate using meta-analysis; finally, systematic reviews should be updated periodically as new evidence emerges. Systematic reviews may also involve the synthesis of qualitative information, for example relating to the barriers to, or facilitators of, intervention effectiveness.

See also

[edit]

References

[edit]
Revisions and contributorsEdit on WikipediaRead on Wikipedia

Impact evaluation

View on Grokipedia
from Grokipedia
Impact evaluation is a rigorous analytical approach in and policy research that seeks to identify the causal effects of interventions—such as programs, policies, or treatments—on specific outcomes by establishing counterfactual scenarios and attributing observed changes to the intervention itself, rather than factors. This distinguishes it from descriptive monitoring or correlational studies, as it prioritizes through techniques that isolate treatment effects from , endogeneity, and external influences. Central methods include randomized controlled trials (RCTs), which randomly assign participants to treatment and control groups to ensure comparability; quasi-experimental designs like difference-in-differences or regression discontinuity, which leverage natural variation or thresholds for identification; and instrumental variable approaches that exploit exogenous sources of variation to address non-compliance or hidden bias. These tools have enabled evidence-based decisions in fields like international development, education, and health, where evaluations have demonstrated, for instance, the ineffectiveness of certain cash transfer programs in altering long-term behaviors or the modest gains from deworming initiatives in improving school attendance. However, impact evaluation's defining achievements—such as informing the scaling of microfinance or conditional cash transfers—coexist with persistent challenges, including heterogeneous treatment effects across contexts that undermine generalizability and the difficulty of capturing mechanisms beyond average effects. Controversies arise from methodological limitations and systemic biases: RCTs, often hailed as the gold standard, can suffer from attrition, spillover effects, or ethical constraints in , while non-experimental methods risk ; moreover, and selection biases in academic and donor-funded studies favor reporting positive or significant results, inflating perceived intervention and skewing policy toward "what works" narratives that overlook failures or null findings. Academic incentives, including tenure pressures and funding from ideologically aligned institutions, exacerbate this optimism, leading to underreporting of negative impacts and overemphasis on short-term metrics over long-run causal chains. Despite these issues, rigorous impact evaluation remains essential for causal realism in resource-scarce environments, provided evaluations incorporate sensitivity analyses, pre-registration to curb p-hacking, and mixed-methods to probe underlying processes.

Definition and Fundamentals

Core Concepts and Purpose

Impact evaluation entails the rigorous estimation of causal effects attributable to an intervention, program, or on targeted outcomes, achieved by comparing observed results against the counterfactual—what outcomes would have prevailed absent the intervention. This approach distinguishes impact from mere by addressing the fundamental identification problem: the counterfactual remains inherently unobservable, necessitating empirical strategies to approximate it, such as or statistical matching to construct comparable control groups. Central concepts include the (ATE), which quantifies the mean difference in outcomes between treated and untreated units, and considerations of heterogeneity, where effects may vary across subgroups, contexts, or over time. The purpose of impact evaluation lies in generating credible to ascertain whether interventions produce net benefits, the scale of those benefits, and the conditions under which they occur, thereby enabling data-driven decisions in resource-constrained environments. In development contexts, it supports the prioritization of effective programs to alleviate and enhance welfare, as scarce public funds demand verification that expenditures yield measurable improvements rather than illusory gains from factors. Beyond , it informs program refinement, assessments, and replication, countering reliance on anecdotal or associational that often overstates due to omitted variables or selection effects. Evaluations thus promote causal realism, emphasizing mechanisms linking inputs to outputs while highlighting failures, such as null or adverse effects, to avoid perpetuating ineffective practices.

Historical Origins and Evolution

The systematic assessment of program impacts, particularly through , originated in early quantitative evaluation practices but gained methodological rigor in the mid-. Initial roots lie in 19th-century reforms, including William Farish's 1792 introduction of numerical marks for academic performance at Cambridge University and Horace Mann's 1845 standardized tests in schools to gauge educational effectiveness. These efforts focused on for rather than . By the early , Frederick W. Taylor's principles (circa 1911) emphasized efficiency metrics, evolving into objective testing movements that laid groundwork for outcome-oriented scrutiny, though without robust controls for confounding factors. The modern era of impact evaluation emerged in the 1950s-1960s, driven by post-World War II expansions in education and social welfare programs, including the U.S. (1958) and (1965), which mandated evaluations amid concerns over program efficacy. The Sputnik launch in 1957 heightened demands for , while the initiatives spurred social experiments to test interventions like income support. and Julian C. Stanley's 1963 monograph Experimental and Quasi-Experimental Designs for Research formalized designs to mitigate threats—such as and maturation—in non-laboratory settings, enabling causal claims from observational data approximations like pre-post comparisons and nonequivalent control groups. This framework professionalized evaluation, distinguishing true experiments from quasi-experiments and influencing fields beyond . Pioneering randomized controlled trials (RCTs) in followed, with the U.S. experiments (1968-1982) randomizing households to assess guaranteed income effects on labor supply, and the RAND Health Insurance Experiment (1971-1982) evaluating cost-sharing's impact on healthcare utilization, informing 1980s policy shifts toward deductibles. In , Mexico's PROGRESA program (1997) employed RCTs to measure effects on school enrollment and health, catalyzing scalable evaluations across and beyond. The 2000s marked explosive evolution, termed the "evidence revolution," with institutions like the Poverty Action Lab (J-PAL, founded 2003) and the International Initiative for Impact Evaluation (3ie, 2008) institutionalizing RCTs and quasi-experimental methods for poverty alleviation. The U.S. Government Performance and Results Act (1993) and UK Modernizing Government initiative (1999) embedded outcome-focused evaluation in . Advances integrated econometric tools, such as instrumental variables and regression discontinuity designs, to handle endogeneity in large-scale data. This period's emphasis on rigorous peaked with the 2019 in Economics awarded to , , and for RCTs demonstrating interventions' micro-level effects on development outcomes. Subsequent growth includes evidence synthesis via systematic reviews and government-embedded labs, though debates persist over generalizability from small-scale trials to policy scale.

Methodological Designs

Experimental Designs

Experimental designs in impact evaluation primarily utilize randomized controlled trials (RCTs), in which eligible units such as individuals, households, or communities are randomly assigned to treatment (receiving the intervention) or control (no intervention) groups to isolate causal effects from factors. This , typically executed through computer algorithms or lotteries, ensures that groups are statistically equivalent on average, both in observed covariates and unobserved characteristics, allowing outcome differences to be credibly attributed to the intervention. RCTs thus provide unbiased estimates of the (ATE), addressing the fundamental challenge of counterfactual reasoning—what would have happened without the intervention—by using the control group as a proxy. Key steps in RCT design include defining the eligible , conducting power calculations to determine required sample size based on expected effect sizes and variability (often aiming for 80% power to detect minimum detectable effects), and verifying post- balance through statistical tests on baseline . Outcomes are measured via surveys, administrative records, or other instruments at baseline and endline, with analysis focusing on intent-to-treat (ITT) effects—comparing groups as randomized—to maintain integrity, or treatment-on-the-treated (TOT) effects using instruments for compliance issues. Regression models may adjust for covariates to increase precision, though unadjusted differences suffice for primary inference under . Variations adapt RCTs to contextual constraints. Individual-level randomization assigns treatment independently to each unit, maximizing statistical power but risking spillovers in interconnected settings. Cluster-randomized trials, conversely, assign intact groups (e.g., villages or ) to treatment or control, mitigating interference while requiring larger samples and intra-cluster adjustments; for example, Mexico's PROGRESA program randomized 506 communities to evaluate conditional cash transfers, demonstrating sustained impacts on enrollment. designs test multiple interventions simultaneously by crossing treatment arms (e.g., combining cash transfers with training), enabling assessment of interactions and main effects within one trial, as in variations of Indonesia's Raskin food program across 17.5 million beneficiaries in 2012. Stratified or blocked ensures balance across subgroups like or , enhancing precision without altering causal identification. Staggered or phase-in designs roll out interventions sequentially, using early phases as controls for later ones in scalable programs. These designs prioritize but demand safeguards against threats like spillovers (intervention diffusion to controls) or crossovers (controls accessing treatment), addressed via geographic separation or monitoring. Ethical requires uncertainty about intervention efficacy and minimal harm from control withholding, often justified by potential phase-in for all post-evaluation. from RCTs, such as a 43% reduction in arrests from Chicago's One Summer Plus job program, underscores their capacity for policy-relevant causal insights when properly executed.

Quasi-Experimental and Observational Designs

Quasi-experimental designs estimate causal impacts of interventions without , relying instead on structured comparisons or natural variations to approximate experimental conditions. These approaches, first systematically outlined by and Julian C. Stanley in their 1963 chapter, address threats to through designs like time-series analyses or nonequivalent control groups, enabling inference in real-world settings where is infeasible, such as policy implementations or large-scale programs. Unlike true experiments, they demand explicit assumptions—such as the absence of contemporaneous events affecting groups differentially—to isolate treatment effects, with validity often assessed via placebo tests or falsification strategies. A core quasi-experimental method is difference-in-differences (DiD), which identifies impacts by subtracting pre-treatment outcome differences from post-treatment differences between treated and control groups, under the parallel trends assumption that untreated trends would mirror counterfactuals. Applied in evaluations like the 1996 U.S. welfare reform, DiD has shown, for instance, that job training programs increased earnings by 10-20% in some cohorts when controlling for economic cycles. Extensions, such as triple differences, incorporate additional dimensions like geography to mitigate violations from heterogeneous trends, though recent critiques highlight sensitivity to staggered adoption in multi-period settings. Regression discontinuity designs (RDD) exploit deterministic assignment rules, estimating local average treatment effects from outcome discontinuities at a , where units near the threshold are quasi-randomized by the forcing variable. In a 2013 evaluation of Colombia's Ser Pilo Paga , RDD revealed a 0.17 standard deviation increase in enrollment for score-justifiers above the eligibility line, with bandwidth selection via optimal methods ensuring precise local inference. Sharp RDD assumes perfect compliance at the cutoff, while fuzzy variants handle partial take-up using IV within the framework; both require checks for manipulation, such as tests showing no bunching. Instrumental variables (IV) address endogeneity by using an exogenous instrument correlated with treatment uptake but unrelated to outcomes except through treatment, yielding estimates for compliers under monotonicity. In Angrist and Krueger's 1991 analysis of U.S. compulsory , quarter-of-birth instruments—leveraging school entry age laws—estimated a 7-10% return to an additional year of , isolating causal effects amid self-selection. Instrument validity hinges on (strong first-stage correlation) and exclusion (no direct outcome path), tested via overidentification in multiple-IV setups; weak instruments bias estimates toward OLS, as quantified in Stock-Yogo critical values from 2005. Observational designs draw causal inferences from non-manipulated data, emphasizing or structural assumptions to mitigate , often via balancing methods like (PSM), which estimates treatment probabilities from covariates to pair similar units. A 2023 review found PSM effective in observational evaluations of interventions, reducing bias by up to 80% when overlap is sufficient, though it fails with unobservables, as evidenced by simulation studies showing 20-50% attenuation under hidden confounders. Advanced observational techniques include panel fixed effects, which difference out time-invariant confounders in longitudinal data, and synthetic controls, constructing counterfactuals as weighted untreated unit combinations to match pre-treatment trajectories. In Abadie et al.'s 2010 California tobacco control evaluation, synthetic controls attributed a 20-30% drop in per-capita cigarettes to the policy, outperforming simple DiD under heterogeneous trends. These methods demand large samples and covariate balance diagnostics, with triangulation—combining, say, PSM and IV—enhancing robustness, as recommended in 2021 guidelines for non-randomized studies. Despite strengths in scalability, observational designs remain vulnerable to model misspecification, necessitating pre-registration and falsification tests to approximate causal credibility.

Sources of Bias and Validity Threats

Selection and Attrition Biases

occurs when systematic differences between treatment and comparison groups arise due to non-random assignment or participation, leading to distorted estimates of causal effects in impact evaluations. In observational or quasi-experimental designs, individuals self-selecting into programs often possess unobserved characteristics—such as motivation or ability—that correlate with outcomes, inflating or deflating apparent program impacts; for instance, remaining after matching techniques can exceed 100% of the experimentally estimated effect in social program evaluations. This threat undermines by violating the assumption of exchangeability between groups, making it challenging to attribute outcome differences solely to the intervention rather than pre-existing disparities. Even in randomized controlled trials (RCTs), can emerge if eligibility criteria or processes favor certain subgroups, though proper typically mitigates it at baseline. Attrition bias, a post-randomization form of , arises when participants exit studies at differential rates between , particularly if dropouts are correlated with outcomes or treatment status, thereby altering group compositions and biasing effect estimates. In RCTs for social programs, such as interventions, attrition rates exceeding 20% often introduce systematic imbalances, with leavers in treatment groups potentially having worse outcomes than stayers, leading to overestimation of positive effects if not addressed. This bias threatens the completeness of intention-to-treat analyses and can amplify in longitudinal evaluations where follow-up surveys fail to retain high-risk participants, as seen in teen prevention trials where cluster-level attrition exacerbates imbalances. Unlike baseline selection, attrition introduces time-varying , as dropout reasons—like program dissatisfaction or external shocks—may interact with treatment exposure. Both biases compromise by eroding the comparability of groups essential for counterfactual estimation; selection operates pre-treatment, while attrition does so post-treatment, but they converge in non-random loss of that correlates with potential outcomes. In development impact evaluations, empirical assessments show that unadjusted attrition can shift effect sizes by 10-30% in magnitude, with bounding approaches or sensitivity analyses revealing the direction of potential distortion. Mitigation strategies include baseline covariates for reweighting, worst-case scenario bounds, or pattern-mixture models, though these require assumptions about missingness mechanisms that may not hold without auxiliary . High-quality evaluations report attrition rates and test for baseline differences among dropouts to quantify threats, emphasizing that low attrition alone does not guarantee unbiasedness if patterns are non-ignorable.

Temporal and Contextual Biases

Temporal biases in impact evaluation refer to systematic errors introduced by time-related factors that confound causal attribution, often threatening by providing alternative explanations for observed changes in outcomes. History effects occur when external events, unrelated to the intervention, coincide with its implementation and influence results; for instance, a concurrent change might inflate estimates of a job training program's employment effects. Maturation effects arise from natural developmental or aging processes in participants, such as improved in children over the study period, which could be mistakenly attributed to an educational intervention. These biases are particularly pronounced in longitudinal or quasi-experimental designs lacking , where pre-intervention trends or secular drifts—broader societal shifts like technological adoption—may parallel the treatment timeline and bias impact estimates upward or downward. Regression to the mean exacerbates temporal issues when extreme baseline values naturally moderate over time, as seen in evaluations of interventions targeting high-risk groups, such as programs where initial severity scores revert without treatment influence. To mitigate, evaluators often employ difference-in-differences methods to test parallel trends or include time-fixed effects in models. Contextual biases stem from the specific setting or environment of the , which can modify intervention effects or introduce local confounders, thereby limiting generalizability and introducing effect heterogeneity. Interaction effects with settings manifest when outcomes vary due to unmeasured site-specific factors, such as cultural norms or institutional support; for example, a program's success in rural areas may not replicate in urban contexts due to differing market dynamics. Spillover effects, where treatment benefits leak to controls within the same locale, contaminate comparisons, as documented in cluster-randomized trials of interventions where community-level biases null findings toward underestimation. Hawthorne effects represent a reactive contextual , wherein participants alter behavior due to awareness of evaluation, inflating impacts in monitored settings like workplace productivity studies. Site selection further compounds issues when programs are evaluated in non-representative locations correlated with higher efficacy, such as motivated communities, leading to overoptimistic extrapolations. Addressing these requires explicit testing for moderators via subgroup analyses or heterogeneous treatment effect estimators, alongside transparent reporting of contextual descriptors to aid assessments.

Estimation and Analytical Techniques

Causal Inference Methods

Causal inference methods in impact evaluation seek to identify and quantify the effects of interventions by estimating counterfactual outcomes, typically under the potential outcomes framework. This framework posits that for each unit ii, there exist two potential outcomes: Yi(1)Y_i(1) under treatment and Yi(0)Y_i(0) under control, with the individual treatment effect defined as Yi(1)−Yi(0)Y_i(1) - Y_i(0). The (ATE) averages this difference across units, but the fundamental challenge arises because only one outcome is observed per unit, necessitating assumptions to link observables to the unobserved counterfactual. Originating from Neyman's work in randomized experiments (1923) and extended by (1974) to broader settings, the framework underpins modern quasi-experimental estimation by emphasizing identification via or exclusion restrictions. These methods are particularly vital in observational data from impact evaluations, where is absent, requiring strategies to mimic experimental conditions through covariates, instruments, or discontinuities. Common approaches include , instrumental variables, regression discontinuity, and difference-in-differences, each relying on distinct identifying assumptions to bound or point-identify causal effects. While powerful, their validity hinges on untestable assumptions, such as no unmeasured confounders or parallel trends, which empirical checks like placebo tests or sensitivity analyses can probe but not fully verify. Propensity Score Matching (PSM) balances treated and control groups by matching on the propensity score, defined as the probability of treatment given observed covariates XX, e(X)=P(D=1∣X)e(X) = P(D=1|X). Under selection on observables (: Y(1),Y(0)⊥D∣XY(1), Y(0) \perp D | X), matching yields unbiased estimates of the ATE for the treated or overall. Introduced by Rosenbaum and Rubin (1983), PSM reduces dimensionality from multiple covariates to one score, often implemented via nearest-neighbor or kernel matching, with caliper restrictions to ensure close matches. In impact evaluations of social programs, such as job training initiatives, PSM has estimated effects like a 10-20% earnings increase from participation, though it fails if unobservables like motivation confound assignment. Sensitivity to model misspecification and common support violations necessitates balance diagnostics, where covariate means post-matching should align across groups. Instrumental Variables (IV) addresses endogeneity from unobservables by leveraging an instrument ZZ correlated with treatment DD (relevance: Cov(Z,D)≠0\text{Cov}(Z,D) \neq 0) but affecting outcomes YY only through DD (exclusion: no direct path from ZZ to YY). The two-stage least squares (2SLS) estimator recovers the local average treatment effect (LATE) for compliers—those whose treatment status changes with ZZ—under monotonicity (no defiers). Angrist, Imbens, and Rubin (1996) formalized LATE as the relevant parameter when heterogeneity exists, applied in evaluations like quarter-of-birth instruments for , yielding IV estimates of 7-10% per year of versus 5-8% from OLS. Weak instruments estimates toward OLS (first-stage F-statistic >10 recommended), and exclusion violations, such as spillover effects, undermine credibility; overidentification tests (Sargan-Hansen) assess multiple instruments. Regression Discontinuity Design (RDD) exploits sharp or fuzzy discontinuities at a known cutoff in the assignment rule, treating units just above and below as locally randomized. In sharp RDD, the treatment effect is the jump in the conditional expectation of YY at the cutoff, estimated via local polynomials or parametric regressions with bandwidth selection (e.g., Imbens-Kalyanaraman optimal). Imbens and Lemieux (2008) outline implementation, including density tests for manipulation and placebo outcomes for bandwidth sensitivity. For policy cutoffs like scholarships at exam score thresholds, RDD has quantified effects such as a 0.2-0.5 standard deviation improvement in future earnings, with strongest near the cutoff but limited to that margin. Fuzzy RDD extends to imperfect compliance using IV logic, where the first-stage discontinuity instruments the treatment probability. Difference-in-Differences (DiD) estimates effects by differencing changes in outcomes over time between treated and control groups, identifying the ATE under parallel trends: absent treatment, gaps would evolve similarly. The estimator is (E[YTT]−E[YTC])−(E[YCT]−E[YCC])(E[Y_{TT}] - E[Y_{TC}]) - (E[Y_{CT}] - E[Y_{CC}]), where subscripts denote treated/ control and post/pre periods. Bertrand, Duflo, and Mullainathan (2004) highlight serial correlation inflating standard errors in multi-period panels, recommending clustered errors or data collapse to two periods for robustness. In evaluations of minimum wage hikes, DiD has shown null or small employment effects (e.g., -0.1% per 10% wage increase), contrasting event-study pre-trends to validate assumptions. Extensions like triple differences add a third dimension to control fixed differences, but violations from differential shocks (e.g., Ashenfelter dips) require synthetic controls or staggered adoption adjustments. Other techniques, such as synthetic control for aggregate interventions, construct counterfactuals as weighted combinations of untreated units matching pre-treatment trends, effective for like policy reforms in single units. Across methods, robustness checks, including applications and falsification on pre-treatment , are essential, as are meta-analyses revealing that quasi-experimental estimates often align with RCTs when assumptions hold, though divergence signals . Integration with for covariate adjustment or double robustness (combining outcome and propensity models) enhances precision but demands large samples to avoid .

Economic Evaluation Integration

Economic evaluation integration in impact evaluation extends causal effect estimation by incorporating cost data to assess , enabling comparisons of interventions' value relative to alternatives. This approach quantifies whether observed impacts justify expended resources, often through metrics like incremental cost-effectiveness ratios (ICERs) or benefit-cost ratios (BCRs). For instance, in development programs, impact evaluations using randomized controlled trials (RCTs) may pair treatment effect estimates on outcomes such as school enrollment with program delivery costs to compute costs per additional enrollee. Such integration supports decision-making on scaling interventions, as seen in analyses by organizations like the International Initiative for Impact Evaluation (3ie), which emphasize prospective cost data alongside experimental designs to avoid biases. Cost-effectiveness analysis (CEA), a primary method, measures the cost per unit of outcome achieved, such as dollars per life-year saved or per child educated, without requiring full of benefits. In RCT-based impact evaluations, CEA typically applies the intervention's average cost per beneficiary to the estimated , yielding ratios like $X per Y% increase in productivity. A 2024 3ie outlines standardized steps for CEA in impact evaluations, including delineating direct and (e.g., staff time, materials, overhead) and sensitivity analyses for uncertainty in effect sizes or cost estimates. Challenges include attributing shared costs in multi-component interventions and using shadow prices for non-traded inputs in low-income settings, where market prices may distort true opportunity costs. Cost-benefit analysis (CBA) advances further by monetizing all outcomes, comparing discounted streams of benefits against costs to derive net present values or internal rates of return. Applied to impact evaluations, CBA requires valuing non-market effects, such as health improvements via willingness-to-pay proxies or human capital models projecting lifetime earnings gains from interventions. A World Bank analysis found that fewer than 20% of impact evaluations incorporate CBA, often due to data demands and methodological debates over valuation assumptions, yet those that do reveal high returns, like BCRs exceeding 5:1 for deworming programs in based on long-term income effects. Integration with quasi-experimental designs demands adjustments for selection biases in cost attribution, using techniques like to estimate counterfactual costs. Despite advantages, integration faces institutional barriers, including underinvestment in cost data collection during trials, where focus prioritizes of impacts over economic metrics. Guidelines from bodies like the World Bank advocate embedding economic components from study inception, with prospective costing protocols to capture fixed and variable expenses accurately. Empirical evidence from underscores the policy relevance, as integrated evaluations have informed reallocations, such as prioritizing cash transfers over less cost-effective subsidies when BCRs differ by factors of 2-10. Ongoing refinements address generalizability, incorporating transferability adjustments for context-specific costs and effects across settings.

Debates and Methodological Controversies

RCT Gold Standard vs. Alternative Approaches

Randomized controlled trials (RCTs) are widely regarded as the gold standard in impact evaluation for establishing causal effects due to randomization, which balances on both observed and unobserved confounders, thereby minimizing and enabling unbiased estimates of average treatment effects under ideal conditions. This approach has been particularly influential in fields like , where organizations such as J-PAL have scaled RCTs to evaluate interventions like programs, yielding precise estimates of effects such as a 0.14 standard deviation increase in earnings from childhood in as of long-term follow-ups reported in 2019. However, proponents acknowledge that RCTs assume stable mechanisms and no spillover effects, which may not hold in complex social settings. Despite their strengths in , RCTs face significant limitations that challenge their unqualified status as the gold standard. Ethical constraints prevent in many contexts, such as evaluating universal programs like national reforms, while high costs—often exceeding $1 million per trial in development settings—and long timelines limit . is another concern, as RCT participants and settings are often unrepresentative; for instance, trials in controlled environments may overestimate effects in diverse real-world applications, with meta-analyses showing effect sizes in RCTs decaying by up to 50% when scaled up. Critics like argue that RCTs provide narrow, context-specific knowledge without illuminating underlying mechanisms or generalizability, potentially misleading if treated as universally superior evidence, as evidenced by discrepancies between RCT findings and broader econometric data in alleviation studies. Alternative approaches, particularly quasi-experimental designs, offer robust when RCTs are infeasible by exploiting natural or policy-induced variation. Methods like regression discontinuity designs (RDD) assign treatment based on a score, approximating near the threshold; for example, an RDD evaluation of Colombia's scholarship program in 2012 estimated a 4.8 increase in enrollment, comparable to RCT benchmarks. Difference-in-differences (DiD) compares changes over time between treated and untreated groups assuming parallel trends, as in Card and Krueger's 1994 minimum wage study, which found no employment loss in fast-food sectors post-1992 hike. Instrumental variables (IV) use exogenous shocks for identification, addressing endogeneity in observational data. These methods rely on testable assumptions—such as no anticipation in RDD or parallel trends in DiD—allowing empirical validation, and often provide stronger by leveraging large-scale administrative data rather than small, artificial samples. The debate pits RCT advocates, including and —who emphasize randomization's avoidance of model dependence against alternatives' reliance on untestable assumptions—against skeptics like Deaton and , who contend that no method guarantees without and , as RCTs can suffer from attrition (up to 20-30% in social trials) or Hawthorne effects. Empirical comparisons reveal mixed results: a 2022 analysis of labor interventions found quasi-experimental estimates aligning with RCTs 70-80% of the time when assumptions hold, but diverging in heterogeneous contexts, underscoring that alternatives can match RCT precision while better capturing policy-relevant variation. In impact evaluation, over-reliance on RCTs, often promoted by institutions with vested interests in experimental methods, risks sidelining credible quasi-experimental evidence from natural experiments, as seen in macroeconomic policy assessments where observational designs have informed reforms like conditional cash transfers in .
ApproachKey StrengthKey LimitationExample Application
RCTsHigh internal validity via Poor scalability, ethical barriers, limited generalizabilityMicrofinance impacts in (2000s trials showing modest effects)
Quasi-Experimental (e.g., DiD, RDD)Leverages real-world for broader applicabilityDepends on assumptions like parallel trends, testable but not always verifiable effects (DiD in 1994 U.S. study)
Ultimately, causal realism demands selecting methods based on context rather than , integrating RCTs for precision where possible with quasi-experimental and mechanistic analyses for robustness, as singular elevation of any approach ignores the pluralistic nature of evidence in complex systems.

Empirical vs. Theory-Driven

In impact , empirical prioritizes observable data and to determine program effects, often employing randomized controlled trials (RCTs) or quasi-experimental designs to isolate causal impacts on outcomes while treating interventions as "black boxes" that link inputs directly to results without explicit modeling of internal processes. This approach, rooted in the positivist paradigm's emphasis on objective measurement and , seeks to establish whether an intervention produces net benefits through rigorous hypothesis testing and control for variables, as seen in evaluations by organizations like the Poverty Action Lab (J-PAL), which reported over 1,000 RCTs by 2023 demonstrating average treatment effects in areas like and . Such methods excel in providing high , with meta-analyses showing RCTs yielding effect sizes that are more precise and less biased than non-experimental alternatives, though they may overlook heterogeneous effects across contexts. Theory-driven evaluation, by contrast, integrates explicit program theories—such as theories of change or realist causal mechanisms—to unpack how interventions generate outcomes via intermediate links, resources, and contextual factors, rather than solely relying on outcome . Originating in the as a of black-box limitations, this method, advanced by evaluators like Huey Chen, posits that understanding "what works for whom, in what circumstances, and why" requires mapping assumed causal pathways and testing them empirically or qualitatively, as applied in assessments by the International Institute for Environment and Development (IIED). For instance, a 2014 study on knowledge translation initiatives used realist evaluation to identify context-mechanism-outcome configurations, revealing why certain programs succeeded in specific settings despite similar average effects. Proponents argue it enhances and scalability by addressing generalizability gaps in purely empirical designs, with Treasury Board of Canada guidelines from 2021 recommending its use to examine causal chains beyond net impacts. The tension between these paradigms reflects broader methodological debates in science, where empirical is lauded for its causal rigor—evidenced by post-positivist refinements acknowledging researcher influence but still prioritizing quantifiable evidence over metaphysical assumptions—yet critiqued for that ignores implementation fidelity and adaptive behaviors. -driven approaches counter this by fostering deeper causal realism through mechanism testing, but they risk if program theories embed unverified ideological assumptions, as noted in critiques of their subjective theory construction potentially amplifying biases in academic settings where qualitative methods predominate. Empirical evaluations have demonstrated superior replicability in contexts, with a 2020 review finding that black-box RCT findings influenced 15% more legislative changes than theory-only assessments, though hybrid models combining both—such as realist RCTs—emerge as pragmatic syntheses to balance evidentiary strength with explanatory depth. In practice, over-reliance on positivist metrics in high-stakes funding decisions, like those from USAID since 2010, has prompted calls for theory integration to mitigate failures in scaling empirically validated pilots, underscoring that while empirical methods ground truth claims in data, theory-driven elements are essential for causal interpretation without supplanting evidential primacy.

Ethical, Practical, and Ideological Critiques

Ethical critiques of impact evaluation, particularly randomized controlled trials (RCTs), center on the moral implications of , which deliberately withholds interventions from control groups to establish . This practice raises concerns about equity and beneficence, as it may deny potentially life-improving treatments to participants in need, especially when preliminary evidence or is absent, violating principles like those in the Declaration of Helsinki. In development contexts, where populations often face or health vulnerabilities, RCTs can exacerbate inequalities by favoring treatment groups, prompting debates over whether such designs are justifiable without assured post-trial access for controls. Critics like argue that conducting RCTs when interventions are suspected to work undermines ethical standards, as it prioritizes experimental purity over participant welfare, potentially amounting to exploitation in low-resource settings. Practical challenges include the high financial and temporal costs of RCTs, which often require large samples, extended follow-ups, and sophisticated , rendering them infeasible for small-scale or urgent programs in resource-constrained environments. Attrition, non-compliance, and contextual dependencies further compromise reliability, as real-world deviates from idealized protocols, leading to underpowered studies unable to detect modest effects. remains a persistent issue; findings from specific, controlled settings—such as deworming programs in rural —frequently fail to replicate or scale in diverse populations or policy environments, limiting their utility for broad decision-making. Ideological critiques portray RCT-centric impact evaluation as emblematic of empirical , which elevates narrow, ahistorical data over theoretical models, contextual nuances, and , fostering a "randomista" that dismisses non-experimental . This approach is accused of technocratic overreach, depoliticizing by framing decisions as purely evidence-driven while sidelining value judgments, power dynamics, and ethical trade-offs inherent to . In , such methods have been labeled neo-colonial, imposing Western scientific paradigms on global South contexts and prioritizing measurable outcomes over holistic, theory-guided interventions that address systemic causes like institutional failures. Proponents of alternatives, including structural economists, contend that RCTs' aversion to prior assumptions hinders causal understanding in complex social systems, where mechanisms demand mechanistic reasoning beyond treatment effects.

Applications and Empirical Evidence

Development and Social Programs

Impact evaluations, predominantly through randomized controlled trials (RCTs), have been extensively applied to development and social programs in low- and middle-income countries, yielding causal evidence on interventions targeting poverty alleviation, health, education, and nutrition. Organizations such as the Abdul Latif Jameel Poverty Action Lab (J-PAL) and the World Bank have conducted or funded numerous RCTs to assess program effectiveness, revealing heterogeneous outcomes where some interventions demonstrate robust benefits while others show modest or null effects. These evaluations emphasize scalable, low-cost programs like and cash transfers, but also highlight challenges such as generalizability beyond pilot settings and long-term . Conditional cash transfer (CCT) programs, which link payments to behaviors like school attendance and health checkups, provide some of the strongest of positive impacts. Mexico's Progresa (later ), launched in 1997, was evaluated using RCTs on over 24,000 households, showing increases in school enrollment by approximately 20% for girls in and improvements in health outcomes, including a 10-18% rise in rates and reduced malnutrition. Long-term follow-ups indicated sustained effects, such as higher consumption and reduced into adulthood, though benefits were more pronounced for targeted poor households. Unconditional cash transfers (UCTs), without behavioral requirements, have been analyzed in a Bayesian of 115 studies across 72 programs, estimating average effects including a 0.08 standard deviation increase in household consumption and reduced , with stronger impacts in acute contexts but limited evidence of transformative poverty escape. In health-focused social programs, initiatives stand out for cost-effectiveness, with RCTs in demonstrating that school-based treatment reduced worm infections and increased school by 25%, alongside long-run earnings gains of up to 20% for treated children tracked into adulthood. A 2022 meta-analysis of multiple studies confirmed modest nutritional benefits, such as 0.3 kg average in children per treatment round, though effects on and height were inconsistent or negligible. Reanalyses of flagship studies have debated effect sizes, attributing some discrepancies to externalities like community-wide treatment spillovers, underscoring the need for careful interpretation in scaling. Microfinance programs, aimed at fostering among the poor, contrast with these successes, as RCTs across six countries found limited causal impacts on household income or consumption, with meta-analyses of seven evaluations reporting negligible for non-entrepreneurial households and only modest adoption among borrowers. These null or small effects challenge earlier observational claims of broad transformative potential, revealing instead that access to credit often supports rather than sustained growth, particularly in saturated markets. Overall, empirical evidence from these applications supports selective investment in high-evidence interventions like CCTs and , which yield positive returns at costs under $100 per beneficiary annually, but cautions against over-reliance on programs like without addressing selection into . Integration with non-experimental methods, such as regressions on observational data, has complemented RCTs for broader policy contexts where is infeasible.

Policy and Institutional Interventions

Impact evaluations of policy and institutional interventions employ methods, such as randomized controlled trials (RCTs) and difference-in-differences (DiD) designs, to measure the effects of reforms on outcomes like , service delivery, and quality. These assessments often reveal mixed results, with successes dependent on contextual factors including political incentives and implementation capacity, while many donor-supported initiatives fail to deliver sustained improvements. For instance, between 1998 and 2008, donor-backed "" reforms in 145 countries resulted in a decline in effectiveness for 50% of recipients, as measured by World Bank Governance Indicators, highlighting challenges in achieving causal improvements through institutional changes. Decentralization policies, which devolve authority to local levels, have been evaluated for their impacts on and public provision. A randomized evaluation in during the early 2000s assigned village to women under quotas, finding that female policymakers increased investments in public and roads—goods disproportionately benefiting women—by 10-15 percentage points compared to male-led villages, demonstrating causal effects on pro-poor outcomes via improved representation. In , the 1994 Popular Participation Law, which decentralized 20% of national revenue to municipalities, led to shifts in spending toward and in poorer areas, with per capita infrastructure investments rising by up to 25% in responsive localities, though overall impacts varied by local . Streamlining administrative institutions, such as one-stop service (OSS) reforms, aims to reduce bureaucratic hurdles for business registration and permits. In , the 2018 OSS institutional overhaul, consolidating licensing across 369 districts, was assessed using a staggered DiD model on 2014-2018 , revealing a short-term negative impact on per-capita GDP growth, with a of -0.011 (p<0.1), attributed to transitional disruptions like capacity gaps and risk-averse implementation. Police institutional reforms, including protocols, have shown more consistent causal benefits in RCTs; a multicity U.S. trial in 2015-2016 found procedural justice increased officer compliance with constitutional standards by 10-20%, reducing citizen complaints without elevating rates. Similarly, a 2024 RCT of police use-of-force in a large agency reported a statistically significant reduction in force incidents post-intervention. Broader evidence from reforms indicates limited success in curbing administrative , with systematic reviews finding that while gains reduce opportunities for graft, sustained declines require complementary , as isolated institutional tweaks often yield null or perverse effects due to entrenched incentives. These findings underscore the importance of rigorous, context-specific evaluations to distinguish effective interventions from those undermined by failures or political short-termism.

Organizations, Initiatives, and Reviews

Key Promoters and Evidence Producers

The Abdul Latif Jameel Poverty Action Lab (J-PAL), established in 2003 at the Massachusetts Institute of Technology, serves as a central hub for promoting randomized controlled trials (RCTs) in impact evaluation, particularly in poverty alleviation and development economics. J-PAL-affiliated researchers have conducted or overseen more than 1,100 randomized evaluations worldwide, generating empirical evidence on interventions such as deworming programs, remedial education, and conditional cash transfers, which have informed scalable policies in over 80 countries. Its founders, including Nobel laureates Abhijit Banerjee and Esther Duflo, emphasize RCTs for establishing causal impacts, training policymakers and researchers through courses and partnerships to prioritize evidence over intuition in program design. Innovations for Poverty Action (IPA), founded in 2002 by economist Dean Karlan, functions as a research network that executes field experiments to test poverty interventions, producing evidence on topics like microfinance efficacy, agricultural innovations, and behavioral nudges. IPA has completed hundreds of RCTs across more than 50 countries, collaborating with governments and NGOs to scale proven programs, such as improving teacher attendance in or reducing fraud in cash transfers, while addressing organizational challenges in embedding rigorous evaluation into operations. It complements J-PAL by focusing on implementation science, providing tools for theory-driven evaluations and partnering on joint initiatives to build capacity for evidence generation in low-resource settings. The International Initiative for Impact Evaluation (3ie), launched in as a grant-making NGO, funds and synthesizes high-quality impact studies to support evidence-informed policies in low- and middle-income countries, emphasizing transparency through systematic reviews and repositories of over 4,000 evaluations. 3ie has disbursed grants for more than 300 primary studies and produced evidence maps on sectors like , and climate adaptation, promoting mixed-methods approaches alongside RCTs to enhance generalizability and uptake by decision-makers. It quality-assures outputs via rigorous protocols, countering by incentivizing registration and reporting of null results. Other notable producers include the World Bank's Strategic Impact Evaluation Fund (SIEF), active since 2008, which has supported over 100 studies measuring program effects in areas like and service delivery, influencing Bank-wide lending decisions with data from RCTs in and . The International Food Policy Research Institute (IFPRI) has conducted causal evaluations since the late , including landmark RCTs on Mexico's PROGRESA program, generating evidence on nutrition-sensitive agriculture and social safety nets adopted in multiple nations. These entities collectively advance a of empirical testing, though their RCT-centric focus has drawn scrutiny for potential overemphasis on narrow, context-specific findings at the expense of broader causal mechanisms.

Skeptics, Critics, and Reform Advocates

Nobel laureate Angus Deaton has critiqued the application of randomized controlled trials (RCTs) in impact evaluation, arguing that they are often misinterpreted as providing unassailable evidence for policy without addressing external validity or causal mechanisms. Deaton and co-author Nancy Cartwright contend that RCTs require minimal theoretical assumptions, which aids persuasion in skeptical contexts but hinders deeper understanding by sidelining prior knowledge and generalizability beyond specific trial conditions. They emphasize that RCTs cannot standalone as "gold standard" proofs, as replication across varied settings is rare, and results may fail to predict outcomes in scaled implementations due to contextual differences. Lant Pritchett has similarly challenged the RCT paradigm in development impact evaluation, highlighting paradoxes in where small-scale trials yield effects that diminish or reverse at larger scales due to implementation challenges and institutional constraints. Pritchett argues that RCTs disproportionately focus on marginal, short-term interventions like private goods (e.g., ) rather than public goods or systemic reforms, diverting attention from transformative questions about and . He critiques the methodology for underemphasizing mechanisms of change and scalability, noting that even positive trial findings often encounter "fade-out" when rolled out nationally, as seen in interventions where contract teacher effects did not persist broadly. Ethical concerns form another core critique, particularly in development contexts where control groups receive no intervention, potentially withholding beneficial treatments from vulnerable populations. Deaton points to cases like cash transfers or health programs where equates to denying aid, raising moral hazards absent equipoise—true uncertainty about —that is harder to establish for social policies than medical ones. Critics like Ravi argue this practice influences research agendas toward low-stakes questions, amplifying disproportionate sway over policy while exposing participants to harms without adequate safeguards. Reform advocates urge integrating RCTs with theory-driven approaches, qualitative insights, and quasi-experimental methods to enhance and policy relevance. Deaton advocates for RCTs within cumulative scientific programs that incorporate mechanistic understanding and historical data, rather than isolated . Pritchett calls for frameworks prioritizing and growth-oriented reforms, arguing that methodological pluralism better addresses barriers than RCT . Such reforms aim to mitigate biases toward feasible but narrow studies, fostering evaluations that inform ambitious interventions despite academia's institutional incentives favoring RCT production.

Recent Developments and Challenges

Technological and Methodological Innovations

Advancements in have enhanced in impact evaluation by addressing high-dimensional data and model misspecification. Double machine learning (Double ML) employs supervised algorithms to flexibly estimate nuisance parameters, such as propensity scores and conditional expectations, within semi-parametric estimators for average treatment effects under unconfoundedness assumptions, thereby improving precision and reduction compared to parametric alternatives. Targeted learning integrates ensemble methods like Super Learner into targeted , allowing for data-adaptive while targeting causal parameters, as demonstrated in policy effect estimations where traditional methods falter with complex covariates. These approaches, formalized in frameworks from 2019 onward, enable evaluators to incorporate vast covariate sets without risks inherent in purely parametric models. Synthetic control methods have seen refinements for broader applicability in non-experimental settings. Generalized synthetic control approaches, which extend the original method by incorporating interactive fixed effects, have shown superior performance over standard difference-in-differences and synthetic controls in simulations involving staggered adoption or heterogeneous treatments, particularly for evaluations with controlled donor pools. Recent extensions, such as using multiple outcomes to construct synthetic counterfactuals, mitigate interpolation biases in single-unit interventions, as applied in re-evaluations of shocks where pre-treatment fit is optimized across dimensions like economic and social indicators. These innovations, building on Abadie's framework, facilitate causal claims in contexts lacking randomized variation, such as regional reforms, with applications documented as early as 2015 in health interventions. Technological innovations leverage for scalable outcome measurement and real-time assessment. has enabled proxy-based evaluations of environmental and agricultural programs by capturing changes in or crop yields without reliance on household surveys; for example, analyses in have used it to assess productivity impacts from development interventions. Imagery data, including nighttime lights and high-resolution sensors, supports quasi-experimental designs for hard-to-measure outcomes like , with World Bank evaluations highlighting its advantages in coverage and timeliness since the early 2020s. Administrative records and call detail records (CDR) provide granular, longitudinal data for difference-in-differences setups, as mapped in systematic reviews linking to outcomes, though causal applications remain limited by endogeneity concerns. Digital tools have transformed for impact evaluation, enabling real-time monitoring and reducing logistical costs. Mobile-based surveys and GPS-enabled applications facilitate continuous tracking in RCTs and quasi-experiments, as seen in India's programs where app-based reporting monitored toilet construction and usage daily, allowing adaptive interventions. Geospatial integration of these tools with data enhances precision in attributing effects, such as in agricultural RCTs measuring plot-level yields via phone . A 2023 3ie systematic map indicates growing use of such in impact studies, particularly for measurement validation, but underscores gaps in rigorous integration due to and issues. These methods, accelerated by post-2020 digital expansions, support faster feedback loops in policy cycles compared to traditional endline surveys.

Barriers to Policy Influence and Scalability

Impact evaluations frequently encounter resistance in translating findings into policy due to political and institutional dynamics that prioritize or expediency over causal . In a analysis of 73 randomized controlled trials conducted across 30 U.S. cities with a national behavioral insights team, positive results prompted adoption in only 27% of cases, often due to bureaucratic inertia, competing priorities, and skepticism about beyond pilot settings. Similarly, policymakers may disregard evaluations conflicting with entrenched interests, as evidenced by persistent underuse of rigorous impact data in domains like , where ideological commitments to unproven approaches prevail despite contrary empirical results. Dissemination challenges further impede influence, including untimely evaluation outputs and poor alignment between researchers' focus on average treatment effects and policymakers' need for context-specific, actionable insights. Academic and donor-driven evaluations, while methodologically sound, often fail to engage decision-makers early, leading to findings that are technically credible but politically inert; for example, systematic reviews identify lack of , relevant as the most cited barrier, compounded by institutional that fragment evidence uptake. This disconnect is exacerbated in polarized environments, where evidence is selectively interpreted to fit partisan narratives rather than assessed on causal merits. Scalability of proven interventions presents distinct hurdles, as pilot successes under controlled conditions rarely persist at larger scopes due to emergent complexities like spillovers, heterogeneous effects, and general equilibrium shifts not captured in randomized designs. Cost structures, for instance, inflate dramatically upon expansion—small-scale programs may yield high returns in trials funded by external grants, but national rollout demands sustained public budgets amid diminishing marginal benefits and frictions, as seen in attempts to scale micro-interventions in low-income settings where logistical and capacity constraints erode . Critiques highlight that many impact evaluations target incremental "islets" of intervention, such as targeted subsidies or nudges, which prove inadequate for systemic requiring institutional overhauls beyond experimental scope. Lant Pritchett argues this micro-focus yields evidence with limited predictive power for scaled policy, as real-world adoption introduces adaptive changes that alter causal pathways; empirical tracking reveals few RCT-backed programs achieve broad rollout, with adoption rates remaining low due to unaddressed factors like or weak . In development contexts, barriers such as these have constrained scaling of even modestly successful trials, underscoring the gap between localized causal identification and feasible policy transformation.

References

  1. https://www.betterevaluation.org/methods-approaches/themes/impact-evaluation
  2. http://www.pamjakiela.com/IE34.pdf
  3. https://www.adb.org/sites/default/files/event/978811/files/session67-dr-daniel-c-suryadarma-impact-evaluation-methods-i-impact-evaluation-methods-ii-rev.pdf
  4. https://hal.science/hal-02866828v1/document
  5. https://thedocs.worldbank.org/en/doc/e4fd355497464272c91b607fb4f24308-0090012025/original/-Arndt-Reichert-Advanced-methods-AR.pdf
  6. https://pubs.aeaweb.org/doi/10.1257/aer.p20151015
  7. https://www.givewell.org/united-states/process/common-evaluation-problems
  8. https://academic.oup.com/rev/article/doi/10.1093/reseval/rvaf033/8215691
  9. https://ssir.org/articles/entry/ten_reasons_not_to_measure_impact_and_what_to_do_instead
  10. https://thedocs.worldbank.org/en/doc/511411529868978577-0160022017/render/2pmMarch8PremandRawlingsmarch32017SPLweek001.pdf
  11. https://ieg.worldbankgroup.org/reports/impact-evaluation-experience-independent-evaluation-group-world-bank
  12. https://documents1.worldbank.org/curated/en/823791468325239704/pdf/Impact-evaluation-in-practice.pdf
  13. https://www.cif.org/sites/cif_enc/files/knowledge-documents/impact_evaluation_0.pdf
  14. https://openknowledge.worldbank.org/entities/publication/f89faa3e-3aba-5b06-ab9c-5fc4dce9be59
  15. https://www.3ieimpact.org/sites/default/files/principles-for-ie.pdf
  16. https://opensiuc.lib.siu.edu/cgi/viewcontent.cgi?article=1056&context=ojwed
  17. https://us.sagepub.com/sites/default/files/upm-assets/110807_book_item_110807.pdf
  18. https://www.sfu.ca/~palys/Campbell&Stanley-1959-Exptl&QuasiExptlDesignsForResearch.pdf
  19. https://www.cgdev.org/sites/default/files/impact-evaluation-timeline.pdf
  20. https://blogs.worldbank.org/en/impactevaluations/history-randomized-control-trials-scurvy-poets-and-beer
  21. https://www.3ieimpact.org/
  22. https://dimewiki.worldbank.org/Randomized_Control_Trials
  23. https://www.povertyactionlab.org/resource/introduction-randomized-evaluations
  24. https://pmc.ncbi.nlm.nih.gov/articles/PMC4910682/
  25. https://www.jameslindlibrary.org/campbell-dt-stanley-jc-1963/
  26. https://www.betterevaluation.org/tools-resources/quasi-experimental-design-methods
  27. https://www.betterevaluation.org/methods-approaches/methods/difference-difference
  28. https://dimewiki.worldbank.org/Difference-in-Differences
  29. https://pmc.ncbi.nlm.nih.gov/articles/PMC11305929/
  30. https://www.betterevaluation.org/methods-approaches/methods/regression-discontinuity
  31. http://www.asasrms.org/Proceedings/y2008/Files/301149.pdf
  32. https://www.urban.org/research/data-methods/data-analysis/quantitative-data-analysis/impact-analysis/quasi-experimental-0
  33. https://pmc.ncbi.nlm.nih.gov/articles/PMC6086368/
  34. https://bmcmedresmethodol.biomedcentral.com/articles/10.1186/s12874-023-02058-5
  35. https://www.sciencedirect.com/science/article/pii/S1048984323000048
  36. https://jech.bmj.com/content/76/11/960
  37. https://pmc.ncbi.nlm.nih.gov/articles/PMC8020490/
  38. https://pmc.ncbi.nlm.nih.gov/articles/PMC10036082/
  39. https://www.pnas.org/doi/10.1073/pnas.93.23.13416
  40. https://sph.unc.edu/wp-content/uploads/sites/112/2015/07/nciph_ERIC13.pdf
  41. https://pmc.ncbi.nlm.nih.gov/articles/PMC5130591/
  42. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_attrition_v2.1.pdf
  43. https://homvee.acf.gov/sites/default/files/2019-06/HomVEE-Attrition-White_Paper-7-2015.pdf
  44. https://opa.hhs.gov/sites/default/files/2020-07/attrition-ta-brief.pdf
  45. https://catalogofbias.org/biases/attrition-bias/
  46. https://pubmed.ncbi.nlm.nih.gov/20573482/
  47. https://www.sciencedirect.com/science/article/abs/pii/S0895435610000442
  48. https://pmc.ncbi.nlm.nih.gov/articles/PMC1444839/
  49. https://pmc.ncbi.nlm.nih.gov/articles/PMC7144753/
  50. https://evalf20.classes.andrewheiss.com/assignment/04-eval-threats/
  51. https://sambodhi.co.in/internal-validity-in-impact-evaluation-overview-importance-and-ways-for-improvement/
  52. https://www.nyudri.org/assets/publications/2015/siteselection.pdf
  53. https://www.tandfonline.com/doi/abs/10.1198/016214504000001880
  54. https://www.causalconversations.com/post/po-introduction/
  55. https://academic.oup.com/biomet/article-abstract/70/1/41/240879
  56. https://naijialiu.github.io/Gov_51/reading/IV_AngristPischke.pdf
  57. https://www.sciencedirect.com/science/article/pii/S0304407607001091
  58. https://www.nber.org/system/files/working_papers/w13039/w13039.pdf
  59. https://economics.mit.edu/research/publications/how-much-should-we-trust-differences-differences-estimates
  60. https://www.nber.org/system/files/working_papers/w8841/w8841.pdf
  61. https://pmc.ncbi.nlm.nih.gov/articles/PMC12499922/
  62. https://www.3ieimpact.org/blogs/new-3ie-handbook-measuring-cost-effectiveness-impact-evaluation
  63. https://cega.berkeley.edu/article/sounds-good-but-what-will-it-cost-making-the-case-for-rigorous-costing-in-impact-evaluation-research/
  64. https://documents1.worldbank.org/curated/en/862091571145787913/pdf/Integrating-Value-for-Money-and-Impact-Evaluations-Issues-Institutions-and-Opportunities.pdf
  65. https://blogs.worldbank.org/en/impactevaluations/why-don-t-economists-do-cost-analysis-their-impact-evaluations
  66. https://escholarship.org/uc/item/31x6m0d9
  67. https://www.betterevaluation.org/methods-approaches/approaches/randomised-controlled-trial
  68. https://www.princeton.edu/~deaton/downloads/Instruments_of_Development.pdf
  69. https://acf.gov/sites/default/files/documents/opre/methodsmeetingsummary2016_final_112017_508.pdf
  70. https://pmc.ncbi.nlm.nih.gov/articles/PMC10795211/
  71. https://pmc.ncbi.nlm.nih.gov/articles/PMC7081045/
  72. https://bookdown.org/mike/data_analysis/sec-quasi-experimental.html
  73. https://www.ahajournals.org/doi/10.1161/CIRCOUTCOMES.123.010078
  74. http://www.jakebowers.org/ITVExperiments/Imbens2010%257Bimbens2010better%257D.pdf
  75. https://pmc.ncbi.nlm.nih.gov/articles/PMC6019115/
  76. https://bmchealthservres.biomedcentral.com/articles/10.1186/s12913-022-08657-0
  77. https://www.cgdev.org/sites/default/files/should-randomistas-continue-rule-revised-jan-2019.pdf
  78. https://www.povertyactionlab.org/
  79. https://pmc.ncbi.nlm.nih.gov/articles/PMC5907717/
  80. https://www.iied.org/sites/default/files/pdfs/migrate/17404IIED.pdf
  81. https://implementationscience.biomedcentral.com/articles/10.1186/s13012-014-0115-y
  82. https://www.canada.ca/en/treasury-board-secretariat/services/audit-evaluation/evaluation-government-canada/theory-based-approaches-evaluation-concepts-practices.html
  83. https://educationalresearchtechniques.com/2025/10/08/postpositivist-paradigm-and-program-evaluation/
  84. https://thomasmtaston.medium.com/where-are-the-shy-positivists-193b6aeb6769
  85. https://www.sciencedirect.com/science/article/abs/pii/0149718989900463
  86. https://www.sciencedirect.com/science/article/pii/S0277953617307359
  87. https://blogs.worldbank.org/en/impactevaluations/the-ethics-of-a-control-group-in-randomized-impact-evaluations-the-start-of-an-ongoing-discussion
  88. https://www.princeton.edu/~deaton/downloads/Deaton_Cartwright_RCTs_with_ABSTRACT_August_25.pdf
  89. https://wagner.nyu.edu/files/faculty/publications/RFN12_Introduction_to_Impact_Evaluations_with_Randomized_Designs.pdf
  90. https://www.project-syndicate.org/commentary/evidence-based-policy-problems-by-ricardo-hausmann-2016-02
  91. https://academic.oup.com/policyandsociety/article/29/2/77/6420789
  92. https://www.princeton.edu/~deaton/downloads/deaton%2520instruments%2520randomization%2520learning%2520about%2520development%2520jel%25202010.pdf
  93. https://www.povertyactionlab.org/policy-insight/microcredit-impacts-and-promising-innovations
  94. https://openknowledge.worldbank.org/entities/publication/22114363-b9f9-53ea-a339-589b6749719a
  95. https://www.nber.org/system/files/working_papers/w25356/w25356.pdf
  96. https://www.aeaweb.org/articles?id=10.1257/jel.20151233
  97. https://www.povertyactionlab.org/evaluation/impact-progresa-health-mexico
  98. https://pmc.ncbi.nlm.nih.gov/articles/PMC2846434/
  99. https://www.nber.org/papers/w32779
  100. https://pmc.ncbi.nlm.nih.gov/articles/PMC5841841/
  101. https://www.evidenceaction.org/programs/deworm-the-world
  102. https://www.3ieimpact.org/evidence-hub/publications/replication-papers/reanalysis-health-and-educational-impacts-school-based
  103. https://economics.mit.edu/sites/default/files/publications/Six%2520Randomized%2520Evaluations%2520of%2520Microcredit.pdf
  104. https://voxdev.org/voxdevlit/microfinance-issue-3/first-generation-microcredit-rcts
  105. https://www.nber.org/system/files/working_papers/w27554/w27554.pdf
  106. https://documents1.worldbank.org/curated/en/694141488365161847/pdf/112413-JRN-PUBLIC-WBER-2014-28-3-evaluation-of-development-programs-randomized-controlled-trials-or-regressions.pdf
  107. https://gsdrc.org/topic-guides/public-sector-institutional-reform/outcomes/impact-of-institutional-reform-on-development-outcomes/
  108. https://economics.mit.edu/files/792
  109. https://www.researchgate.net/publication/357576047_THE_IMPACT_EVALUATION_OF_THE_INSTITUTIONAL_REFORMS_OF_THE_ONE-STOP_SERVICE_OSS_ON_ECONOMIC_PERFORMANCE_IN_INDONESIA
  110. https://pmc.ncbi.nlm.nih.gov/articles/PMC9168920/
  111. https://www.tandfonline.com/doi/full/10.1080/24751979.2024.2412333
  112. https://pubmed.ncbi.nlm.nih.gov/37131927/
  113. https://www.povertyactionlab.org/j-pal-courses
  114. https://poverty-action.org/impact
  115. https://poverty-action.org/publication/resources-and-tools-impact-evaluation
  116. https://poverty-action.org/publication/identifying-when-why-and-how-use-impact-evaluations
  117. https://www.worldbank.org/en/programs/sief-trust-fund
  118. https://www.ifpri.org/blog/ifpri-and-causal-impact-evaluation-evidence-for-real-life-policies/
  119. https://www.cgdev.org/blog/reinvigorating-impact-evaluation-global-development
  120. https://lantpritchett.org/wp-content/uploads/2019/12/RCTs-and-the-big-questions_10000words_june30.pdf
  121. https://www.brookings.edu/articles/randomized-control-trials-for-development-three-problems/
  122. https://lantpritchett.org/wp-content/uploads/2019/01/BsGfacultyseminarnovember2018.pdf
  123. https://onlinelibrary.wiley.com/doi/abs/10.1111/rode.12996
  124. https://arxiv.org/abs/1903.00402
  125. https://pmc.ncbi.nlm.nih.gov/articles/PMC7080394/
  126. https://direct.mit.edu/rest/article-pdf/doi/10.1162/rest_a_01592/2513975/rest_a_01592.pdf
  127. https://pmc.ncbi.nlm.nih.gov/articles/PMC5111584/
  128. https://lambdapartners.org/emerging-trends-in-impact-evaluation-7-innovative-approaches-to-assessing-development-projects/
  129. https://ieg.worldbankgroup.org/evaluations/leveraging-imagery-data-evaluations
  130. https://pmc.ncbi.nlm.nih.gov/articles/PMC8354555/
  131. https://www.3ieimpact.org/sites/default/files/2021-01/3ie-NLT-Marketing-flyer.pdf
  132. https://www.3ieimpact.org/media/videos/state-play-big-data-impact-evaluation-3ie-big-data-systematic-map
  133. https://pubs.cgdev.org/evidence-to-impact/2-recommendations/recommendation-2-digital-transformation/index.html
  134. https://www.journals.uchicago.edu/doi/10.1086/729447
  135. https://www.tandfonline.com/doi/full/10.1080/00220388.2023.2284673
  136. https://pmc.ncbi.nlm.nih.gov/articles/PMC3909454/
  137. https://www.frontiersin.org/journals/communication/articles/10.3389/fcomm.2025.1632305/full
  138. https://health-policy-systems.biomedcentral.com/articles/10.1186/s12961-017-0192-x
  139. https://ideas.repec.org/a/eee/wdevel/v127y2020ics0305750x19304668.html
  140. https://blogs.worldbank.org/en/developmenttalk/implementing-successful-small-interventions-large-scale-hard
  141. https://lantpritchett.org/wp-content/uploads/2021/05/taking-the-con-out-of-rct_may2021.pdf
  142. https://abtgovernance.com/2020/04/21/if-randomised-control-trials-rcts-improve-global-development-outcomes-why-then-are-we-still-fighting-about-them/
  143. https://www.sciencedirect.com/science/article/abs/pii/S0305750X19304668
Add your contribution
Related Hubs
User Avatar
No comments yet.