Monitoring is a form of evaluation or assessment, though unlike outcome or impact evaluation, it takes place shortly after an intervention has begun (formative evaluation), throughout the course of an intervention (process evaluation) or midway through the intervention (mid-term evaluation).
Monitoring is not an end in itself. Monitoring allows programmes to determine what is and is not working well, so that adjustments can be made along the way. It allows programmes to assess what is actually happening versus what was planned.
Monitoring allows programmes to:
- Implement remedial measures to get programmes back on track and remain accountable to the expected results the programme is aiming to achieve.
- Determine how funds should be distributed across the programme activities.
- Collect information that can be used in the evaluation process.
When monitoring activities are not carried out directly by the decision-makers of the programme it is crucial that the findings from those monitoring activities are coordinated and fed back to them.
Information from monitoring activities can also be disseminated to different groups outside of the organization which helps promote transparency and provides an opportunity to obtain feedback from key stakeholders.
There are no standard monitoring tools and methods. These will vary according to the type of intervention and objectives outlined in the programme. Examples of monitoring methods include:
- Activity monitoring reports
- Record reviews from service provision (e.g. police reports, case records, health intake forms and records, others)
- Exit interviews with clients (Survivors)
- Qualitative techniques to measure attitudes, knowledge, skills, behavior and the experiences of survivors, service providers, perpetrators and others that might be targeted in the intervention.
- Statistical reviews from administrative databases (i.e. in the health, justice, interior sectors, shelters, social welfare offices and others)
- Other quantitative techniques.
Impact Evaluation
Impact evaluation assesses the changes that can be attributed to a particular intervention, such as a project, program or policy, both the intended ones, as well as ideally the unintended ones. In contrast to outcome monitoring, which examines whether targets have been achieved, impact evaluation is structured to answer the question: how would outcomes such as participants’ well-being have changed if the intervention had not been undertaken? This involves counterfactual analysis, that is, “A comparison between what actually happened and what would have happened in the absence of the intervention.” Impact evaluations seek to answer cause-and-effect questions. In other words, they look for the changes in outcome that are directly attributable to a program.
Impact evaluation helps people answer key questions for evidence-based policy making: what works, what doesn’t, where, why and for how much? It has received increasing attention in policy making in recent years in the context of both developed and developing countries. It is an important component of the armory of evaluation tools and approaches and integral to global efforts to improve the effectiveness of aid delivery and public spending more generally in improving living standards. Originally more oriented towards evaluation of social sector programs in developing countries, notably conditional cash transfers, impact evaluation is now being increasingly applied in other areas such as the agriculture, energy and transport.
Impact evaluation measures the difference between what happened with the programme and what would have happened without it. It answers the question, “How much (if any) of the change observed in the target population occurred because of the programme or intervention?”
Rigorous research designs are needed for this level of evaluation. It is the most complex and intensive type of evaluation, incorporating methods such as random selection, control and comparison groups.
These methods serve to:
- Establish causal links or relationships between the activities carried out and the desired outcomes.
- Identify and isolate any external factors that may influence the desired outcomes.
For example, an impact evaluation of an initiative aimed at preventing sexual assaults on women and girls in town x through infrastructural improvements (lighting, more visible walkways, etc.) might also look at data from a comparison community (town y) to assess whether reductions in the number of assaults seen at the end of the programme could be attributed to those improvements. The aim is to isolate other factors that might have influenced the reduction in assaults, such as training for police or new legislation.
While impact evaluations may be considered the “gold standard” for monitoring and evaluation, they are challenging and may not be feasible for many reasons, including:
- They require a significant amount of resources and time, which many organizations may not have.
- To be done properly, they also require the collection of data following specific statistical methodology, over a period of time, from a range of control and intervention groups, which may be difficult for some groups.
Impact evaluations may not always be called for, or even appropriate for the needs of most programmes and interventions looking to monitor and evaluate their activities.
- To measure programme impact, an evaluation is typically conducted at the start (known as a baseline) and again at the end (known as an endline) of a programme. Measurements are also collected from a control group with similar characteristics to the target population, but that is not receiving the intervention so that the two can be compared.
- Attributing changes in outcomes to a particular intervention requires one to rule out all other possible explanations and control for all external or confounding factors that may account for the results.
An evaluation of the impact of a campaign to raise awareness around the provisions of a recently enacted law on violence against women for example would need to incorporate:
- Baseline data on awareness of the law’s provisions prior to the campaign for the intervention group;
- Endline data on awareness of the law’s provisions after the campaign for the intervention group;
- Baseline data on awareness of the law’s provisions prior to the campaign for a closely matched control group not exposed to the campaign; and
- Endline data on awareness of the law’s provisions after the campaign for a closely matched control group not exposed to the campaign.
Endline data allows the programme to see if there were external/ additional factors that might influence the level of awareness among those not exposed to the campaign. If the study design does not involve a randomly-assigned control group, it is not possible to make a definitive statement regarding any differences in outcome between areas with the programme and areas without the programme.
However, if statistically rigorous baseline studies with randomly assigned control groups cannot be conducted, very useful and valid baseline information and endline information can still be collected.
Evaluation requires technical expertise and training. If the programme does not maintain the capacity in-house, external evaluators should be hired to assist.
There are five key principles relating to internal validity (study design) and external validity (generalizability) which rigorous impact evaluations should address: confounding factors, selection bias, spillover effects, contamination, and impact heterogeneity.
- Confounding occurs where certain factors, typically relating to socioeconomic status, are correlated with exposure to the intervention and, independent of exposure, are causally related to the outcome of interest. Confounding factors are therefore alternate explanations for an observed (possibly spurious) relationship between intervention and outcome.
- Selection bias, a special case of confounding, occurs where intervention participants are non-randomly drawn from the beneficiary population, and the criteria determining selection are correlated with outcomes. Unobserved factors, which are associated with access to or participation in the intervention, and are causally related to the outcome of interest, may lead to a spurious relationship between intervention and outcome if unaccounted for. Self-selection occurs where, for example, more able or organized individuals or communities, who are more likely to have better outcomes of interest, are also more likely to participate in the intervention. Endogenous program selection occurs where individuals or communities are chosen to participate because they are seen to be more likely to benefit from the intervention. Ignoring confounding factors can lead to a problem of omitted variable bias. In the special case of selection bias, the endogeneity of the selection variables can cause simultaneity bias.
- Spillover (referred to as contagion in the case of experimental evaluations) occurs when members of the comparison (control) group are affected by the intervention.
- Contamination occurs when members of treatment and/or comparison groups have access to another intervention which also affects the outcome of interest.
- Impact heterogeneity refers to differences in impact due by beneficiary type and context. High quality impact evaluations will assess the extent to which different groups (e.g., the disadvantaged) benefit from an intervention as well as the potential effect of context on impact. The degree that results are generalizable will determine the applicability of lessons learned for interventions in other contexts.