Loading.....

Frequently Asked Questions (FAQs) about impact evaluations

 1. What are the objectives of impact evaluations in the GEF Evaluation Office?

 

Impact evaluation in the GEF Evaluation Office seeks to determine the long-term effects of GEF support, how these were achieved and what can be done to strengthen them. Such evaluation work is part of the overall exploration of the results achieved by the GEF and has a strong focus on learning lessons from experience. Results-management systems of the GEF and its implementing partners have a major emphasis and knowledge base on the outcomes achieved by the end of GEF financial support to projects. This information is of great importance, but does not tell us whether the long-term global environmental benefits, to which the GEF is expected to contribute, have been or are likely to be achieved. 

 

2. What is the approach undertaken by the GEF Evaluation Office?

  

Impact evaluation has several major schools of thought and the Evaluation Office has adopted a “mixed method” approach, which draws on the strengths of each. It has undertaken work of an experimental and quasi-experimental nature, as well adopting a core methodology, which is based on a “Theory of Change” approach.

 

3. What was the process from the development of the theory-based approach to impact evaluation?

 

The approach was initially developed and implemented at field level through an evaluation of the long-term effects (impacts) of three GEF-supported projects in East Africa, which sought to enhance biodiversity through the improved operation of protected areas.
 
Fieldwork was conducted to explore the extent to which the anticipated long-term changes had occurred, based on theories that established in detail how such impacts were intended to be delivered or catalyzed through GEF support. As a first step in the fieldwork, the theories were verified and refined by local stakeholders, who had been involved in project implementation. In addition to generating a reliable assessment of what impacts had been achieved and how, the theory-based methodology was found to lend itself to a generic approach, which could form the basis of impact evaluation in all focal areas in which the GEF operates. Furthermore, this approach could be applied at several levels, ranging from full field-level impact evaluation to a desk-review process based on existing project terminal evaluations. This opened up the possibility of exploring impacts portfolio-wide.
 

4. How was the approach mainstreamed into the Evaluation Office work?

 

Based on the results and methodological development of the initial set of impact evaluations, the approach was mainstreamed into the work of the Office, initially into the results analysis of the Fourth Overall Performance Study of the GEF (OPS4). In OPS4, the overall impact evaluation work was completed at three levels: 
 
  • a detailed fieldwork-based impact evaluation of the GEF portfolio of support to the Phase Out of Ozone Depleting Substances in Countries with Economies in Transition. This covered activities in eighteen countries 
  • a set of ten semi-detailed fieldwork-based evaluations of projects completed some years previously, using a new theory-based methodology developed by the Evaluation Office known as the Review of Outcomes to Impacts (ROtI)
  • an analysis of the entire cohort of more than two hundred projects included in OPS4, based on the same ROtI methodology, but implemented as desk reviews, based on project terminal evaluations or completion reports and other available documents.

 

This three-tiered approach was found to be effective at addressing one of the fundamental difficulties of impact evaluation, namely the need to address both internal and external validity of evaluation evidence.

 

5. What are the differences between desk ROtIs and field ROtIs?

 

The desk ROtIs  provide a clear “broad brush” picture of the progress towards impacts of the portfolio, with a high degree of external validity, but on the basis relatively restricted detail on any specific project. Full impact evaluations, on the other hand, provide a very high degree of internal validity, based on extensive fieldwork, but cannot in the short to medium term be scaled up to generate an overview of the entire portfolio, which would have high external validity.
 

The field-Rotis bridge this gap to a considerable extent, by providing field-based evidence of impact on a number of projects in a cost-effective manner. The three approaches together were found to provide a clear initial picture of progress towards impacts of the overall GEF-portfolio, the details of which can be filled in over time through additional full impact evaluations, whilst keeping the overview provided by the ROtI approach. This overall stream of theory-based work continues to be supported by a number of explorations of impacts achieved, using experimental and quasi-experimental approaches.

 

6. What are some of the key findings that emerged from the Impact Evaluation work?

 

The findings emerging from the impact evaluation work are many and complex. They are summarized in the Annual Reports on Impact presented to the GEF Council and examined in detail in the Information documents listed below. Some highlights are as follows:
 
  • the GEF portfolio has made a number of substantial and verifiable contributions to achieving its intended global environment objectives
  • the initial results of these contributions can be clearly identified at the time of completion of GEF projects, the great majority of which successfully deliver their intended outcomes in order for the short to medium-term outcomes of projects to deliver or contribute to their intended long-term global environment impacts, many activities need to continue after project completion.
  • projects which are most successful in terms of impacts have ensured that the systems and resources necessary for the continuation of activities are already in place and functioning before project completion
  • the continuation of activities is ultimately dependent on strong long-term national commitment to achieving the environmental objectives.