Definitions Monitoring is the continuous assessment of project implementation in relation to agreed schedules, and of the use of inputs, infrastructure, and services by project beneficiaries. Monitoring: Provides managers and other stakeholders with continuous feedback on implementation. Identifies actual or potential successes and problems as early as possible to facilitate timely adjustments to project operation.
Evaluation is the periodic assessment of a project's relevance, performance, efficiency, and impact (both expected and unexpected) in relation to stated objectives: Project managers undertake interim evaluations during implementation as a first review of progress, a prognosis of a project's likely effects, and as a way to identify necessary adjustments in project design. The use of mid-term reviews of ongoing projects has spread quickly in the last decade. Terminal evaluations, conducted at the end of a project, are required for project completion reports. They include an assessment of a project's effects and their potential sustainability.
Need for effective monitoring and evaluation Monitoring and evaluation (M&E) is increasingly recognized as an indispensable tool of both project and portfolio management. The acknowledged need to improve the performance of development assistance calls for close attention to the provision of management information, both to support the implementation of projects and programs and to feed back into the design of new initiatives. M&E also provides a basis for accountability in the use of development resources. Given the greater transparency now expected of the development community, governments and agencies assisting them need to respond to calls for more "success on the ground" with examples of development impact and with evidence that they have systems in place that support learning from experience. Used carefully at all stages of the project cycle, monitoring and evaluation can help to strengthen project design and implementation and stimulate partnership with project stakeholders. It can: Influence sector assistance strategy. Relevant analysis from project and policy evaluation can highlight the outcomes of previous interventions, and the strengths and weaknesses of their implementation. Improve project design. Use of project design tools such as the logframe (logical framework) results in systematic selection of indicators for monitoring project performance. The process of selecting indicators for monitoring is a test of the soundness of project objectives and can lead to improvements in project design. Incorporate views of stakeholders. Awareness is growing that participation by project beneficiaries in design and implementation brings greater "ownership" of project objectives and encourages the sustainability of project benefits. Ownership brings accountability. Objectives should be set and indicators selected in consultation with stakeholders, so that objectives and targets are jointly "owned". The emergence of recorded benefits early on helps reinforce ownership, and early warning of emerging problems allows action to be taken before costs rise. Show need for mid-course corrections. A reliable flow of information during implementation enables managers to keep track of progress and adjust operations to take account of experience.
M&E design: five components Good M&E design during project preparation is a much broader exercise than just the development of indicators. Good design has five components, discussed in turn in what follows: Clear statements of measurable objectives for the project and its components, for which indicators can be defined. A structured set of indicators, covering outputs of goods and services generated by the project and their impact on beneficiaries. Provisions for collecting data and managing project records so that the data required for indicators are compatible with existing statistics, and are available at reasonable cost. Institutional arrangements for gathering, analyzing, and reporting project data, and for investing in capacity building, to sustain the M&E service. Proposals for the ways in which M&E findings will be fed back into decision making.
These five components help to ensure that M&E is relevant, within the capacity of the borrower's institutions, and used to good effect. Project objectives Projects are designed to further long-term sectoral goals, but their immediate objectives, at least, should be readily measurable. Thus, for example, a health project might be designed to further the sectoral goals of a reduction in child mortality and incidence of infectious diseases, but have an immediate, measurable objective of providing more equitable access to health services. Objectives should be specific to the project interventions, realistic in the timeframe for their implementation, and measurable for evaluation. India's District Primary Education Project, for example, set out its objectives at the district level in clear statements linked directly to indicators: Capacity building: District sub-project teams would be fully functional, implementing sub-project activities and reporting quarterly on progress. In-service teams would be functioning, with augmented staff and equipment, providing support for planning and management, teacher in-service training, development of learning materials, and program evaluation. Reducing dropout and improving learning achievement: School/community organizations would be fully functional for at least half the schools, and dropout rates would be reduced to less than 10 percent. Learning achievements in language and mathematics in the final year of primary school would be increased by 25 percent over baseline estimates. Improving equitable access. Enrollment disparities by gender and caste would be reduced to less than 5 percent.
By asking how objectives can be measured (for example, what are suitable indicators for equity of access?), and how component activities lead to those objectives, project objectives will be more precisely defined. In this way M&E design contributes directly to the quality of projects at entry to the portfolio. The quality of a new project also depends on ownership by stakeholders and awareness of the scope and limitations of project actions. Monitoring may need to include information about supporting actions that are likely to affect the success of the project. Problem analysis is a tool that can help to identify such actions (see Box 1).
Structured indicators Indicators provide the quantitative and qualitative detail to a set of objectives. They are statements about the situation that will exist when a objective is reached. The ability to define an indicator in consultation with stakeholders, and agree the target value and timing, is a demonstration that project objectives are clearly stated, are understood, and are supported. This agreement brings accountability. Project objectives Types of indicator Goal Impact Purpose Outcomes or effects Outputs Outputs Activities Process Inputs Inputs
To make information on progress available right from the early stages of implementation and throughout the project cycle, indicators need to be structured. The logframe approach to project design provides an efficient structure by postulating a hierarchy of objectives for which indicators are required. Its terminology needs to be understood in the context of the sector to which it is being applied, but the table below illustrates the most widely used terms.
Starting at the bottom of the table, input indicators are quantified and time-bound statements of resources to be provided. Information on these indicators comes largely from accounting and management records. Input indicators are often left out of discussions of project monitoring, though they are part of the management information system. A good accounting system is needed to keep track of expenditures and provide cost data for performance analysis of outputs. Input indicators are used mainly by managers closest to the tasks of implementation, and are consulted frequently, as often as daily or weekly. Examples: vehicle operating costs for the crop extension service; levels of financial contributions from the government or cofinanciers; appointment of staff; provision of buildings; status of enabling legislation. Process indicators measure what happens during implementation. Often, they are tabulated as a set of contracted completions or milestone events taken from an activity plan. Examples: Date by which building site clearance must be completed; latest date for delivery of fertilizer to farm stores; number of health outlets reporting family planning activity; number of women receiving contraceptive counseling; status of procurement of school textbooks.
Box 1: Problem analysis Project objectives need to be structured to match the analysis of problems that the project is trying to overcome. Problem analysis is a brainstorming technique by which stakeholders identify the causes and effects of problems. A summary of the analysis can be visualized in a tree diagram. Project objectives are then structured to resolve those problems and can be represented as a mirror image of the problem tree diagram. Problems which the project cannot deal with directly, but which must be resolved if objectives are to be achieved, become topics for other strategies, or risks to the project if no action is taken. Action that is needed for project benefits to be achieved, but cannot be taken directly by the project, must be monitored as closely as project activities and may even be made the subject of loan conditionality. Examples would include changes in legislation or action by other government agencies. These examples would appear as "Important assumptions" in the logframe.
Output indicators show the immediate physical and financial outputs of the project: physical quantities, organizational strengthening, initial flows of services. They include performance measures based on cost or operational ratios. Examples: Kilometers of allweather highway completed by the end of September; percentage of farmers attending a crop demonstration site before fertilizer top-dressing; number of teachers trained in textbook use; cost per kilometer of road construction; crop yield per hectare; ratio of textbooks to pupils; time taken to process a credit application; number of demonstrations managed per extension worker; steps in the process of establishing water users" associations. Impact refers to medium or long-term developmental change. (Some writers also refer to a further class of outcome indicators, more specific to project activities than impact indicators, which may be sectoral statistics, and deal more with the direct effect of project outputs on beneficiaries.) Measures of change often involve complex statistics about economic or social welfare and depend on data that are gathered from beneficiaries. Early indications of impact may be obtained by surveying beneficiaries" perceptions about project services. This type of leading indicator has the twin benefits of consultation with stakeholders and advance warning of problems that might arise. Examples of impact: (health) incidence of low birth weight, percentage of women who are moderately or severely anemic; (education) continuation rates from primary to secondary education by sex, proportion of girls completing secondary education; (forestry) percent decrease in area harvested, percent increase in household income through sales of wood and nonwood products. Examples of beneficiary perceptions: proportion of farmers who have tried a new variety of seed and intend to use it again; percentage of women satisfied with the maternity health care they receive. (See Box 2.)
Box 2: An example of structured indicators The following example from the Philippines comes from a component to reduce the prevalence of cervical cancer as part of the Women's Health and Safe Motherhood Project. Inputs Number of cryotherapy machines procured and delivered Quantities of drugs, reagents, and medical supplies procured and delivered Number of healthcare providers trained in cervical cancer management by type of provider and training Proportion of healthcare providers able to provide Pap smears, cervical biopsies, hysterectomy and cryotherapy, by type of provider Number and staff months of technical assistance provided Process Percentage of women over 35 who have had at least one Pap smear Number of cases detected Number of cervical biopsies performed for suspicious lesions Number of cryotherapy procedures performed for early invasive cancer Number of simple and radical hysterectomies performed Output Proportion of women with access to cervical management services Percentage of women who received effective cancer treatment according to protocols/ standards Impact Decline in: incidence of invasive cancer deaths due to cervical cancer
Classifying project objectives according to their level draws attention to the fact that management will need to develop systems to provide information at all levels, from basic accounting through to statistics about project objectives. Box 3 shows the nature and location of responsibility for indicators at each level in the logframe. Box 3: Indicators in a hierarchy Level of objectives Nature of the indicators Responsibility
Goal Long-term statistical evidence National/ sectoral agencies of project impact
Purpose Social and economic surveys of Project/ independent evaluators project outcomes and effects Leading indicators giving Project staff advance warning of beneficiary perceptions and response to the project
Outputs Management records Project staff Internal reporting Activities Task management of Project staff project processes Financial accounts Management records of progress
Inputs Financial accounts Project staff Management records of resources available and used
The control of activities and their direct results or outputs is within the management of the project, and can largely be dealt with by internal record-keeping and analysis. Indicators of inputs, process, and outputs are mostly generated from within project management. By contrast, the achievement of project objectives normally depends on how project beneficiaries respond to the goods or services delivered by the project. Evidence of their response and the benefits they derive requires consultation and data collection that may be outside the scope of management. It is important to identify how beneficiaries are expected to respond to project services, because managers will need evidence of that response if they are to modify their activities and strategy. Indications that beneficiaries have access to, are using, and are satisfied with project services give early indication that the project is offering relevant services and that direct objectives are likely to be met. Such evidence--market research--may be available sooner and more easily than statistics of impact such as changes in health status or improvements in income. Market research information is an example of a leading indicator of beneficiary perceptions that can act as a proxy for later, substantive impact. Other leading indicators can be identified to give early warning about key assumptions that affect impact. Examples would include price levels used for economic analysis, passenger load factors in transport projects, and adoption of healthcare practices. When planning the information needs of a project there is a difference between the detail needed for day-to-day management by the implementing agency or, later, for impact evaluation, and the limited number of key indicators needed to summarize overall progress in reports to higher management levels. For example, during construction of village tubewells, project managers will need to keep records about the materials purchased and consumed, the labor force employed and their contracting details, the specific screen and pump fitted, the depth at which water was found, and the flow rate. The key indicators however, might be just the number of wells successfully completed and their average costs and flow rates. Exogenous indicators are those that cover factors outside the control of the project but which might affect its outcome, including risks (parameters identified during economic, social, or technical analysis, that might compromise project benefits); and the performance of the sector in which the project operates. Concerns to monitor both the project and its wider environment call for a data collection capacity outside the project and place an additional burden on the project's M&E effort. A recent example of a grain storage project in Myanmar demonstrates the importance of monitoring risk indicators. During project implementation, policy decisions about currency exchange rates and direct access by privately owned rice mills to overseas buyers adversely affected the profitability of private mills. Management would have been alerted to the deteriorating situation had these indicators of the enabling environment been carefully monitored. Instead, a narrow focus on input and process indicators missed the fundamental change in the assumptions behind the project. The relative importance of indicators is likely to change during the implementation of a project, with more emphasis on input and process indicators at first, shifting to outputs and impact later on. This is a distinction between indicators of implementation progress and indicators of development results. Data collection Project field records. Indicators of inputs and processes will come from project management records originating from field sites. The quality of record keeping in the field sets the standard for all further use of the data and merits careful attention. M&E designers should examine existing record-keeping and the reporting procedures used by the project authorities to assess the capacity to generate the data that will be needed. At the same time, they should explain how and why the indicators will be useful to field, intermediate, and senior levels of project management. The design of field records about, say, farmers in extension groups, people attending a clinic, or villagers using a new water supply, will affect the scope for analysis later. The inclusion of simple socioeconomic characteristics such as age and sex may significantly improve the scope for analysis. A good approach is to structure reporting from the field so that aggregates or summaries are made at intermediate stages. In this way, field staff can see how averages or totals for specific villages or districts enable comparisons to be drawn and fieldwork improved. Surveys and studies. To measure output and impact may require the collection of data from sample surveys or special studies (including, where appropriate, participatory methods). Studies to investigate specific topics may call for staff skills and training beyond those needed for regular collection of data to create a time series. Where there is a choice, it is usually better to piggyback project-specific regular surveys on to existing national or internationally supported surveys than to create a new data collection facility. Special studies may be more manageable by a project unit directly, or subcontracted to a university or consultants. If the special studies are to make comparisons with data from other surveys it is vital that the same methods be used for data collection (see below). In the project plan, proposals to collect data for studies should include a discussion of: the objectives of the study or survey; the source of data ; choices and proposed method of collection; and likely reliability of the data.
Data comparability. Some desired indicators of impact, such as mortality rates, school attendance, or household income attributable to a project, may involve comparisons with the situation before the project, or in areas not covered by the project. Such comparisons may depend on the maintenance of national systems of vital statistics, or national surveys. Before data from such sources are chosen as indicators of project impact the designer needs to confirm that the data systems are in place and reliable and that the data are valid for the administrative area in question and for any control areas. Potential problems in making comparisons with existing data include incomplete coverage of the specific project area; the use of different methods to collect data, such as interviewing household members in one survey and only household heads in another; and changes in techniques such as measuring crop output in one survey and collecting farmers' estimates in another. Problems such as these can invalidate any comparison intended to show changing performance. To give the comparability needed for evaluation, study proposals should explain and justify the proposed approach and ensure consistency in methods. The complexity of the statistics and problems of attributing causality mean that often it is more appropriate to use the delivery of services and beneficiary response as proxy indicators than to attempt to measure impact (see Box 4).
Box 4: Limitations of survey data Agricultural change Writing in the context of agriculture projects, Casley and Kumar (1987) examined the scope for measuring change and showing that the cause of the change was project activities. They concluded that for indicators such as crop yields in the variable conditions of smallholder rainfed farming it would be necessary to maintain a consistent, high quality time series for at least nine years to be confident of demonstrating a rising trend of up to 12 percent a year. Even with such a data set, the ability to make inferences about cause and effect will be affected by exogenous influences on the farmer. These cannot be held constant or isolated from project activities. It is unlikely that survey data would permit rigorous causal evaluation. They conclude that inferences from case studies are likely to be more effective. Poverty and household income Difficulties in making comparisons between a Household Budget Survey carried out in 1989/90, and a later Integrated Household Survey in 1992/93, for Uganda, show that even under conditions of close supervision and rigorous design, small changes in the way in which questions about household consumption are put, the layout of the survey form, and guidance given to enumerators can undermine comparability. (See Appleton, 1996.) Designers of M&E surveys need to make special provision for comparability with existing data from project baseline or national surveys by using common survey instruments and methods. The idea that comparisons can be made between data collected using different methods is unlikely to pay off.
Participatory methods of data collection can bring new insights into peoples' needs for project planning and implementation, but are no less demanding on skills than questionnaire surveys. They are time-consuming and require substantial talent in communication and negotiation between planners and participants (see Box 5).
Box 5: Measuring impact in several ways Proposals to evaluate the Agriculture Sector Investment Program (ASIP) in Zambia include a twin-track approach to measuring impact: A quantitative survey program will be used to estimate the program's impact on production and incomes. This will be managed by the Central Statistical Office (CSO) from their annual random sample of 5,000 small- and medium-scale farmers. Additional questionnaires will be used to obtain farmers' assessments of ASIP subprograms. Systematic client consultation will be used to elicit feedback from the intended beneficiaries on the effectiveness of the design of project subcomponents. The University of Zambia will manage the impact evaluations and will also analyze data on agricultural sector performance from CSO and other secondary data.
Institutional arrangements; capacity building Good M&E should develop the capacity of the borrower and build on existing systems. Capacity building is widely acknowledged to be important but is often poorly defined. It means: upgrading skills in monitoring and evaluation, which include project analysis, design of indicators and reporting systems, socioeconomic data collection, and information management; improving procedures, to create functional systems that seek out and use information for decisions; and strengthening organizations to develop skilled staff in appropriate positions, accountable for their actions.
The M&E capacity requirements of the project should be designed in the context of the evaluation capacity needs of sectoral and national institutions in the country concerned (see Lessons & Practices No. 4, "Building Evaluation Capacity"). Virtually all implementing agencies will have existing reporting systems. The M&E design should aim to build on those arrangements but develop further the technical skills required to plan information needs, design data collection, execute studies and surveys, analyze the data, and report results in a format that is relevant to management. Most M&E units have a quick-response capability to investigate implementation problems or disappointing beneficiary responses, and advise on rapid correction measures. An assessment of the flow of information and degree of detail needed by each level of management will help to clarify the indicators that need to be measured. Many operational decisions are made at field or regional levels and the flow of reporting should cater to those needs. For complex projects, or where there is no reporting system, report proformas will need to be designed. The agency managing the project should also distinguish between the type of information required for its own internal management and for reporting to higher levels of government and to the Bank. Because evaluation is concerned mainly with impact, which will be measurable towards the end of implementation or in later years, it is often better done by a separate agency, independent from implementation. Monitoring, however, is a tool of good management, and the responsible unit should be located close to project management. The resources, training, and technical assistance for the unit should be specified. In many projects, management will want to integrate monitoring with other systems such as financial accounting and computerized project management. This wider concern with the management information system is likely to be of more immediate value to management than donor-driven monitoring and should be supported in the project plan. Feedback An indication needs to be given at the design stage about the proposed use of M&E information, beyond Bank formalities such as mid-term or completion reviews. A good flow of information is closely linked to the development of accountability within the project, sector, government, and Bank. In many countries, information on projects and programs is poor and difficult to access, and the mechanisms for feedback are weak or non existent. The highest payoffs to evaluation arise at the policy and program level, but project-level evaluation offers an easier and less sensitive starting point in many countries. The uses of the information can be structured and scheduled according to the needs of the participants: Project management will need to monitor expenditure and progress against bar chart schedules, weekly and at least monthly. Outputs are unlikely to be measurable at less than three-monthly intervals, and some may need much longer. Consultations with beneficiaries, or surveys of their satisfaction with project services, should be timed to supply information to use in planning project activities. The time period for reporting may vary with the level of management: for example, monthly at district level, quarterly at region or state. Some flows of information need to be timed to fit into national budget planning activities. Annual funding may depend on the results from previous work. Periodic and mid-term reviews provide milestones by which information has to be ready.
In projects where operating performance standards are quoted as an objective, or where decentralized processes call for localized capacity to plan and manage work programs and budgets, designers will need to describe how and when M&E findings will be used to shape work plans and contribute to program or policy development. In Mexico, for example, the Second Decentralization and Regional Development Project plans to incorporate monitoring of implementation into its regular management procedures. Annual plans are to be prepared for each component, including an element of institutional development, and these will form the basis of annual monitoring. The analysis of implementation will depend on the functioning of a central database about sub-projects, created in each state from standardized data sheets. The database will produce the reports required for the project approval procedures, giving an incentive to field staff to use the system. Results from the implementation database will be analyzed in order to target field reviews and a mid-term review. The project has no specific monitoring and evaluation unit. Instead, each management sub-unit responsible for technical oversight of a component is responsible for ensuring the quality and timeliness of data collection, and for producing and analyzing reports. These reports will be presented by project component and be used to help diagnose technical and institutional implementation issues, propose and conduct studies, and plan institutional development and training. Experience with implementation Even with a good design for M&E, the Bank's experience shows that success during implementation depends heavily on a sense of ownership by the borrower, adequate capacity in borrower institutions, and sustained interest from the task and project managers throughout the life of the project. Two factors are important here. One is that the borrower's sense of ownership of the project provides a stimulus to transparent management and good information about progress. The other is that often borrowers doubt the value of adopting what may be costly and time consuming procedures to collect, analyze, and report information. In such circumstances sound design is especially important, with monitoring information providing a clear input to management decision making and, often, an emphasis on the early gains to be had from monitoring and on institutional procedures that encourage the use of monitoring data to trigger further implementation decisions.
Suggested reading Appleton, Simon, "Problems in Measuring Changes in Poverty over Time," IDS Bulletin Vol. 27 No 1, 1996. Brighton, UK: Institute for Development Studies. Casley, Dennis J. and Krishna Kumar, Project Monitoring and Evaluation in Agriculture. Washington, D.C.: World Bank, 1987. Operations Evaluation Department, World Bank, "Building Evaluation Capacity", Lessons & Practices No. 4, November 1994. --- "Monitoring and Evaluation Plans in Staff Appraisal Reports in fiscal year 1995". Report No. 15222, December 1995.
|For more information on this subject or to receive information on other Operations Evaluation Department products, please contact the Operations Evaluation Department at: firstname.lastname@example.org, Tel: (202) 458-4497, Partnerships and Knowledge Programs, World Bank Group, 1818 H Street, Rm. H3-304, Washington, DC 20433 or visit our website at: http://www.worldbank.org/oed/ ||
© Copyright 2001 OED, All Rights Reserved