|
A key feature of successful organizations is their ability to learn from experience and to react to market or client responses to their actions. In an environment determined more and more by markets rather than by states, political institutions, the press, and the public are demanding more efficiency and responsiveness. The capacity to absorb information, assess performance, and respond flexibly is becoming as vital for governments as for private firms.
Evaluation agencies exist at least nominally in many developing countries, and national audit offices exist in almost all. Generally, though, the evaluation capacities do not play an important part in decision making. They could--and should. This issue of Lessons & Practices looks at the steps needed to build and benefit from evaluation capacity in the public sector in developing countries.
Evaluation can strengthen public sector management
Used effectively, evaluation capacity can play a critical role in four areas of a nation's public sector management:
- Influencing policy analysis and formulation. Careful analysis of the costs and benefits of existing policies is key to informed, tough-minded, policy analysis and formulation. In today's more open economies, policy consequences are more complex and sequencing issues more important. Objective evaluation also gives comfort to decision makers faced with tough political choices; it shows them what is and is not working, and why.
- Improving resource allocation and budgetary process. How efficiently are government revenues being spent? Will shifts in budgetary allocations increase efficiency? To answer such questions, governments need to base their resource allocation and budgetary processes solidly in the evaluation of public expenditures.
- Improving investment programs and projects. Evaluation of investment projects can help engender a performance-oriented culture within government agencies. The examination of completed projects is an exercise in public accountability, which is important in its own right. Beyond accountability, good ex post evaluation provides feedback to management--about weak points in budgetary processes, about the performance of public agencies, about management quality, and incentives in the public sector. There are also important links back to policy evaluation, for example with respect to criteria for and levels of external borrowing for public projects. All these factors make investment-level evaluation a potentially powerful instrument for learning from both success and failure, and for gaining insights into how public institutions need to change.
- Examining fundamental missions. A vital dimension of public sector management is the evaluation of institutions or the government itself: the assessment of the relevance, performance, and cost of government agencies and ministries, and of alternative ways of doing public business. Performance-based auditing is one instrument for such reappraisals. What is critical is a capacity for such evaluation independent of the mainstream bureaucracy but with access to it.
Evaluation in developing countries
In developing countries, demand for evaluation began to emerge in the mid-1980s. Several countries have evaluation units attached to planning, finance, or prime ministers' offices. The best examples of strong institutional mechanisms for evaluation are in East Asia (see Box 1: In Korea, an emphasis on operational performance). A movement to strengthen evaluation is underway in Latin America, led by Brazil, Chile, Colombia, and Mexico, inspired by severe resource constraints and public demands for greater accountability in the public sector.
But in most developing countries an evaluation culture--or the consistent use of feedback in formulating policies and allocating resources--is only incipient. Sensitivity to public criticism and the fear of political fallout from evaluation findings are inhibiting factors. Many countries still lack the essential requirements of effective evaluation: the quality of information and access to it is often poor, mechanisms for feedback into the decision making process are weak, and a culture of accountability is not firmly in place.
Current problems. Common constraints and problems are:
- Lack of interest and commitment to the evaluation function at the political level. This attitude is often then manifested at the bureaucratic level.
- Lack of feedback mechanisms for applying evaluation findings. This results in both lack of demand for evaluation results and lack of institutional links between those who carry out evaluation and those who need to use its findings.
- More attention to preparing and appraising programs and projects than to evaluating their performance on completion.
- Little involvement of national staff in evaluating externally financed programs and projects.
- Limited attention to the quality and timeliness of information and to the need for objectivity and reasonable independence in conducting evaluation.
- The high cost of evaluation research and lack of access to rapid, low-cost research methods.
- Shortage of trained staff.
The Bank's experience. To help borrower countries to strengthen their evaluation capacity, the Bank started with efforts to ensure borrowers monitored and evaluated the projects it financed. Then it introduced a requirement for borrowers to contribute to project completion reports on the projects it supports. Since 1987, it has had a program to support the development of broadly based evaluation capacity (see Box 2: Evaluation capacity development program). Countries assisted by this program include Bangladesh, Brazil, Chile, China, Colombia, Indonesia, Mexico, Morocco, Pakistan, Philippines, Sri Lanka, Thailand, Venezuela, and Zimbabwe. The Evaluation Capacity Development Program (ECDP) has also supported development agencies including the West African Development Bank, the Andean Development Corporation, the Caribbean Development Bank, and the Islamic Development Bank.
The experience, which has been mixed, emphasizes the following prerequisites for effective evaluation:
- Evaluation will not thrive unless there is a demand for it. Demand for evaluation results, and a performance-oriented culture in the public sector, are essential if evaluation is genuinely to affect public policy and resource allocation.
- Conditions must favor the development of reliable and accessible information. There is scope here for institutional development assistance, by the Bank and other agencies, to strengthen the basic institutional and professional infrastructure of accountability, ranging from firm enforcement of covenants on financial reporting to much broader support, for example through accountancy development projects, financial sector adjustment loans, or public sector management operations.
- Evaluation results must be fed back into a receptive and responsive decision making process. Government leaders, parliaments, and publics need a basis for identifying both specific shortcomings and systemic problems in government performance. A free flow of information helps managers at all levels. At the technical level, managers need feedback to identify what is working and what is not--and why. Senior decision makers must be able to monitor not only major policies and programs but the performance of the institutions running them, so they can specify reform measures, where needed.
- Evaluation implies accountability. Evaluation findings must have--and be seen to have--consequences for the institutions and the individuals who are accountable for the conduct of public business.
Lessons for planners
Serve decision makers' needs. Evaluation can be best developed if it is seen by all concerned--within both the country and the development community--as a way to learn and to improve the performance of the public sector. Efforts to build evaluation capacity must be designed to serve the real needs of domestic decision makers. Demand can arise from a prominent client or key stakeholder who sees the benefit of using evaluation results to improve public performance.
Key officials should take full advantage of the demand that already exists in some parts of government. Donors and international financial institutions can, through their loan requirements and influence, help nurture demand by domestic agencies, but should take care not to substitute for it.
Plan in the context of public sector management. Capacity building for evaluation needs to be planned and undertaken within the context of improvements in public sector management. The performance of the evaluation system will depend on the clarity of the institutional relations within the country, the responsibilities of the function, and the means available. Any country considering establishing or strengthening an evaluation function in the public sector will need to address the factors that are listed in Box 3: Factors to consider in institutionalizing evaluation.
Start modestly. It may be prudent to start modestly and put in place simple mechanisms that can be implemented immediately, rather than starting with the design and development of a comprehensive or very sophisticated set-up.
In principle, the evaluation of country policies and programs promises the highest payoff. But such evaluation is politically and bureaucratically more sensitive. It is also more demanding methodologically and may take longer to establish itself. Meeting demand for sectoral and project-level evaluation within line ministries may often provide a more effective starting point.
Developing a country-specific strategy
In most developing countries, building up an effective evaluation capacity will require sustained, complementary efforts in three areas: to develop and nurture genuine demand and political support for evaluation; to improve supply capacity by building relevant skills and institutions; and to build financial information and dissemination infrastructure.
On the demand side, in countries such as China and Thailand, auditors general played an initial important role in raising awareness about the need to establish a national evaluation function. These agencies have also pursued the organization of evaluation training programs for their own staff. In some countries the executive branch has followed the lead and taken steps to institutionalize the function. In Thailand, for example, a task force under the prime minister, with the participation of the auditor general, is studying how to establish an evaluation function that will cover government agencies and public enterprises. In Brazil and Colombia, constitutional requirements have generated demand for evaluation from the legislative and executive branches; both these countries have moved to institutionalize the function and train staff.
On the supply side, the Bank has supported Brazil in several workshops and seminars on evaluation in government and to discuss evaluation experience (for example on the environment, public sector management). In Colombia, the Bank has provided support for public policy evaluation through a public financial management project. In Chile, where human resources are not a constraint, the Bank has been asked for advice about how to establish an evaluation function to complement the highly developed resource allocation system in place. In China and Indonesia, the Bank's Institutional Development Fund is supporting programs that focus on demand as well as supply constraints.
The choice of interventions, however, needs to be closely tailored to a country's particular circumstances. The following stylized framework may help assess country needs and opportunities.
Countries where demand, supply, and infrastructure are weak. To nurture demand for, and show the practical utility of, evaluation, best practices in evaluation from within and outside the country could be disseminated. On the supply side, it may be prudent to start with some simple steps. For example, the audit function may be strengthened and asked to undertake selective evaluations especially useful to senior policy makers. To alleviate capacity constraints and to develop local skills, multilateral and bilateral agencies may jointly evaluate their projects with country personnel. If training in basic disciplines relevant to evaluation is needed, donors and international financial institutions could fund such training at local universities. Development of modern budgeting, auditing, and accounting systems may be useful. Interventions to develop evaluation capacity should be simple, cost-effective, and focused on activities that demonstrate the usefulness of evaluation.
Countries where demand exists, capacity is weak. There may be receptivity for training and information on best practices and also for the use of government commissions/task forces to evaluate important programs and policies. This approach can be used, strengthened through technical assistance, even if local skills and institutional mechanisms in government are weak. Where skills in government are limited, there may be scope for using non-governmental organizations in evaluation and to listen to public feedback. Introducing transparent performance standards, budgetary classifications, and modern financial and banking codes is also likely to have a high payoff. The activities described in the preceding paragraph may be useful in this case, too.
Countries where demand is weak, capacity is promising. To build up demand, it may be useful to link evaluation activities to public expenditure reviews and other public sector management activities, while seeking opportunities in the course of the country dialogue for raising awareness among decision makers and legislators of the practical utility of effective evaluation. Local personnel may need more practice-oriented -- rather than academic -- training. This need has emerged in China, for professionals who are unfamiliar with evaluation practices and methods. Where good research institutes exist, encouraging local personnel to work on evaluation may be a good beginning even when government is not well organized to fully use the evaluation results.
Demand and supply are relatively well developed. Here, it may be most useful to strengthen the link between evaluation and resource allocation decisions and policy reform, and to pursue improvements in the financial and informational infrastructure.
In most countries, different parts of the evaluation function are at different stages of development. This suggests the need to build on opportunities where they exist, and to accept that progress will not follow a predetermined path. The design of country strategies for developing evaluation capacity should be based on a full appreciation of governments' evaluation needs. It will normally be necessary to assess the demand within government for different types of evaluation and determine where the use of evaluation capacity will be most productive.
Instruments for the medium term
- Assess the legal framework affecting evaluation. Changes may be needed to institutionalize evaluation and ensure its continuity (see Box 4: Colombia legislates an evaluation function).
- Exploit opportunities at the national and sector levels to support stakeholders who are interested in using evaluation results to improve the performance of policies and programs.
- Incorporate evaluation findings in analysis and dialogue on public sector management issues.
- Incorporate evaluation and monitoring arrangements into project design and implementation plans, and include specific indicators and evaluation work in terms of reference for mid-term program/project reviews.
- Use project initiation workshops, mid-term reviews, and public expenditure reviews and/or other regular activities to encourage and apply lessons of formal and informal evaluations of policies, programs, and projects.
- Promote evaluation by users and communities affected by programs and projects. Encourage agencies to evaluate their own performance and to establish feedback mechanisms to ensure results are applied.
- Promote joint evaluations with external assistance agencies and international financial institutions. Coordination with the development community can help strengthen evaluation.
- To be effective, evaluation work must be rigorous and credible. Ensure that appropriate skills, training, methodologies, and standards are provided. Here, private sector and academic institutions can be valuable allies.
"Mainstreaming" the development of evaluation capacity
A Task Force on Evaluation Capacity Development reported to the Joint Audit Committee of the Bank's Board of Executive Directors in July 1994. It stressed the close links between domestic evaluation capacity in borrowing countries and the Bank's broader agenda on institutional development and portfolio management, and recommended that the Bank take up a more proactive approach to supporting evaluation capacity development. In particular, it recommended that assistance for developing evaluation capacity should no longer be regarded as a distinct initiative but should form an integral part of the Bank's country portfolio management work and of its support for public sector management reform. The report of the Task Force proposes a strategy for this purpose.
Box 1: In Korea, an emphasis on operational performance
Korea has evaluated government performance since 1962. Evaluation is the responsibility of the Performance Evaluation Bureau (PEB) within the Economic Planning Board.
Headed by a director general, PEB has five directors, 20 deputy directors, and 22 staff members. It evaluates the performance of government and of public enterprises, with the aim of increasing managerial efficiency. It establishes and refines the standard guidelines for evaluating government performance that are to be applied by all government ministries for their self-evaluations. It collects and analyzes the results of these evaluations, and reports on them twice a year to the cabinet council and the president. PEB also handles problems that may arise during the implementation of important government programs.
The performance evaluation system focuses on the operational performance of the government's major policies and programs. It covers three kinds of evaluation:
- Intensive analysis of government programs that have special economic and social importance.
- Ministerial self-evaluation.
- Evaluation of major government implementation programs. These include policies and projects, presidential directives, and presidential campaign commitments.
Evaluation results are used to locate performance problems and to determine future resource allocations. PEB formulates recommendations to correct identified problems, and the Office of the Budget and ministries concerned are consulted to guarantee adequate feedback, before the recommendations are implemented. Once PEB has submitted its report to the cabinet council and the president, a press release on the report is issued.
Box 2: Evaluation capacity development program
In 1987, the Bank established an evaluation capacity development program (ECDP) in response to specific requests from borrowers. Managed by operations evaluation staff, the program has the following elements:
- Designing action programs: After high-level country authorities have appointed a senior official to head the evaluation function, the task manager and staff visit Bank headquarters to discuss goals and constraints with evaluation, operations, and EDI staff. These discussions help identify the relevant experience within the Bank and internationally. Action plans are designed for instituting or strengthening evaluation, and a framework is laid out for cooperation with OED and others. To help focus action plans and system development, organization and process-diagnostic studies may be recommended and planned. Follow-up meetings take place periodically in Washington or in the borrowing country.
- Building awareness: Seminars and workshops on evaluation methods and findings are held for government officials. Presenting the findings from the Bank's own evaluations helps develop both the awareness of potential users of evaluation results and a consensus for support of evaluation within the country.
- Documentation: Evaluation material from OED and other sources, including international development agencies, is made available.
- Training: Seminars and workshops are held on the role and conduct of evaluation, with OED staff primarily contributing course material, case studies, and lectures. On-the-job training is provided through engaging borrowing country staff as observers or participants in OED's evaluation work. OED takes staff on secondment, where this constitutes a priority within the framework of cooperation arrangements. OED staff increasingly spend time during evaluation missions to discuss evaluation issues with the host institution.
- Parallel evaluations: OED considers topics for parallel evaluation with borrowing countries that have made a commitment to develop their evaluation function. Parallel evaluation is undertaken if these joint efforts coincide with OED priorities, as reflected in its work program, and do not affect OED's independence. OED retains full responsibility for its reports and encourages the host institutions to produce their own.
Box 3: Factors to consider in institutionalizing evaluation
How does a country go about building a public sector evaluation function? This checklist, in approximately sequential order, shows items that will need review and action.
Relation to other public sector functions and institutions
- Where evaluation is set up at the executive level of government, its relation to the management of the national budget, to public sector expenditure decisions, and to the development of the strategic plan.
- The relation between executive-level evaluation and the functions of the auditors or controllers general, or other independent oversight institutions of the legislature.
- Legal requirements for setting up an evaluation function.
- Relation of evaluation to the audit or internal control functions.
Centralization or decentralization
- The appropriate division of responsibility between the central government, line ministries, provincial and local authorities.
- Rules and regulations governing the evaluation function.
Location
- Independence of the function.
- Links to the presidency, prime minister's office, and the planning, finance, and budget systems.
- Legislative branch requirements.
- Requirements of individual agencies' line management, and the role of self-evaluation.
- Relation of evaluation to research and academic institutions and private sector entities.
Scope, coverage, focus
- Policies, programs, or projects.
- All public sector resource uses, or only public expenditures financed through public debt.
- All sectors, including the parastatals.
- Priority of externally funded programs and projects.
- Impacts and the human development incidence of public resource use.
- Accountability aspects of public sector decisions, and learning from experience and feedback to the decision making processes.
- Evaluation reporting channels and the follow-up to evaluation work.
Financial resources, staffing, information requirements
- M&E systems.
- Financial resources.
- Professional skills and training.
- Methods, guidelines, and norms.
Box 4: Colombia legislates an evaluation function
In 1990 the minister of finance and the head of the National Planning Department (NPD) called for a study of how to design a national evaluation system. The study led the authorities to conclude that a public sector evaluation function would become a reality only if there was a fundamental law requiring it.
In 1991 the Colombian constitution was being rewritten. Authorities took the opportunity to draft an article making public sector evaluation mandatory. The article conferred on the NPD the responsibility of establishing a national evaluation system.
Late in 1991 the NPD organized a high-level meeting of present and past public sector officials and private sector representatives to:
- review alternative organizational approaches;
- discuss evaluation experience arising from large, controversial public sector projects;
- build consensus on the direction of the regulations regarding the evaluation function.
Participants included members of the constitutional assembly, the minister of finance, the controller general, the head of NPD, the attorney general, and other sector ministers. Also participating were the deputy controller general of the United States, the World Bank's director-general, operations evaluation, and other international officials.
The NPD has established an evaluation function, and the responsible officers are developing evaluation plans. Yet to be determined are the roles of the other key agents, such as the controller general. In September 1993 the Bank approved a public financial-management project that incorporates a component for public policy evaluation. This component aims at, among other things, verifying that public resources have been well programmed and honestly and efficiently spent.
Box 5: Observations on developing evaluation capacity by Robert Klitgaard, Professor of Economics, University of Natal, South Africa
Why evaluate? Evaluation has several purposes:
- To test or find clues for predictive relationships that will help inform subsequent decision making.
- To create incentives for agents. Measuring performance carefully and providing intense incentives are complements. Better evaluations, other things equal, enhance incentives and reduce the risks for agents.
- To enable agents to make mid-course corrections. Evaluation data are the more useful, the greater are the local and temporal variability in key factors affecting a program's success, and the more susceptible the program is to adaptations by its managers.
- To foster accountability and transparency. Quite apart from what it shows about the success of particular projects or managers, a policy of purposive evaluations may engender a climate less subject to suspicion and abuse, and more open to the inputs of clients.
How much evaluative capacity?
How much evaluation capacity to establish depends on how helpful the evaluation findings are expected to be, and on the uses to which the findings and the evaluation process are to be put. These dimensions vary considerably among tasks, agencies, projects, and governments.
Broadly speaking, the value of evaluation depends on its ability to affect managerial decisions. This ability depends not only on the quality of the evaluation but also on the incentives for politicians and bureaucrats, and perhaps for aid officials and organizations. The economics of organization suggests that one should spend more on evaluation when agents' incentives are affected by evaluations and when what agents do greatly affects the "profitability" of the project or organization. One implication is that if evaluation findings cannot affect agents' incentives much, one should not evaluate much. For evaluations to make a difference, incentives need to be linked to evaluations.
Some of the managerial literature suggests that evaluations are particularly useful when the environment or the technology is subject to uncertainty, where it changes over time or locality or both, and where managers are permitted to make mid-course corrections. Thus evaluative capacity is less useful in well-known, standardized technologies under stable conditions where managers in any case have little freedom of action.
Establishing a vigorous evaluation function may be particularly important when governments particularly need accountability and transparency--for example, when a general lack of confidence must be overcome. A strong, visible, indeed expensive, commitment to evaluation may send a signal that has a value far beyond that measured in decision-analytic or managerial terms.
These ideas may help explain why evaluation has not developed very far in some contexts. For example, a government will probably undervalue evaluation when officials have weak links between pay and performance, little authority to do mid-course correction, and little input or effect on project quality, and when evaluation has high opportunity costs. By improving incentives and decision processes, the value of evaluation rises--which, other things equal, generates demand for better evaluation capacity.
| For more information on this subject or to receive information on other Operations Evaluation Department products, please contact the Operations Evaluation Department at: eline@worldbank.org, Tel: (202) 458-4497, Partnerships and Knowledge Programs, World Bank Group, 1818 H Street, Rm. H3-304, Washington, DC 20433 or visit our website at: http://www.worldbank.org/oed/ |
| |
|
|
|
 |
© Copyright 2001 OED, All Rights Reserved |
|
|
|
