Establishing USAID as a Leader in Evidence-Based Foreign Aid

Figure 1. Percent of sampled USAID evaluations that met each of GAO’s evaluation quality criteria

Image

Source: US Government Accountability Office, 2017. “Foreign Assistance: Agencies Can Improve the Quality and Dissemination of Program Evaluations.”

In addition, despite higher numbers of evaluations overall, impact evaluations—studies that can measure results attributable to a USAID program—remain rare.^[12] While some parts of the agency—notably, parts of the Bureau for Economic Growth, Education, and Environment and the Democracy, Rights and Governance Center—have had periods of intentional investment in impact evaluations, they’ve been a low priority for much of the agency. Of course, impact evaluations aren’t always appropriate or feasible,^[13] but they offer valuable learning opportunities for interventions where the evidence base is limited or mixed.

There are a number of constraints to producing more and higher quality evaluations at USAID. Time-strapped field staff are under pressure to execute programming, manage contracts, and fulfill reporting requirements. This can leave limited time to pursue evaluations and compress evaluation timelines, which can compromise their quality, especially the quality of sampling and data collection.^[14]

Capacity constraints also play a role. Since 2011, USAID has hired more staff with evaluation expertise.^[15] Nevertheless, many M&E staff, who spend most of their time on performance monitoring tasks, have limited evaluation experience or expertise. And when staff have limited time, incentive, or technical background, even strong evaluation training and guidance from PPL can only go so far. At USAID’s headquarters in Washington, both PPL and the pillar bureaus have in-house evaluation experts who can provide support to the field, but these resources are available by request, and opportunities to request support—both for identifying evaluation opportunities and planning for quality studies—can easily be overlooked.

Another challenge that plagues evaluation is one of timeliness. Evaluation, especially impact evaluation, is perceived as slow and expensive—and that is frequently a fair assessment. Program managers are reluctant to conduct evaluations if it will slow program implementation. And if evaluation results won’t be available until years after a program has concluded, program managers have little incentive to pursue them. Partly in response to questions of timeliness, there’s growing interest—including at USAID—in new, more rapid evaluation methodologies.^[16] The agency is also exploring opportunities to use administrative data and other data sources (e.g., satellite or geospatial data) in places of slower, more costly, and sometimes duplicative specialized surveys.^[17] But these remain somewhat rare.

While all these constraints apply to evaluations across the board, they are more acute for impact evaluations, which typically require more time and specialized knowledge.

Evidence informs policy and program decisions more episodically than systematically

The value of evaluation lies largely in its use. Indeed, basing funding decisions on analyses and evidence is a core principle of USAID’s Program Cycle.^[18] But its implementation is inconsistent in practice. While the vast majority of USAID evaluations are used to inform some kind of decision making (usually just within the unit that commissioned the study), it’s not clear how well the broader universe of evidence—including evaluations conducted by other donors and organizations—is brought to bear.^[19] In at least one documented case, an incomplete review of evidence led USAID to solicit bids for a program with a disproven theory of change.^[20] As this example demonstrates, failure to examine the evidence case for an approach can lead to inefficient spending. It can also miss opportunities to pursue more effective approaches, or—where proposed approaches are untested (or undertested)—to plan an accompanying evaluation, as required by policy. USAID seeks to avoid this for large programs by requiring senior leadership to review proposals; as part of this process, staff are asked to include the evidence case for the selected approach.^[22] But while this can be a helpful prompt, it’s unclear how heavily the evidence case is weighted relative to the other criteria under review, how the strength and comprehensiveness of the evidence presented is evaluated, or even what expectations for remediation—if warranted—would be at such a late stage in the design process.

There are several barriers to greater uptake of evidence in program design.^[22] Again, time and capacity constraints loom large. Where evidence exists it’s often inaccessible to busy staff with little time to conduct a full evidence review and limited expertise in concepts relevant to understanding econometric research.^[23] Summarizing and synthesizing evidence can help, and USAID and other organizations like 3ie have several tools and processes (e.g., evidence gap maps, synthesis reports, newsletters, knowledge sharing platforms, evidence summits) to help missions learn from evaluation findings, but these are not always widely used.^[24]

Another barrier to evidence use is skepticism of its relevance. This is partially related to quality but not exclusively. Certainly, low-quality evaluations—those without a valid methodology or credible findings— won’t convey useful information. But questions of relevance can also surround high-quality studies. Because the results of an individual evaluation may not always be generalizable to other contexts, staff may (sometimes rightly) feel its findings are irrelevant to the project they are designing. Multiple evaluations of similar interventions in different contexts can strengthen the evidence base for a given approach, but the kind of replication and synthesis needed to achieve this type of meta-analysis has been rare. Evaluation resources are often spread broadly, and researchers tend to have professional incentives to pursue “cutting edge” questions.

It is also important to note that the primary objective of a significant portion of USAID funds is advancing US foreign policy interests—even if the investments are nominally about development. When development objectives are ancillary to the program’s core goals, staff may have less scope—or less time—to bring evidence to bear on program design.^[25]

Costing and cost-effectiveness are understudied, though nascent efforts are encouraging

With aid budgets under pressure, USAID’s value for money will likely come under increased scrutiny. To be well positioned to defend development spending, USAID and foreign aid advocates would benefit from a robust understanding not only of program results (i.e., was the program better than nothing) but also answers to questions about opportunity cost: did a project work well enough to justify spending money on it compared to using those funds for something else? Is a project’s impact per dollar greater than that of an alternative?^[26]

But while impact evaluations have been rare, impact evaluations that include the kind of cost analysis necessary to understand impact per dollar are rarer still.^[27] And where cost analysis has been done, the underlying cost data has been of mixed quality and methodologies have often varied, limiting their comparability across studies.^[28]

USAID has started to tackle the question of cost effectiveness. It funds and participates in the Costing Community of Practice, run out of the University of California, Berkeley’s Center for Effective Global Action, and the Office of Education established a cost measurement initiative to analyze program costs and link them to outcomes.^[29] In addition, DIV has launched a series of costed impact evaluations that attempt to compare the per-dollar results of several “traditional” aid programs with those of cash transfers. The premise of this exercise, known as cash benchmarking, is that since cash transfers have a demonstrated ability to shift individual or household outcomes and are among the lowest cost interventions, they make a useful benchmark to determine whether a “traditional” program adds any value.^[30] But while these efforts are promising, they are relatively new and niche and will require a champion for cost effectiveness to advance them.

Responsibilities for promoting evidence remain fragmented and reach to missions is limited

USAID made a number of bureaucratic changes to better implement its new focus on evidence. In 2010, the agency established the Bureau for Policy, Planning and Learning (PPL) and within that, the Office of Learning, Evaluation, and Research (LER) which sets operational policy and provides agency-wide guidance on monitoring, evaluation, and learning. The same year, USAID launched DIV, a unique program that identifies and rigorously tests new solutions to development problems and helps scale those that prove successful. In 2014, the agency created the Global Development Lab, which became the bureaucratic home for DIV, along with (among other units) the Center for Development Research (CDR), which supports the creation of scientific knowledge and evidence around USAID’s development priorities, and the Office of Evaluation and Impact Assessment (EIA), which supports evaluations of innovative approaches, especially related to science and technology.

These have been important structural advances to help refocus the agency on evidence and evaluation. But the configuration also has limitations. Responsibility for evidence and learning ends up fragmented across the agency, not only across the aforementioned units but also across pillar bureaus whose evaluation points of contact have a significant role in supporting evaluations and disseminating evidence. This may make it harder for mission staff to know where to turn for what type of support. And some functions, like capacity building around evidence and dissemination of evidence-based learning, may be duplicated. In addition, LER and DIV have had limited reach to missions where most evaluation efforts are managed. And because they are small units sitting within much larger bureaus, they—and their focus on evidence—can be overshadowed by their respective bureaus’ other activities.^[31]

USAID’s impending restructuring provides an opportunity—and underscores the need to—ensure evidence and evaluation functions are elevated rather than sidelined in the shift. In response to a Trump administration call for agency reorganization, USAID put forward an agency-wide transformation plan, which includes several shifts in bureaucratic structure. Under the proposed reorganization, DIV will move into the new Bureau of Development, Democracy and Innovation, and LER and EIA will be combined into a new Office of Learning and Evaluation within a proposed new Bureau for Policy, Resources and Performance—though the latter still awaits congressional approval .^[32] A core goal of the next phase of restructuring should be to elevate and consolidate the agency’s evidence, evaluation, and learning functions and extend their reach to better support missions.^[33]

Policy recommendations

USAID has an opportunity to be a leader in evidence-based foreign aid. To elevate an evidence-oriented agenda at USAID and overcome barriers that undermine the generation of high-quality evidence and its systematic use, the next administration should prioritize the following actions. Some of these would involve the creation of new staff positions with corresponding resource implications. The next administration should make the case for increased hiring to fill these positions in an early budget request to Congress—highlighting their role in improving the effectiveness and efficiency of appropriated funds, as well as the decline in US direct hires over the past five years.^[34]

The White House should:

• Nominate a USAID administrator who will champion evidence and commit to advancing evidence-based policymaking. High-level political support is critical for shifting agency practice and culture.^[35] The nominee for USAID administrator should have a track record as an evidence champion who is clearly committed to advancing evidence-based policymaking at USAID. The nominees to head PPL (or the proposed new Bureau for Policy, Resources and Performance) and the pillar bureaus must be similarly committed to evidence, but administrator-level support will be critical for other senior leaders to be effective in their pursuit of evidence-based policymaking.

USAID should:

• Create a new, consolidated evidence and evaluation unit with leadership that reports directly to the administrator. Such a unit would consolidate and expand the evidence, evaluation, and learning functions—currently housed across PPL, the remnants of the Global Development Lab, and what’s now the Bureau of Development, Democracy and Innovation—under a new senior leadership position empowered, through its direct link to the administrator, to push forward an evidence-oriented agenda.^[36] The unit’s structure would highlight and elevate evidence, evaluation, and learning functions rather than subsuming them under multiple larger bureaus. It would also reduce duplication and allow a more streamlined evidence and evaluation strategy. Two of the new or expanded functions of this unit should include:
  • Stronger, centralized impact evaluation services: USAID should establish a team of impact evaluation support specialists who would proactively work with missions to identify opportunities to pursue impact evaluation, to ensure methods and sampling are adequate to the questions being asked, and to manage the implementation and dissemination of the evaluations. It is unrealistic for specialized impact evaluation skills to be diffused across missions, especially when this type of evaluation is often a small part of a typical mission M&E officer’s portfolio. Managed by the centralized evidence and evaluation unit, these staff could reside in Washington (with linkages to the pillar bureaus) and/or regional missions in order to serve multiple missions and be deployed, as needed, to embed within operational teams.^[37] While this function could be partially covered by existing staff, it may require budget for additional staff salaries.
  • “Evidence broker” functions: For communication about evidence to be relevant to policy and program decision makers, it must be tailored to their immediate needs.^[38] Establishing a cadre of evidence broker staff—employing a hybrid set of analytical, policy, and communications skills—can, from a central or embedded position, socialize the results of new studies and translate relevant findings into targeted recommendations for program design.^[39] Evidence brokers can also play a central role in the review and approval of the evidence case presented for new programs. This recommendation would entail creating new staff positions.
• Strengthen evaluation skills within missions. Even with centralized impact evaluation services, some mission-level evaluation functions will remain important to identifying opportunities for impact evaluation, to work with a centralized impact evaluation support team, and to ensure high quality performance evaluations. Given the amount of time M&E staff spend on performance monitoring and fulfilling reporting requirements, USAID should separate monitoring and evaluation functions into two staff roles at the mission level.^[40] In selecting evaluation staff, they should weigh heavily a familiarity and experience with quantitative and qualitative evaluation methods. This recommendation would entail creating new field staff positions.
• Build evidence use and generation into the procurement and program design processes. USAID should frame its award solicitations around evidence generation and use.^[41] This would come with no additional budget requirements.
  • Where the evidence base is weak or contradictory, USAID is less able to provide strict technical guidance about implementation and the agency should pursue more flexible award types that allow for experimentation and testing and should evaluate bids, in part, on how well the proposal would accommodate evaluation. Awards for untested approaches should start small with opportunities to expand depending on evidence of effect.
  • Where there is a stronger evidence base, either a (more prescriptive) contract or (more flexible) cooperative agreement may be used, but solicitations for either type of award should adequately reflect the state of evidence. In a solicitation for a contract, USAID should summarize the body of evidence upon which its proposal is built. For cooperative agreements, the solicitation should require bidders to summarize the relevant evidence that motivates their proposed approach and bids should be scored, drawing on the expertise of evidence brokers or other M&E staff, on how well the proposed approach demonstrates an understanding of the existing evidence.
  • For co-creation processes, evidence brokers or other M&E staff should be involved in early convenings to lay out the evidence base, discuss theories of change, and outline expectations for building evidence.
• Focus impact evaluation resources more strategically. To improve the utility and relevance of its evaluations, LER (or its successor), in partnership with the pillar bureaus, should create a strategy for each sector that serves to focus impact evaluation resources on a limited set of key questions.^[42] Narrower than a typical learning agenda—which, in the name of consensus, tends to encompass too many questions without meaningful prioritization^[43]—the strategy should emphasize questions about common or highly funded USAID approaches and include space for replication rather than exclusively focusing on “cutting edge” research. While strategy development itself carries no budget implications, implementing it will require bureaus and missions to set aside program funds on the order of three to five percent to finance evaluations.
• Invest in faster, less expensive impact evaluation methods. To reduce barriers to pursuing impact evaluation, PPL (or its successor) should encourage and incentivize methods that lower the cost of impact evaluation and yield more timely results. These should include prioritizing the use of administrative data where possible, accompanied, as relevant, by support to improve administrative data quality; expanding efforts to use other non-survey data sources, like satellite or geospatial data; and experimenting with new rapid evaluation methodologies.
• Advance cost-effectiveness analysis, both internally and industry wide. USAID should redouble its efforts to understand comparative cost-effectiveness. USAID’s next steps—which can be accomplished at minimal additional cost—should include expanding its support for and leadership in multi-stakeholder efforts to establish a common costing methodology to enable more consistent and available cost evidence.^[44] Since each organization benefits from the generation of evidence by others, a coordinated effort is critical. The agency should then adopt a commonly accepted costing methodology and ensure impact evaluations begin to include cost data.^[45] USAID should also expand its cash benchmarking work by prioritizing questions for costed impact evaluation and synthesizing findings on the range of per-dollar results of different interventions.^[46] This effort should also include external coordination. As a pioneering donor in cash benchmarking, USAID should seek to convene other donors interested in costed impact evaluation and/or cash benchmarking to join forces to identify gaps and address them in a strategic and intentional way.

Additional reading

Sarah Rose and Amanda Glassman, 2017. “Advancing the Evidence Agenda at USAID.” CGD Note. Center for Global Development.

Sarah Rose, 2018. “As USAID Thinks about Procurement and Program Design, It Should Keep Evidence in Mind.” CGD Blog. Center for Global Development.

Sarah Rose and Amanda Glassman, 2018. “Committing to Cost-Effectiveness: USAID's New Effort to Benchmark for Greater Impact.” CGD Note. Center for Global Development.

Center for Global Development Evaluation Gap Working Group, 2006. When Will We Ever Learn: Improving Lives through Impact Evaluation. Center for Global Development.

The Lugar Center and The Modernizing Foreign Assistance Network, 2017. From Evidence to Learning: Recommendations to Improve U.S. Foreign Assistance Evaluation.

Thanks to Annette Brown, Anne Healy, Rebecca Wolfe, and many others for useful conversations that helped inform the work.