Executive Office of the President, Memorandum, 26th July 2013

EXECUTIVE OFFICE OF THE PRESIDENT
OFFICE OF MANAGEMENT AND BUDGET
WASHINGTON, D.C. 20503
THE DIRECTOR July 26, 2013

MEMORANDUM TO THE HEADS OF DEPARTMENTS AND AGENCIES

SUBJECT: Next Steps in the Evidence and Innovation Agenda
Executive Summary
The President recently asked his Cabinet to carry out an aggressive management agenda for his second term that delivers a smarter, more innovative, and more accountable government for citizens. An important component ofthat effort is strengthening agencies' abilities to continually improve program performance by applying existing evidence about what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery. This is especially impmiant given current fiscal challenges, as our nation recovers from a deep recession and agencies face tough choices about how to meet increased demand for services in a constrained resource environment.
To help agencies move forward in harnessing evidence and evaluation, this memo:
•
Provides guidance for 2015 agency Budget submissions and describes plans to prioritize Budget requests that strengthen the use of evidence and innovation.
•
Invites agencies to participate in a series of workshops and interagency collaborations organized by the Executive Office ofthe President to help agencies develop and strengthen proposals that catalyze innovation and learning. While much ofthe focus will be on proposals that can be implemented without additional resources, there will be limited funding available in the President's 2015 Budget for strong proposals that require some new funding.
Using Evidence and Innovation to Improve Government Performance
2015 Agency Budget and Performance Submissions
Agencies are encouraged to both: (1) draw on existing credible evidence in formulating their budget proposals and performance plans and (2) propose new strategies to develop additional evidence relevant to addressing important policy challenges. Agency requests are more likely to be fully funded ifthey show a widespread commitment to evidence and innovation.
Evidence in agency budget submissions and performance plans.
Agencies are encouraged to allocate resources to programs and practices backed by strong evidence of effectiveness while trimming activities that evidence shows are not effective. In addition, major new policy proposals, and agency perfmmance plans, should be accompanied by a thorough discussion of existing evidence, both positive and negative, on the effectiveness of those proposals in achieving the policy objective or agency priority goal. Such evidence includes evaluation results, performance measures, and other relevant data analytics and research studies, with a preference for high quality experimental and quasi-experimental studies. (Please include citations for evidence discussed.) Moreover, evidence should be regularly considered during agencies' data-driven reviews led by their Chief Operating Officers and in annual strategic review processes.
New proposals for developing evidence
In their budget requests, agencies are also encouraged to include new proposals for developing evidence that can be used to improve existing programs or to infmm decisions about new programs. (This includes proposals that build on and enhance cunent efforts.) Recognizing the current budgetary pressures on agencies, OMB encourages agencies to focus their energies on a small number ofhigh-quality proposals that meet one or more ofthe following tests:
• They address important policy questions and generate evidence that could be actionable. In particular, evaluations should measure the outcomes that are relevant for judging whether a program or intervention is achieving its goals.
• They will yield credible evidence ofprogram or policy impacts, for example by utilizing randomized controlled trials or carefully designed quasi-experimental techniques.
• They will help agencies direct a larger share ofresources towards evidence-based practices, for example by modifying grant criteria or better disseminating information.

Agencies are encouraged to consider the following cross-cutting strategies. Specific examples of each strategy are provided in Attachment A.

1. Harnessing data to improve agency results: Proposals should enable agencies and/or researchers to access and utilize relevant data to answer important questions about program outcomes while fully protecting privacy. For example, by linking data on program participants to administrative data on earnings, college-going, health, or other outcomes, agencies may be able to improve their understanding ofprogram performance and ultimately improve results. Projects should build on the recent Executive Order, "Making Open and Machine Readable the New Default for Government Information," as well as on the Memorandum "Sharing Data While Protecting Privacy" (M-11-02). We especially encourage proposals that use administrative data to track important outcome measures for federal grant programs, and we are open to proposals that substitute higher quality administrative data for existing grantee reporting requirements.

2. High-quality, low-cost evaluations and rapid, iterative experimentation: Proposals should help agencies improve the quality and timeliness of evaluations, for example by building evaluation into ongoing program changes; reducing costs by measuring key outcomes in existing administrative data sets; and drawing on private sector approaches that use frequent, low-cost experimentation to test strategies to improve results and retum on investment. Proposals should utilize randomized controlled trials or careful quasiexperimental techniques to measure the effect ofinterventions on important policy outcomes. We particularly welcome proposals that draw on behavioral insights to improve results and lower costs in direct operations.

3. Using innovative outcome-focused grant designs: Proposals should expand or improve the use of grant program designs that focus Federal dollars on effective practices while also encouraging innovation in service delivery. These include tiered-evidence grants, Pay for Success initiatives and other pay for performance approaches, Performance Partnerships allowing blended funding, waiver demonstrations, incentive prizes, competitive incentive funds that encourage the use of evidence-based practices in fmmula grants, or other strategies to make grant programs more evidence focused.

4. Strengthening agency capacity to use evidence: Proposals should strengthen agency capacity by promoting knowledge-sharing among government decision-makers and practitioners through clearinghouses that help translate strong research into practice; enhancing the skills ofmanagers, program officers, and review panels to assess and use available evidence; and developing common evidence frameworks to better distinguish strong from weak evidence and measure cost effectiveness.

5. Other agency-specific needs: Agencies may propose other strategies that would significantly improve their capacity to use or build evidence to achieve better results or increase cost-effectiveness in high priority programs. In addition to developing strategies to use evidence to promote continuous, incremental improvement, agencies are also · encouraged to submit proposals that would test higher-risk, higher-return innovations with the potential to lead to more dramatic improvements in results or reductions in cost.

While agencies are encouraged to submit proposals that can be implemented within cunent statutory authorities, legislative changes will also be considered. (Please note where a proposal would require legislative changes.) Agencies may also propose new investments in evidence-building infrastructure for high-priority areas in cases where the benefits substantially outweigh the costs. Agencies may wish to consider new financing approaches, set-asides that designate a small fraction of funding for evaluation and evidence development; and partnerships with other federal agencies, state and local governments, non-profit organizations, and academic institutions. We particularly encourage proposals that cross agency boundaries or other functional silos.
Agencies should work with their OMB contacts to agree on a format within their 2015 budget submissions to: (1) explain agency progress in using evidence and (2) present their plans to build new knowledge ofwhat works and is cost-effective. An example of a template that could be used to provide this information to Resource Management Offices is available at https://max.gov/omb/evidence.
Workshop Series and Interagency Collaborations
To support agencies in developing and refining proposals, this September we will begin an interagency collaboration process with a kickoff briefing or call followed by a series of workshops (see Attachment B for details). An initial list of workshop topics is below; we may schedule additional workshops based on agency demand and continue this series after agency budget submissions are finalized to support implementation. Versions ofthese workshops may be tailored to agencies at different stages of experience with evidence-based practices.
• Workshop I: How can agencies focus evaluation resources on the most important
program and policy questions?
• Workshop II: How can agencies use administrative data sets from multiple programs and levels ofgovernment to answer important questions while protecting privacy?
• Workshop III: How can agencies conduct rigorous program evaluations and data
analytics on a tight budget?
.• Workshop IV: How can agencies use their existing authority to tum a traditional
competitive grant program into an innovative, evidence-based one?
• Workshop V: How can agencies harness research findings from the social and behavioral sciences to implement low-cost approaches to improving program results?
The workshops will be designed to build and share knowledge across the Federal government as well as to identify expertise and resources to help agencies implement strong proposals. Beyond the workshops, OMB, DPC, CEA, and OSTP are available to provide other fmms of assistance:

• Technical assistance in designing evaluations and improving tools. This may include connecting your agency with Intergovernmental Personnel Act (IP A) assignments or consultation from outside experts to help design and implement your proposals. For example, a number ofexternal organizations, such as the NYU Governance Lab, J-PAL North America, the Pew-MacArthur Results First initiative and the Coalition for Evidence-Based Policy are seeking Federal partners for evidence and innovation initiatives designed to improve results at the Federal, State, and local levels.
• Guidance and/or technical assistance in meeting government-wide requirements, including the Federal Acquisition Regulation, grants policy circulars, and Paperwork Reduction Act clearance requirements. For example, OMB helped USDA develop a generic clearance package to facilitate review and approval of behavioral insights research covered by the Paperwork Reduction Act.
Additional, up-to-date information on the workshop series, as well as on other available resources, can be found at https://max.gov/omb/evidence.
Next Steps
Agencies should work with senior leadership, including Deputy Secretaries; budget, performance and evaluation officials; program officials; and other relevant staff in order to (1) fulfill the requirements ofthe memo within your 2015 Budget submission; and (2) ensure participation in the EOP workshops and interagency collaboration.
As follow up, please designate up to two agency leads to work with policy, program, budget, evaluation and management support offices to coordinate agency participation in the workshops and send these to Dan Rosenbaum and Andy Feldman ofOMB at evidence@omb.eop.gov by August 15th. Agency leads should be well positioned to ensure workshop participants are able to engage with senior agency leadership on potential applications ofnew tools and approaches. If agencies have suggestions on other topics for workshops, would prefer to have less formal exploratory meetings to discuss preliminary ideas, or are interested in accessing the types oftechnical assistance mentioned above, please send those suggestions and requests to evidence@omb.eop.gov or to your OMB Resource Management Office points of contact.

Attachment A
Examples of Evidence and Innovation Strategies and Tools
Administrative data collected by Federal, State, or local agencies to run programs can be a valuable resource for program improvement and for helping agencies, consumers, andproviders make more informed decisions.
(1) Linking data across programs and levels of government while fully protecting privacy
Linking data across programs can lower evaluation costs and improve their quality, streamline reporting requirements for program providers and participants, and answer important questions about program performance. A number of Federal agencies are cunently developing or using protocols and processes to share personally identifiable data to permit such linkages in ways that fully adhere to laws, regulations, and policies designed to protect individual privacy and confidentiality.
• Example: The Department of Housing and Urban Development has partnered with the Department ofHealth and Human Services to match HUD administrative data with Centers for Medicare & Medicaid Services data. The two agencies recently completed a successful match that will improve understanding of the characteristics of seniors living in publicly subsidized housing and how supportive housing interventions may affect their health care use.
(2) Provider scorecards
Reliable data from government agencies can be used to create provider scorecards that compare how well different service providers perform. Scorecards are a tool for agencies and consumers to make more informed decisions and choices-and for providers to better understand and improve their performance. If data on participant characteristics are available, such as education level or income, scorecards can go a step further by enabling more detailed comparisons of alternative providers that serve people with similar characteristics.
• Example: The College Scorecard, launched earlier this year, highlights key indicators about the cost and value of colleges and universities to help high school students choose a post-secondary school that meets their needs. It is produced by the Department ofEducation and posted on its web site. The Scorecard includes data on costs, graduation rates, loan default rates, and average student debt-and average earnings ofrecent graduates will be added soon.
Many innovative companies use rapidly conducted randomized field trials to identifY highimpact
innovations and move them quickly into production. In the public sector, low-cost, frequent field tests do not replace longer-term, rigorous evaluations -they supplement them.

They allow innovative administrators to say: "Might this help boost results? Let's try it and see ifit works. "
(1) Applying behavioral insights to improve results and lower costs in direct operations
Human decision making is central to many public policy interventions. Major advances have been made in research regarding the influences that drive people's decisions and choices, and these new insights can significantly improve policy outcomes at a lower cost.
• Example: Research has revealed the power of"social norms" on behavior, meaning the influence of what others do on our decisions. Building on this insight, the Fiscal Service at the Treasury Department has recently updated the text and format of letters sent to individuals with delinquent debt to the federal government. The new letters, which will be tested against the older version using a randomized control trial, use simplified · language, personalization, and a reference to social norms (i.e., the fact that 94% of outstanding debts are paid off on time and that the recipient is in the fraction that has not yet paid) to motivate a higher rate ofdebt repayment.
(2) Using high-quality evaluation to answer important policy and program questions
Rigorous impact evaluations, especially those using random assignment to program and control groups, can provide strong evidence on key policy or program questions within an agency. They can help determine whether a program works and whether an alternative practice might work better.
• Examples: Current Federal evaluations cover a diverse set ofissues, including the Occupational Safety and Health Administration examining the effectiveness of on-site consultation, inspections, and corrective action letters on worker injury/illness rates, the Millennium Challenge Corporation examining the impact of road improvements in El Salvador or commercial training activities in Ghana, and the Department of Energy examining the effects of smmi grids and dynamic pricing on household energy use.
(3) High-quality, low-cost evaluations that piggy-back on existing programs and datasets
By drawing on existing data to measure outcomes and on program changes that are being implemented anyway, agencies can conduct high-quality randomized evaluations at low cost. For example, when a program change is being phased in gradually or a program is oversubscribed, pmiicipants could in some cases be selected based on random assignment, allowing for rigorous evaluation.
• Example: Hawaii's Opportunity Probation with Enforcement (HOPE) Program is a supervision program for drug-involved probationers. The program was evaluated using a randomized control trial at a cost of about $150,000 for the evaluation. The low cost for this rigorous evaluation was achieved by measuring outcomes using administrative data (e.g., arrest records) that the state already collected for other purposes, rather than doing costly new data collection. The study found that HOPE group members were 55 percent less likely than control group members to be re-arrested during the first year.
Because many Federal dollars flow to States, localities, and other entities through competitive andformula grants, grant reforms are an important component ofstrengthening the use of evidence in government. The goals include encouraging a greater share ofgrant fimding to be spent on approaches with strong evidence ofeffectiveness and building more evaluation into grant-making so we keep learning more about what works.
(1) Pay for Success
Pay for Success offers innovative ways for the government to pminer with philanthropic and private investors to fund proven and promising practices and to significantly enhance the return on taxpayer investments. Under this model, investors provide the up-front capital for social services with a strong evidence base that, when successful, achieve measurable outcomes that improve the lives offamilies and individuals and reduce their need for future services. Government pays when these measurable results are achieved. The PFS model is particularly well-suited to cost-effective interventions that produce government savings, since those savings can be used to pay for results.
• Examples: The Department ofJustice is coordinating PFS projects to use more effective prisoner re-entry interventions to reduce recidivism and its associated costs. And the Department ofLabor has launched an effmi to test new and more effective strategies for delivering workforce development and preventative social services that cut across existing program siloes, increase job placement and improve job retention.
(2) Tiered-evidence grant designs
"Tiered-evidence" or "innovation fund" grant designs focus resources on practices with the strongest evidence, but still allow for new innovation. In a three-tiered grant model, for example, grantees can qualify for 1) the "scale up" tier and receive the most funding; 2) the "validation" tier and receive less funding but evaluation support; or 3) the "proof of concept" tier and receive the least funding, but also suppmi for evaluation. With a tiered-evidence approach, potential grantees know that to be considered for funding, they must provide demonstrated evidence behind their approach and/or be ready to subject their models to evaluation. The goal is that, over time, interventions move up tiers as evidence becomes stronger. So far five agencies have launched or proposed 13 tiered grant programs in the areas such as education, teenage pregnancy prevention, home visitation programs, workforce, international assistance, and more.
• Example: The Department ofEducation's Investing in Innovation Fund (i3) invests in high-impact, potentially transformative education interventions, ranging from new ideas with significant potential to those with strong evidence of effectiveness that are ready to be scaled up. Based on the success ofi3, the Department recently issued proposed regulations that would allow its other competitive grant programs to adopt this threetiered model.
(3) Performance Partnerships and Waiver Demonstrations
Performance Partnership pilots enable States and localities to demonstrate better ways to use resources, by giving them flexibility to pool discretionary funds across multiple Federal programs serving similar populations and communities in exchange for greater accountability for results. With waiver demonstrations, Federal agencies suspend certain programmatic requirements in discretionary or mandatory programs to support State and local innovations that are then rigorously evaluated to learn what works and what is cost effective.
• Example: The 2014 Budget would authorize up to 13 State or local performance partnership pilots to improve outcomes for disconnected youth. Pilot projects would support innovative, efficient, outcome-focused strategies using blended funding from separate youth-serving programs in the Departments of Education, Labor, Health and Human Services, Housing and Urban Development, Justice, and other agencies.
(4) Using competitive grants to promote use of evidence in formula grants
Formula grant programs are often the largest grant programs in government, so they are a critical area for advancing more results-focused government. Agencies can improve the effectiveness of formula grant programs by using competitive grants to encourage adoption of evidence-based approaches within formula grants. For instance, agency competitions cim give preference points to State and local applicants implementing evidence-based practices with their formula funds. And formula grants to States can include set-asides for States to award competitively to promote use of evidence.
• Example: For HHS, the 2014 Budget proposes to require that States use five percent of their mental health block grant allocation for grants that use the most effective evidencebased prevention and treatment approaches. The Senate Appropriations Committee adopted this policy in its recent bill.
(6) Multi-phase grant competitions
The quality of grant-funded projects can be enhanced by conducting a multi-phase selection process. In the first phase, before selection, agencies can share research findings with potential applicants to ensure they are integrated into project designs and implementation strategies. Expert input can also be used to develop program models or variations within models that the grant program could test and evaluate. Moreover, preference points can be given to applicants that implement research-informed models and agree to participate in a rigorous evaluation. Multi-phase designs are particularly useful when there are many applications ofvarying quality, where a streamlined pre-application process can identify leading proposals.
• Example: The Promoting Readiness of Minors in the Supplemental Security (PROMISE) program began with coordinated planning by the Depmtments ofEducation, HHS, Labor and the Social Security Administration to review existing research and gather input from experts to develop an integrated service delivery model that was incorporated into the grant solicitation. The next phases are grantee selection and rigorous evaluation of grantees' approaches.
Evaluation is useful only to the extent that it is being used for decision making. An evaluation plan that focuses evidence-building resources on the most relevant and actionable issues helps generate useful knowledge. Common evidence standards and What Works Clearinghouses, meanwhile, help make existing evidence more useful to decision makers.
(1) Agency-wide evaluation plans
An agency-wide evaluation plan developed with senior policy and program officials can focus evaluation resources on high priority issues-for example, questions that are most important for improving program results-and on rigorous methodologies that produce actionable insights.
• Example: The Department of Labor has a Chief Evaluation Office (CEO) that works closely with program offices to develop and implement evaluation agendas set by policy officials. It also promotes high standards for data systems; monitors and reviews research and evaluation plans initiated by DOL agencies to ensure they are consistent with departmental goals and the highest standards of empirical rigor; works to institutionalize an evidence-based culture through seminars and forums on evaluation topics and findings; and maintains an active connection with outside experts to ensure that the Department is aware ofrelevant research and evaluation findings and activities.
(2) Common evidence guidelines for various types of research studies
Common research standards and evidence frameworks across agencies can facilitate evaluation contracting, information collection clearance, and the strengthening or creation ofresearch clearinghouses and repositories about "what works." They also help agencies use results from different types ofhigh quality studies to identify effective programs, improve programs, and encourage innovative new approaches.
• Example: Evaluation officials from the Departments of Education, Labor, Health and Human Services, and the National Science Foundation are jointly developing common evidence guidelines for research studies that can be a resource for improving the quality of studies throughout the Federal Government.
(3) Cross-agency learning networks
Inter-agency working groups of evaluation and program officials within the Federal Government can share best practices, including helping spread effective procurement practices, developing common evidence guidelines, and better integrating evaluation and performance measurement
efforts. Other cross-agency groups are forming learning networks around specific policy issues in order to share relevant research and develop shared evaluation strategies.
• Example: The Small Business Administration and the Departments of Agriculture and Commerce, with guidance from OMB and CEA, are working together with the Census Bureau to find more robust ways to evaluate the impact ofFederal business technical assistance programs. The goal of the working group is to develop a standard methodology for measuring the impact of these types of technical assistance programs across the Federal Government.
(4) What Works Clearinghouses
"What works" clearinghouses are repositories that synthesize evaluation findings in ways that make research useful to decision-makers, researchers, and practitioners. Moreover, as Federal innovation funds and other programs provide financial incentives for using and building evidence, these repositories provide useful tools for understanding what interventions are ready for replication or expansion and disseminating results.
• Examples: Cunent "what works" clearinghouses include the Department of Justice's CrimeSolutions.gov, the Department of Education's What Works Clearinghouse, the Substance Abuse and Mental Health Services Administration's National Registry of Evidenced-based Programs and Practices, and the Department ofLabor's new Clearinghouse ofLabor Evaluation and Research.

Attachment B
Details on Overview Briefing and Initial Workshops
Overview briefing: A kickoff briefing or call for agency leads will provide an overview of tools available to help programs strengthen their abilities to generate and use evidence to improve program performance. It will also preview the workshops. (First week ofSeptember)
The following is an initial list of workshops. OMB and White House policy councils will organize additional workshops on topics in Attachment A based on agency interest. An up-todate workshop schedule can be found at https://max.gov/omb/evidence.
Workshop 1: How can agencies focus evaluation resources on the most important program and policy questions? (Second week ofSeptember)
• Overview: This workshop will engage participants in a focused discussion about the strategies certain agencies use to focus rigorous, independent evaluation on high priority, actionable research questions. Examples will include the Department ofLabor's use of a Chief Evaluation Officer to coordinate agency-wide evaluation plans, including working with policy, program, evaluation and perfmmance management officials to create annual learning agendas for each division. Other examples will include the use of an evaluation policy statement by the Administration for Children and Families at the Department of Health and Human Services and the statutory structure ofthe Education Department's Institute for Educational Sciences, which led to significant improvements in the quality of ED's evaluations.
• Agency preparation and takeaways: Using a diagnostic checklist to assess the quality, relevance, and independence of their evaluation activities, participants in the workshop will assess the strengths of their own evaluation organizations and identify challenges and potential strategies for. overcoming them.
Workshop II: How can agencies use administrative data sets from multiple programs and levels of government to answer important questions while protecting privacy? (Date TBD)
• Overview: This workshop will examine several case studies where Federal agencies have answered compelling programmatic questions by linking data at the Federal level or with a State or local government or other entity. The session will explore:
o How to develop an effective partnership among all the parties involved, including policy officials, research experts, and legal counsel.
o What steps must be taken to ensure compliance with statutes, regulations, and policies governing privacy and confidentiality.
o How to design a data match to ensure it will answer key research questions, including strategies that use aggregated data.
• Agency preparation and takeaways: Participants should come to the workshop with at least one potential data sharing opportunity in mind that would help their agency toanswer an important performance or evaluation question. They will fill out a planning template during or after the session to apply the concepts they learn and help their agencies identify clear steps for progress.
Workshop III: How can agencies conduct rigorous program evaluations and data analytics on a tight budget? (Date TED)
• Overview: What low-cost strategies can agencies use to: (1) conduct strong program evaluations, including experimental and quasi-experimental studies, to identify effective strategies for delivering services and achieving program goals or (2) support data analytics on ways to achieve better results at lower cost? This workshop will review ways that agencies can:
o Embed testing of alternative strategies into their existing grant programs or direct operations.
o Maximize the use of high quality statistical or administrative data currently being collected and reduce the need for costly special purpose surveys.
o Form partnerships with academic experts, including using externally funded Intergovernmental Personnel Act (IP A) assignments, to design and conduct rigorous evaluations and data analyses and reduce evaluation costs.
• Agency preparation and takeaways: Participants should come to the workshop with one or more potential evaluation topics that focus on issues important to their agency. Participants will identify specific options to meet these evaluation needs based on the strategies discussed.
Workshop IV: How can agencies use their existing authority to turn a traditional competitive grant program into an innovative, evidence-based one? (Date TED)
• Overview: At this workshop, the Department ofEducation will explain how program and research officials partnered to design and implement the Investing in Innovation ("i3") program and how the same innovation fund (or "tiered-evidence") model is now being adopted by other programs across the agency. The Development Innovation Ventures (DIV) program at USAID, the Workforce Innovation Fund (WIF) at the Department of Labor, the Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program and the Teen Pregnancy Prevention Program (TPP) at the Department of Health and Human Services, and the Social Innovation Fund (SIF) at the Corporation for National and Community Service may describe their variations ofthe tiered model. The workshop will explore:
o What features make a grant program a good candidate to become an innovation fund?
o What are the perceived legal barriers and how might they be overcome?
o What expertise and resources are needed compared to a traditional grant program?
o What does an innovation fund grant solicitation look like?
o How does the selection process differ from a traditional program?
o How do these grant programs measure success?
• Agency preparation and takeaways: Participants should have at least one potential program candidate in mind when they attend the workshop. They will fill out a planning template during or after the session to apply the concepts they learn and help their agency consider which programs are the best candidates for the tiered-evidence approach.
Workshop V: How can agencies harness research findings from the social and behavioral sciences to implement low-cost approaches to improving program results? (Date TBD)
• Overview: This workshop will review ways in which agencies can apply empirical insights about human judgment and decision-making to federal programs and policies in order to improve outcomes or reduce costs. It will also explore how agencies can:
o Design and evaluate rigorous experiments, using randomized control trials where possible, to test the efficacy ofthese interventions.
o Fmm partnerships with academic experts, including using externally funded IPA assignments, in order to receive conceptual advice on cutting-edge research findings that should inform how policies are designed; and technical support on designing, evaluating, and iterating experimental field studies.
• Agency preparation and takeaways: Participants should come to the workshop with one or more potential program areas that could benefit from the application of low-cost behavioral solutions. Materials to help brainstorm about these areas will be provided in advance.