Pay for Success: Collaboration among Federal Agencies Would Be Helpful as Governments Explore New Financing Mechanisms: GAO

September 2015 United States Government Accountability Office Report to the Chairman, Committee on the Budget, U.S. Senate PAY FOR SUCCESS Collaboration among Federal Agencies Would Be Helpful as Governments Explore New Financing Mechanisms GAO-15-646  September 2015 PAY FOR SUCCESS Collaboration among Federal Agencies Would be Helpful as Governments Explore New Financing Mechanisms What GAO Found Pay for Success (PFS), also known as Social Impact Bonds, is a new contracting mechanism to fund prevention programs, where investors provide capital to implement a social service— for example, to reduce recidivism by former prisoners. If the service provider achieves agreed upon outcomes, the government pays the investor, usually with a rate of return, based on savings from decreased use of more costly remedial services, such as incarceration. Stakeholders from the 10 PFS projects in GAO’s study said that PFS offers potential benefits to all parties in the project. For example, governments can implement prevention programs that potentially lead to reduced spending on social services and transfer the risk of failing to achieve outcomes to investors. Roles of Organizations Involved in PFS Projects Highlights of GAO-15-646, a report to the Chairman, Committee on the Budget, U.S. Senate Why GAO Did This Study Federal, state, and local agencies play an important role in improving social outcomes for society’s most vulnerable populations. A small number of state, local, and foreign governments are employing PFS to fund efforts designed to better serve these vulnerable populations. GAO was asked to provide information about PFS. This report examines (1) how selected PFS projects have been structured and what potential benefits these projects can provide; (2) how selected PFS contracts have been structured to address potential project risks; and (3) the potential roles for the federal government’s involvement in PFS projects. To address these objectives, GAO reviewed relevant literature on PFS; selected 10 PFS projects that reflected a variety of policy areas and were in different stages of implementation in state and local governments in the United States and the United Kingdom, where PFS originated; reviewed key documents and interviewed stakeholders from organizations that played a role in the selected projects; and reviewed documents and interviewed officials from the eight federal agencies that participated in an informal PFS working group. What GAO Recommends To identify and broadly disseminate information on leading practices and lessons learned, the Director of OMB should establish a formal means for federal agencies to collaborate on PFS. OMB concurred with this recommendation and is working with agencies to explore options for collaboration. View GAO-15-646. For more information, contact Michelle Sager at (202) 512-6806 or sagerm@gao.gov. The PFS projects in GAO’s review sought to manage potential risks in the feasibility assessment, design, and implementation of complex PFS contracts. For example, without safeguards, there is a risk that paying for outcomes could create perverse incentives, such as focusing on individuals who are easiest to serve rather than those most in need. To address this risk, PFS contracts included various provisions, such as only including those with the greatest need in the evaluation that determines if the government makes payments. To date, federal government involvement in PFS has been limited. The Office of Management and Budget (OMB) has encouraged agencies to explore the use of PFS as appropriate. Potential roles federal agencies could play in PFS projects include making outcome payments or helping build capacity. We have previously reported that collaborative mechanisms, such as interagency groups, can be used to implement programs and share information. However, a formal mechanism for federal agencies to collaborate on PFS does not exist. Given the evolving nature of PFS, a mechanism for federal agencies to collaborate on PFS would increase access to leading practices. United States Government Accountability Office  Contents Letter 1 Background 5 PFS Projects Were Structured in a Variety of Ways and Sought to Achieve Potential Benefits, Such as Improved Social Outcomes and Cost Savings 11 Participating Organizations Sought to Manage Risks in the Feasibility Assessment, Design, and Implementation Phases of Complex PFS Contracts 33 The Federal Government Could Play a Role in Funding, Building Capacity for, and Managing Investor Risk in PFS, but a Formal Mechanism to Collaborate and Share Lessons Learned Does Not Exist 58 Recommendation for Executive Action 65 Agency Comments and Our Evaluation 66 Objectives, Scope, and Methodology 67 Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies 71 Overview of PFS Projects in Our Study 73 Example of United Kingdom Rate Card for Social Impact Bonds 77 Comments from the Office of Management and Budget 79 GAO Contact and Staff Acknowledgments 80 Appendix I Appendix II Appendix III Appendix IV Appendix V Appendix VI T ables Table 1: Federal Agencies with Funds Appropriated for PFS Projects 6 Page i GAO-15-646 Pay For Success Figures Table 2: Examples of Outcome Payment Structures from Selected Pay for Success Projects 45 Table 3: Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies 71 Table 4: Overview of U.S. PFS Projects in Our Study 73 Table 5: Overview of United Kingdom (UK) PFS Projects in Our Study 75 Figure 1: Types of Organizations Most Commonly Involved in Pay For Success Projects and the Roles They Play 7 Figure 2: Phases of a PFS Project 9 Figure 3: Roles and Funding Characteristics of the Massachusetts Juvenile Justice Pay For Success Initiative 14 Figure 4: Roles and Funding Characteristics of the New York City Rikers Island Project 16 Figure 5: Roles and Funding Characteristics of the New York Increasing Employment and Improving Public Safety Pay For Success Project 18 Figure 6: Roles and Funding Characteristics of the Utah High Quality Preschool Project 20 Figure 7: Roles and Funding Characteristics of the Peterborough Prison Project 22 Figure 8: Roles and Funding Characteristics of the London Homelessness Project 24 Figure 9: Roles and Funding Characteristics of the It’s All About Me Project 26 Figure 10: Estimated Savings and Benefits and Outcome Payments at Various Reductions in Recidivism Rates 39 Figure 11: United Kingdom Department for Works and Pensions (DWP) Rate Card 78 Abbreviations AIM4Fresno ARRA CEO CNCS DLS Asthma Impact Model For Fresno American Recovery and Reinvestment Act of 2009 Center for Employment Opportunities Corporation for National and Community Service Maryland Department of Legislative Services Page ii GAO-15-646 Pay For Success DOCCS DOJ DWP Education HHS HUD i3 IAAM IMLS JRI Labor LEAs MIECHV NY DOL NVQ OMB P3 PFS PROMISE RCT REA/RES SIF SIB SSA SSI Treasury UK New York Department of Corrections and Community Supervision Department of Justice UK Department for Works and Pensions Department of Education Department of Health and Human Services Department of Housing and Urban Development Investing in Innovation Fund It’s All About Me Adoption Project The Institute of Museum and Library Services Justice Reinvestment Initiative U. S. Department of Labor local educational agencies Maternal, Infant, and Early Childhood Home Visiting Program New York Department of Labor National Vocational Qualifications Office of Management and Budget Performance Partnership Pilots for Disconnected Youth Pay for Success Promoting the Readiness of Minors in Supplemental Security Income random control trial Reemployment and Eligibility Assessments and Reemployment Services Social Innovation Fund Social Impact Bond Social Security Administration Supplemental Security Income Department of the Treasury United Kingdom This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately. Page iii GAO-15-646 Pay For Success 441 G St. N.W. Washington, DC 20548 Letter September 9, 2015 The Honorable Mike Enzi Chairman Committee on the Budget United States Senate Dear Mr. Chairman: Federal, state, and local agencies play an important role in improving outcomes for our society’s most vulnerable populations, who often face complex social or economic problems. Agencies at all levels of government can improve the lives of vulnerable populations and make effective use of available resources by allocating them to prevention programs that demonstrate the promise to innovatively solve intractable social challenges. These challenges include reducing recidivism rates of recently released prisoners, helping homeless individuals find stable accommodation, and expanding access to quality early childhood education for economically disadvantaged children. Moreover, by defining and achieving prevention outcomes, agencies can potentially reduce future budgetary pressures by lessening the need for costly remediation programs. A small number of foreign and U.S. state and local governments are employing a contracting mechanism known as a Social Impact Bond (SIB)1 or a Pay for Success (PFS)2 program to fund preventive efforts designed to more effectively serve vulnerable populations. In a PFS contract, private or philanthropic capital is used to fund an intervention that targets the achievement of specific outcomes, such as a reduction of the recidivism rate and an increase in the employment rate of a target population. After confirming that outcomes have been achieved, a government entity repays the other funders, usually with a rate of return. 1The term SIB is used in the United Kingdom (UK). According to the UK Centre for Social Impact Bonds, a SIB is a way of financing a Payment by Results (PBR) contract. Under a PBR contract, government pays a service provider if it achieves certain results. 2Throughout this report, we refer to the mechanism as Pay for Success, as the term is used more widely than Social Impact Bond in the United States. Both terms are used by other entities to describe this contracting mechanism. Page 1 GAO-15-646 Pay For Success Proponents of PFS believe that this contracting mechanism allows government entities to implement innovative, evidence-based social services that better serve vulnerable populations, and that it does so at a reduced level of risk to agency budgets because government payments are contingent on achieved outcomes. On the other hand, skeptics of PFS believe that the mechanism is unlikely to reduce the level of risk to agency budgets and could actually increase a government entity’s cost of implementing a program as a result of the higher administrative costs associated with the complexity of the project design. To date, four federal agencies have allocated resources to support PFS projects. The President and members of Congress have made proposals to expand federal support for PFS projects. Since fiscal year 2014, the President’s budgets have proposed a $300 million appropriation for a new PFS Incentive Fund in the Department of the Treasury to expand the role of the federal government and support state and local governments and other intermediaries seeking to establish PFS projects. Members of Congress have also introduced legislation during the 113th and 114th Congresses on PFS to expand federal support for PFS projects.3 You asked us for information to help Congress have a better understanding of PFS. This report examines (1) how selected PFS projects have been structured and what potential benefits these projects can provide; (2) how selected PFS contracts have been structured to address potential project risks; and (3) the potential roles for the federal government’s involvement in PFS projects. To address these objectives, we identified and reviewed relevant literature on PFS, including work produced by public sector, nonprofit, academic, and private sector organizations, and interviewed individuals with public management expertise who had examined PFS and published work on or related to PFS. Based on the literature review and the interviews, we identified jurisdictions in the United States and abroad that were implementing, planning to implement, or studying the potential of PFS projects. For illustrative purposes, we selected seven U.S. and three United Kingdom (UK) cases to review in depth. We selected cases from the UK to capture the country’s experience as the first to develop and 3Social Impact Partnership Act, H.R. 1336, 114th Cong. (2015); Social Impact Partnership Act, S. 1089, 114th Cong. (2015); Social Impact Bond Act, H.R. 4885, 113th Cong. (2014); Pay for Performance Act, S. 2691, 113th Cong. (2014). Page 2 GAO-15-646 Pay For Success implement PFS projects as well as the location with the longest experience with this financing mechanism. For the U.S. projects we selected the four PFS projects that were being implemented in the United States at the time we began our review in July 2014. Those projects were the Massachusetts Juvenile Justice PFS Initiative; the New York City Rikers Island Project; the New York PFS project: Increasing Employment and Improving Public Safety; and the Utah High Quality Preschool Program. Because those projects focused heavily on reducing recidivism and improving early childhood education, we selected two additional U.S. projects that were in development to reflect an additional policy area (improving health outcomes). Those projects were the South Carolina Maternal Health PFS Initiative and the Asthma Impact Model for Fresno (AIM4Fresno). Finally, we selected one case in which a state considered but ultimately decided not to move forward with a PFS project. That case was Maryland’s review of a potential PFS initiative focused on reducing recidivism. For the UK projects, we selected three projects that varied in policy area, structure, and the participating organization that conceived them. Those cases were the Peterborough Prison Project; the London Homelessness Project; and the It’s All About Me Project on reducing wait times for adoption. We reviewed documents relevant to the case illustrations, including feasibility studies, contracts, and evaluation reports. We conducted in- depth interviews with representatives of organizations participating in each of these projects, including representatives from government entities, intermediaries who help manage PFS projects, service providers, investors, evaluators, and validators, who validate evaluation results. We refer to individuals from these organizations by using particular categories of stakeholders, such as “investors,” when the distinction is meaningful. We refer to them collectively as “stakeholders” when all organization representatives provided a similar point of view. To assess the PFS contract structure, we reviewed our prior reports and contracting literature to identify contracting practices from the federal experiences relevant to PFS contracts, which are complex service acquisitions. We compared these contracting practices to the PFS contracts in our study to establish their relevance to the design and implementation of PFS contracts. We also examined federal agencies’ involvement in PFS by reviewing relevant documents, including the President’s Budget for fiscal years 2012 through 2016. We identified eight federal agencies that were involved in an informal PFS working group and have played different roles Page 3 GAO-15-646 Pay For Success vis-á-vis PFS. The Office of Management and Budget (OMB), working with the White House Office of Social Innovation and Civic Participation, has provided informal guidance on PFS efforts and participated in the informal working group. The Department of the Treasury (Treasury) participated in the informal working group. The Departments of Labor (DOL), Housing and Urban Development (HUD), and Justice (DOJ) and the Corporation for National and Community Service (CNCS) have funded or plan to fund specific PFS projects. The Departments of Health and Human Services (HHS) and Education (Education) have been exploring opportunities to pursue the PFS concept. We interviewed officials from each of these agencies about their involvement in PFS. Of the 8 agencies in the working group, we also interviewed officials and reviewed budget documents from 6 agencies that typically fund social programs—CNCS, DOJ, DOL, Education, HHS, and HUD—to identify illustrative examples of how these agencies have used evidence-based approaches in their decision making more generally. We conducted our work from July 2014 to September 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 4 GAO-15-646 Pay For Success Background OMB Guidance Encourages Agencies to Use Evidence-Based Approaches, Including PFS Since 2010, OMB has used its annual budget guidance to broadly encourage federal agencies to use evidence to consider the effectiveness of their programs and to institutionalize the use of evidence to foster innovation rooted in research and rigorous evaluation. In a 2013 memo, OMB identified areas in which agencies could improve their use of evidence, including: (1) strengthening evaluation capacity and proposing new evaluations; (2) developing high-quality, low-cost evaluations and rapid, iterative experimentation; (3) using innovative outcome-focused design; and (4) increasing agency capacity to use evidence.4 In 2015, OMB required agencies to include an evidence template with their fiscal year 2017 budget submissions. The template should provide an overview of evidence-building strategies and identify related priorities.5 The six federal agencies in our study that typically fund social programs said they used evidence to inform a range of program proposals or decisions. See appendix 2 for illustrative examples of these agencies’ evidence-based programs. As part of its agenda to base management and policy decisions on evidence, the White House and OMB have encouraged agencies to explore PFS, where appropriate, as one tool to incorporate evidence into grant making. As of July 2015, four federal agencies have awarded PFS grants or are developing PFS grant proposals (see table 1). According to OMB’s 2013 Memo on the Next Steps in the Evidence and Innovation Agenda ...evidence includes evaluation results, performance measures, and other relevant data analytics and research studies, with a preference for high-quality experimental and quasi-experimental studies. Source: OMB | GAO-15-646 4Office of Management and Budget, Next Steps in the Evidence and Innovation Agenda, OMB Memorandum M-13-17 (Washington, D.C.: July 26, 2013). 5Office of Management and Budget, Fiscal Year 2017 Budget Guidance, OMB Memorandum M-15-11 (Washington, D.C.: May 1, 2015). Page 5 GAO-15-646 Pay For Success Table 1: Federal Agencies with Funds Appropriated for PFS Projects Agency    Program name   Amount   Description Department of Labor    Workforce Innovation   $24 million   In fiscal year 2013, DOL awarded grants to support outcome payments Fund for anti-recidivism PFS projects in New York and Massachusetts. The New York and Massachusetts PFS projects in our review were the recipients of these grants. (DOL) Corporation for    Social Innovation   $12 million   In fiscal year 2014, CNCS awarded $12 million in grants to National and Community Service (CNCS) intermediaries to provide technical assistance to states and agencies interested in pay for success projects that cover a range of issues, such as youth development, asthma management, homelessness prevention and storm water infrastructure. Fund Department of    PFS Supportive   Up to $10   In fiscal year 2014, HUD and DOJ entered into an interagency Housing and Urban Development (HUD) and Department of Justice (DOJ) Housing Demonstration million agreement to develop a demonstration project to test the effectiveness and cost-effectiveness of the permanent supportive housing concept funded via a PFS model. For this demonstration, the target population consists of people experiencing homelessness who are high users of corrections facilities, homeless services, health care and other crisis services. Source: GAO analysis of data provided by selected agencies. | GAO-15-646 PFS Is a Mechanism for Contracting for Social Outcomes PFS is a program funding mechanism that some U.S. state and local and foreign governments are using to contract for prevention services to address entrenched social problems. The first PFS project began in the United Kingdom in 2010. In a PFS contract, private investors provide upfront capital that a service provider uses to implement an evidence- based prevention program. An intermediary organization can solicit funding from the investors, facilitates the flow of capital from the investors to the service provider, and oversees the provider’s work throughout the project. If the program is successful, the government entity makes payments to the intermediary, which in turn may repay the investors their capital along with interest. These payments are, in theory, based on the savings that result from decreased use of government services. PFS is intended to allow government to transfer the risk of paying for a program that does not achieve desired outcomes from the government to other participating organizations. If the program is successful, the government obtains improved outcomes for vulnerable populations. In addition, PFS can potentially capture cost savings by reducing the need to provide costly remediation services to the individuals in vulnerable populations. However, stakeholders said that PFS can also be used to contract for outcomes that a government deems worthwhile, even if the outcomes do not generate savings. Because the government makes payments based on the service provider meeting outcome targets, all parties to the contract have a vested interest in the success of the service provider. Page 6 GAO-15-646 Pay For Success An individual PFS project comprises a series of contracts and other vehicles, such as loan and grant agreements, that defines entities’ roles in the project and identifies the risks they each agree to assume and the benefits they each stand to receive by virtue of their participation. Although individual PFS projects may vary in terms of roles and responsibilities for the parties to the contract, there are five main types of organizations that commonly participate in a PFS project: government, intermediaries, investors, service providers, and evaluators (see figure 1). Figure 1: Types of Organizations Most Commonly Involved in Pay For Success Projects and the Roles They Play Page 7 GAO-15-646 Pay For Success Government originates a contract for defined outcomes and pays on the basis of those outcomes being achieved. In the United States, state and local governments have initiated PFS projects to date, while in the UK both national and local-level governments have initiated them. Examples of government entities that have been involved in PFS projects include state or local programmatic agencies and central management offices, which often work together collaboratively on a project. As mentioned above, DOL and CNCS have awarded grants to support state and local PFS projects. Investors provide capital to finance a program. They provide capital on the condition that the government will repay the investment if outcomes are achieved, usually with a rate of return. In U.S. projects to date, capital has been provided by a mix of investor types, such as banks, foundations, and high net worth individuals, whereas in UK projects, capital has been provided solely by social impact investors.6 Intermediaries enter into a government contract in a PFS project and usually receive a fee for the services they provide. Intermediaries can take on a variety of roles in a PFS project, such as providing technical support to governments on the feasibility of a project, identifying the service provider, raising operating funds from investors, negotiating the terms of contracts or agreements with governments, service providers, and investors; disbursing payments to service providers; and overseeing day-to-day performance management of the project. Examples of intermediaries that have been involved in PFS projects include nationally focused nonprofit organizations with public policy and financial expertise and a locally focused nonprofit organization with experience implementing programs within local communities. Service providers enter into contracts with an intermediary or a government to administer the evidence-based prevention program. Examples of service providers that have been involved in PFS projects include nonprofit or charitable organizations with established track records of implementing programs in specific policy areas, such as job training or homelessness prevention. 6Social impact investors seek investments that intentionally target specific social objectives along with a financial return and measure the achievement of both types of outcomes. G8 Social Impact Investment Taskforce, Impact Investment: The Invisible Heart of Markets (Sept. 15, 2014). Page 8 GAO-15-646 Pay For Success Evaluators measure a program’s outcomes or evaluate a program’s effect on the target population. The evaluator’s work assures the other participating organizations that defined outcomes were achieved and are likely due to the program, rather than to other factors, including chance. Outcome payments are based on these findings. In some cases, a validator, independent of the evaluator, is responsible for certifying the evaluator’s findings. Examples of evaluators that have been involved in existing PFS projects include a research office within a government department and an organization with experience evaluating programs within a specific public policy area, such as early childhood education. The PFS projects we reviewed generally had three phases: a feasibility assessment, project design, and program implementation (see figure 2). Figure 2: Phases of a PFS Project Practices from Federal Experience with Complex Service Acquisitions Can Inform PFS Projects As described previously, a PFS project is a complex set of contracts through which multiple organizations work together to deliver social service programs. PFS projects are similar to complex acquisitions. Complex acquisitions are defined, in part, by the difficulty in defining precise terms of exchange for all the characteristics of the product or service at the time of the contract negotiation. We have previously identified key factors that government entities should incorporate into complex acquisitions to achieve improved service acquisition outcomes.7 Key literature on complex acquisitions we reviewed reinforces these 7GAO, Defense Acquisitions: Tailored Approach Needed to Improve Service Acquisition Outcomes, GAO-07-20 (Washington, D.C.: Nov. 9, 2006). Page 9 GAO-15-646 Pay For Success factors.8 Taken together, this understanding can inform the design and management of PFS contracts. To address the inherent ambiguity in complex acquisitions, parties can include processes in the contract to manage decision making and to make ongoing adjustments during implementation. Specifically, providing clear standards in the following areas can facilitate PFS contracts: • Valid and well-defined requirements. We have previously reported that establishing a valid need for a service, and then translating that need into a service acquisition requirement, is essential for obtaining the right outcome.9 Literature on complex contracting underscores the need for well-defined requirements, but states that it can be challenging to define all requirements upfront in a complex acquisition. Undefined requirements leave room for interpretation that, if not carefully managed, can lead to outcomes that are inconsistent with expectations.10 • Properly structured business arrangements. We have previously reported that once requirements have been defined, it becomes necessary to develop a business arrangement to meet the requirements and protect the government’s interests. At a basic level, properly structuring business arrangements includes defining a clear scope of expected contractor performance and developing an objective means of assessing the contractor’s performance. There should be clearly established relationships among the tasks the contractor is expected to perform, the contract terms and conditions, and performance evaluation factors and incentives.11 Literature on complex contracting underscores the importance of properly structured business arrangements, including governance rules, which should define how the government and contractor will interact to make decisions during the contract’s implementation.12 8Trevor L. Brown, Matthew Potoski, and David M. Van Slyke, Complex Contracting: Government Purchasing in the Wake of the U.S. Coast Guard’s Deepwater Program (New York: Cambridge University Press, 2013). 9GAO-07-20. 10Brown, Potoski, and Van Slyke, Complex Contracting. 11GAO-07-20. 12Brown, Potoski, and Van Slyke, Complex Contracting. Page 10 GAO-15-646 Pay For Success • Proactively managed outcomes. We have previously reported that managing and assessing post-award performance entails various activities performed by government officials to ensure that the delivery of services meets the terms of the contract, including adequate oversight resources, proper incentives, and a capable workforce for overseeing contractor activities. Each of these activities requires metrics and tools to encourage contractors to provide superior performance and to manage and document acceptable contractor performance.13 Literature on complex contracting also underscores the importance of rewarding successful outcomes, particularly in cases where it is difficult to define requirements, such as efforts to acquire services.14 PFS Projects Were Structured in a Variety of Ways and Sought to Achieve Potential Benefits, Such as Improved Social Outcomes and Cost Savings PFS Projects Were Structured to Implement Prevention Programs to Address Entrenched Social Problems In our selected case illustrations, governments used PFS to address entrenched social problems. The individuals who face these problems have complex, multi-faceted needs and are often poorly served by existing interventions or programs. The organizations participating in the PFS projects in our study said that these individuals can be better served through prevention programs focused on defined outcomes. Stakeholders told us that PFS is just one tool among many for addressing social problems and that PFS is not a panacea. 13GAO-07-20. 14Brown, Potoski, and Van Slyke, Complex Contracting. Page 11 GAO-15-646 Pay For Success Stakeholders said PFS is most appropriate in policy areas in which populations face complex, entrenched problems. One stakeholder also noted that PFS may work best on projects that supplement, not supplant, existing services, such as prevention programs for which existing services have been insufficient in achieving desired outcomes. Projects in our study used PFS to implement programs intended to reduce recidivism among an adolescent jail population, gaps in academic achievement among preschoolers, homelessness for difficult-to-serve individuals, wait times for adoption, and to provide preventive health care to address poor birth outcomes and asthma. Page 12 GAO-15-646 Pay For Success Massachusetts: Reducing Recidivism and Increasing Employment among Ex- Offenders The Massachusetts Juvenile Justice PFS Initiative, which began in October 2013, was designed to address high rates of recidivism among young men. Every year approximately 4,000 high-risk young men age out of the state’s juvenile justice system or are released from adult probation.15 Sixty-four percent of these young men are incarcerated at least once within 5 years of release. Since its initial design, the project has been expanded to serve young men connected to the adult corrections system who are at high risk of future adult incarceration. The project is being used to fund an intervention provided by Roca, Inc.,16 which has been shown to reduce incarceration rates for the young men it serves by 33 percent. Massachusetts will make payments on the basis of reduced incarceration, increased employment, and for each individual program participant who meets 9 or more times per quarter with a Roca staff member to discuss job readiness issues. The project’s target impact is a 40 percent reduction in participants’ days in incarceration compared to a control group during the 5 years following their enrollment in the program. At that level, Massachusetts expects to make $21 million in outcome payments and realize $22 million in gross savings during the approximately 6-year lifespan of the project. If participants’ days in incarceration are reduced by 55 percent, Massachusetts expects to make the maximum $28 million in outcome payments and realize $33 million in gross savings. The U.S. Department of Labor (DOL) awarded Massachusetts an $11.7 million grant for the project. Up to $10.8 million of this amount is available for outcome payments associated with the first 2 years of service for up to 535 young men.17 Massachusetts made its first payments related to the job readiness metric in May 2015. The state will make its first payments related to the incarceration and employment metrics in the fall of 2017. See figure 2 for an example of the roles of the organizations participating in the project and the size of the project’s investment and potential savings. 15Third Sector Capital Partners, The Massachusetts Juvenile Justice PFS Initiative: Project Brief (Boston, MA: 2014). 16Roca is a nonprofit organization that works with high-risk young people ages 17 to 24 in Massachusetts to help them break cycles of poverty, violence, and incarceration. 17The remainder of the grant will fund the project’s evaluation and other administrative costs. Massachusetts Juvenile Justice PFS Initiative The “Roca Model,” which the project is funding, includes 4 basic elements: outreach to young men; intensive case management; life skills, educational, prevocational, and employment programming; and a connection to work opportunities. Project stakeholders believe that the Roca Model will be able to help young men exiting the criminal justice system to reduce the average 2.4 years they spend in prison for new convictions after they are released. Source: Commonwealth of Massachusetts, Roca, and Third Sector Capital | GAO-15-646 Page 13 GAO-15-646 Pay For Success Figure 3: Roles and Funding Characteristics of the Massachusetts Juvenile Justice Pay For Success Initiative aMassachusetts contracted with Third Sector Capital Partners and Roca for the Juvenile Justice PFS Initiative. bThe senior loan will be repaid before the junior loan, and receive higher returns. cThird Sector Capital Partners and Roca will be repaid their deferred service fees if the program achieves its target impact of a 40 percent reduction in days in incarceration. Page 14 GAO-15-646 Pay For Success New York City: Reducing Recidivism in an Adolescent Jail Population Adolescents released from Rikers Island jail in New York City have a 47 percent chance of returning to jail within 1 year.18 The New York City Rikers Island Project, which began in September 2012, was designed to address this recidivism rate. To do so, Osborne Association and Friends of Island Academy19 administered a form of cognitive behavioral therapy called Moral Reconation Therapy to adolescents at Rikers Island.20 Project participants estimated that the city could save $1.7 million in net long-term costs if participants’ future days in jail are reduced by 11 percent compared to a similar group that did not receive the therapy. They estimated the city could save $20.5 million if participants’ future days in jail are reduced by 20 percent. In July 2015, the project’s evaluator determined that the program did not lead to reductions in recidivism rates for participants in its 1-year assessment of the project. The New York City government did not make outcome payments to investors and the program was discontinued as of August 31, 2015. See figure 3 for a description of the roles of the organizations participating in the project and the size of the project’s investment and potential savings. This project grew out of the Mayor Bloomberg Administration’s Young Men’s Initiative (YMI), an effort to improve the lives of black and Latino men, who experience much higher rates of poverty, unemployment, and homicide than their white and Asian peers. Ideas for programs within YMI exceeded available funding, so city leaders decided to try to fund a program within YMI with PFS, which they saw as a way to fund an innovative approach during a period of constrained government funding. Source: MDRC | GAO-15-646 The New York City Rikers Island Project 18MDRC, Financing Promising Evidence-Based Programs: Early Lessons from the New York City Social Impact Bond (New York, NY: Dec. 2013). 19The Osborne Association is a nonprofit social service organization that provides treatment, educational, and vocational services to men and women involved in the criminal justice system in New York. Friends of Island Academy is a nonprofit youth development organization that provides prerelease and transitional services on Rikers Island and neighborhood-based services in education, employment and well-being for high-risk New York City youth involved in the justice system 20Moral Reconation Therapy is a cognitive behavioral program designed to help offenders reevaluate their choices and enhance their decision-making abilities. Page 15 GAO-15-646 Pay For Success Figure 4: Roles and Funding Characteristics of the New York City Rikers Island Project aNew York City contracted with MDRC for the Rikers Island Project. In July 2015, the evaluator determined that the program did not lead to reductions in recidivism rates for participants in its 1-year assessment of the project. The New York City government did not make outcome payments to investors and the program was discontinued as of August 31, 2015. Page 16 GAO-15-646 Pay For Success New York: Increasing Employment and Improving Public Safety The New York PFS project, which began in October 2013, was designed to reduce the high recidivism rate and increase the low employment rate of high-risk individuals recently released from state prison each year. According to a report on the project,21 these individuals spend an average of 460 days in prison or jail in the 5 years following their release. The project is being used to fund an intervention by the Center for Employment Opportunities (CEO).22 A random control trial (RCT) evaluation showed CEO’s intervention reduced recidivism for recently released participants by 16 to 22 percent and days incarcerated for high- risk participants by 30 percent. New York will make payments on the basis of reduced recidivism, increased employment, and engagement in transitional jobs for recently released, high-risk individuals. New York estimates that it can capture savings and benefits ranging from $13 million if recidivism is reduced by 10 percent to $37 million if recidivism is reduced by 40 percent over a 5-year period. The U.S. Department of Labor (DOL) awarded New York a $12 million grant for the project. Up to $11.1 million is available for outcome payments associated with the first 2 years of the project.23 The first outcome payment would occur in 2017. See figure 4 for a description of the roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits. New York: Increasing Employment and Improving Public Safety New York recently established the broader “Work for Success” initiative, which aims to improve the process by which those who have served time in prison are trained and connected to businesses looking to hire. The state has launched the PFS project to complement the initiative because it focuses on delivering results for hard-to-serve, formerly incarcerated individuals, and ensures that resources are only expended if results are achieved. Source: State of New York, Center for Employment Opportunities, and Social Finance | GAO-15-646 21State of New York, Center for Employment Opportunities, and Social Finance, Investing in What Works: “Pay for Success” in New York State: Increasing Employment and Improving Public Safety (March 2014). 22The Center for Employment Opportunities is a nonprofit organization that provides employment services, including life skills education, short-term paid transitional employment, and full-time job placement, for men and women with criminal records. 23The grant will also be used for other expenses, such as funding the validator for the project. Page 17 GAO-15-646 Pay For Success Figure 5: Roles and Funding Characteristics of the New York Increasing Employment and Improving Public Safety Pay For Success Project aNew York contracted with Social Finance for the Increasing Employment and Improving Public Safety project. bThe first-loss guarantee is intended to protect up to $1.3 million of investor principal, or approximately 10 percent of the total capital raised. Page 18 GAO-15-646 Pay For Success Utah: Reducing Gaps in Academic Achievement among Preschoolers In 2011, a 3-year study conducted by Utah’s Granite School District showed that between 25 percent and 33 percent of low-income students in the district would likely be placed in special education without a preschool intervention.24 The Utah PFS project was designed to decrease the use of special education and to address academic achievement gaps for low-income students who enter kindergarten without having attended high-quality preschool. The project began in school year 2013-2014, with outcome payments being supported by Salt Lake County and the United Way of Salt Lake. Beginning in school year 2014-2015, Utah became the outcome payor. The project aims to expand Granite School District’s high-quality preschool program statewide, which evidence has shown reduces by 95 percent the need for special education among the low-income students who would otherwise likely be placed in special education programs. Economically disadvantaged students who are eligible to receive a free or reduced price lunch are eligible to be enrolled in the program. Of these students, those with the most extreme negative scores (as reflected by their scoring two standard deviations below the mean) on a test taken at the beginning of preschool are tracked from kindergarten through sixth grade to verify whether they are enrolled in special education.25 Beginning in the summer of 2016, Utah will make annual payments to investors that equal 95 percent of cost savings for each year from kindergarten through sixth grade that each designated child is not enrolled in special education. Utah estimates it can save approximately $2,700 per year if a child is not enrolled in special education. See figure 5 for a description of roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits. 24Janis Dubno, Voices of Utah Children, A Sustainable Financing Model High Quality Preschool for At-Risk Children: Results from the Granite School District in Utah (Salt Lake City, UT: Sept. 7, 2011). 25According to the project intermediary, the predicted use of special education is based on scoring at or below 70, or two standard deviations below the mean, on the Peabody Picture Vocabulary Test. According to the intermediary, research has supported a relationship between the test and later school and special education outcomes. The Peabody Picture Vocabulary Test measures receptive vocabulary in English. Utah High Quality Preschool Project The United Way of Salt Lake (UWSL), which led the development of the Utah PFS project in partnership with Voices for Utah Children, Salt Lake County, and Granite School District, said that the project is a component of its focus on “collective impact,” a place-based, evidence-driven, collaborative approach to providing social services. UWSL believes the PFS model is a mechanism for leveraging a cross-sector collaborative strategy to bring the effective preschool program to scale to improve outcomes for more at-risk children. Source: United Way of Salt Lake | GAO-15-646 Page 19 GAO-15-646 Pay For Success Figure 6: Roles and Funding Characteristics of the Utah High Quality Preschool Project aUtah contracted with the United Way of Salt Lake for the High Quality Preschool Project. bThe subordinate loan funding reduces risk to the senior lender if the preschool program proves to be ineffective. Page 20 GAO-15-646 Pay For Success United Kingdom: Improving Services for Short-Sentenced Prisoners The Peterborough Prison Project, which began in September 2010, was designed to reduce the recidivism rate among prisoners released from Peterborough Prison after serving a sentence of less than a year by improving their access to supportive social services. The project is being used to fund a range of services for the prisoners, including housing assistance, drug or alcohol treatment, and mental health support. The UK Ministry of Justice agreed to make outcome payments on the basis of reductions in recidivism of 10 percent for each program cohort or group of prisoners served during a 2-year time frame, or a 7.5 percent reduction for the three cohorts as a whole.26 In April 2014, the Ministry of Justice announced that it would close the Peterborough Project 2 years early because of a shift to the Transforming Rehabilitation program, which will provide statutory services similar to those provided in the Peterborough Project for all short-sentenced prisoners. See figure 6 for a description of roles of the organizations participating in the project and the size of the project’s investment and potential savings and benefits. United Kingdom: Peterborough Prison Project Social Finance, the project’s intermediary, determined social impact bonds could be effective in the policy area of criminal justice. Specifically, it identified the challenges that were preventing social service organizations from successfully helping short-sentenced prisoners reduce their recidivism rates and worked with stakeholders to develop a project that would help to overcome the challenges and improve the prisoners’ outcomes. Source: Social Finance | GAO-15-646 26Social Finance, Social Impact Bonds: The One Service. One Year On (London, United Kingdom: November 2011). Page 21 GAO-15-646 Pay For Success Figure 7: Roles and Funding Characteristics of the Peterborough Prison Project aThe UK Ministry of Justice contracted with Social Finance (UK) for the Peterborough Prison project. bA special purpose vehicle is a legal entity that is created solely for a particular financial transaction or to fulfill specific objectives. Page 22 GAO-15-646 Pay For Success United Kingdom: Improving Services for Homeless Individuals The London Homelessness Project, which began in November 2012, was designed to improve outcomes for a cohort of entrenched homeless individuals who do not respond to traditional models of resettlement and who are not targeted by other initiatives. The project funds two providers to deliver services tailored to the specific needs of the individuals within the cohort. The Greater London Authority will pay the providers for achieving outcomes including a reduction in the number of individuals sleeping on the street; an increase in individuals who sustain stable accommodation and gain employment; a reduction in average use of the healthcare system for accident or emergency episodes; and reconnecting individuals to a country outside of the UK if he or she is a foreign national. Targets for each of these outcomes are based on historical data on homeless individuals within London. The project does not include an evaluation for the purposes of determining outcome payments because it was not possible to create a matched comparison group for the cohort. This is because other cities in the UK do not have as complete data on homeless individuals as does London and the cohort is unique within London. A feasibility study estimated the net present value of providing services across the five outcome areas to the entire cohort over 5 years to be approximately £30 million (approximately $49 million). If all of the project outcomes are achieved, the maximum outcome payments and administrative costs will total £5 million (approximately $8 million), thus resulting in substantial public sector savings.27 See figure 7 for a description of roles of the project’s participating organizations. United Kingdom: London Homelessness Project The Mayor of London in 2012 established the Rough Sleeping Commissioning Framework 2011-2015, which outlined 17 different services for homeless individuals. The London Homelessness Project sits alongside those services and was designed to support homeless individuals who are unable to access some services, due to complex needs or antisocial behavior. Source: UK Department for Communities and Local Government | GAO-15-646 27United Kingdom Department for Communities and Local Government, Qualitative Evaluation of the London Homelessness Social Impact Bond: First Interim Report (London, United Kingdom: September 2014). Page 23 GAO-15-646 Pay For Success Figure 8: Roles and Funding Characteristics of the London Homelessness Project aThe Greater London Authority contracted with Thames Reach and St. Mungo’s Broadway for the London Homelessness project. Page 24 GAO-15-646 Pay For Success United Kingdom: Reducing Wait Times for Adoption The It’s All About Me (IAAM) adoption project, which began in October 2013, was designed to find adoptive homes for children that otherwise would not find them. The project funds adoption agencies to recruit parents to adopt hard-to-place children. The agencies also provide these parents with support services. The program’s goal is for up to 140 of these children to be adopted each year. A UK local government pays a service provider defined amounts at different points in the adoption process, such as when a service provider places a child with an adoptive family, and after 1 and 2 years of the child remaining with the adoptive family. According to the project intermediary, while the maximum cost is greater than what a local authority usually would pay adoption agencies, it is also estimated to be about half of the cost of a child remaining in foster care for 2 years.28 Evaluations are being conducted on the health and psychological outcomes for the adopted children; outcomes for families who adopt the children; and how decision making changes within local governments as a result of the project. See figure 8 for a description of the roles of the organizations participating in the project. United Kingdom: It’s All About Me Project Many UK children wait for over a year for adoption while in care of the government. The program seeks adoptive parents for hard-to- place children and provides these parents with training and support during the adoption process. Source: IAAM | GAO-15-646 28It’s All About Me, How IAAM Works, accessed May 14, 2015, http://iaamadoption.org/?page_id=9. Page 25 GAO-15-646 Pay For Success Figure 9: Roles and Funding Characteristics of the It’s All About Me Project South Carolina Maternal Health PFS Initiative aA local authority contracts with an adoption agency on a case-by-case basis. In addition to the case illustrations identified above, we examined 3 additional PFS initiatives that have yet to be implemented or were not implemented. Stakeholders in South Carolina are developing a PFS initiative focused on controlling costs and improving health and other outcomes of mothers and newborns in South Carolina’s Medicaid program. The project is expected to expand the Nurse-Family Partnership program, which, according to stakeholders, is an evidence-based nurse home visiting program for first-time, low-income mothers and their children, to rural and Page 26 GAO-15-646 Pay For Success Asthma Impact Model for Fresno (AIM4Fresno) under-served parts of the state.29 Trained professionals provide services and support to pregnant women and families with young children, primarily during visits to families’ homes. According to a feasibility study prepared for the project, multiple trials and evaluations have demonstrated NFP’s effectiveness, which includes reduced emergency room visits and better health outcomes in later years.30 In Fresno, California, a nonprofit organization is conducting a demonstration project to determine whether a PFS project is feasible. The project, which began in March 2013, was designed to improve the health of low-income children with asthma and reduce the costs that result from emergency treatments. Fresno, according to the stakeholders, has one of the highest asthma rates in the United States, and approximately 20 percent of children there have been diagnosed with the disease. The stakeholders are conducting an evaluation to demonstrate the social and financial benefits of a comprehensive in-home asthma management program for children at high risk of asthma emergencies.31 The program includes in-home education, environmental assessment, and remediation, such as mold removal to help manage asthma. Based on the outcomes of the demonstration project, stakeholders plan to pursue PFS financing to scale up the program to more children who can benefit. In January 2013, the Maryland Department of Legislative Services (DLS) published the findings of its review of the feasibility and potential benefits and risks of a PFS project for financing reentry programs for state prisoners.32 The review considered, among other things, whether a PFS project could generate cost savings for the state. In its review, DLS assumed that the PFS project would fund a pilot program to serve 1,250 participants over 5 years and reduce recidivism by 10 percent or 20 29South Carolina Department of Health and Human Services, Request for Information: Social Impact Bond RFI #1 (Sept. 17, 2013). 30Institute for Child Success, Using Pay for Success Financing to Improve Outcomes for South Carolina’s Children: Results of a Feasibility Study (Greenville, S.C.: September 2013). 31Social Finance, The California Endowment Awards Grant to Social Finance and Collective Health (Boston, MA: March 25, 2013). 32Maryland Department of Legislative Services, Office of Policy Analysis, Evaluating Social Impact Bonds as a New Reentry Financing Mechanism: A Case Study on Reentry Programming in Maryland (Annapolis, MD: January 2013). Maryland Review of a Potential PFS Initiative Page 27 GAO-15-646 Pay For Success Governments, Service Providers, and Investors Pursued PFS for Potential Performance and Financial Benefits Potential Benefits for Governments percent. DLS concluded that even if the pilot program reduced recidivism by 20 percent, that result would be insufficient to close a wing of a prison or an entire prison facility, which DLS stated are the largest cost savings associated with avoided reimprisonment. Therefore, the state would be unable to realize cost savings through the project. Maryland did not pursue a PFS project to finance reentry programs after the report was released. Stakeholders in PFS projects stated that governments face significant challenges in implementing programs that effectively address entrenched social problems. They also stated that government can benefit from using PFS as a tool to address these challenges. The benefits of PFS include the following: Funding for prevention programs and potential cost savings. Government stakeholders said constrained budgets in recent years have made it difficult for governments to fund new prevention programs, even though they would potentially lead to cost savings resulting from a reduced demand for remediation services. Because investors provide capital that a government only has to repay if the specified outcomes are achieved, a PFS project helps governments implement programs that they might not otherwise be able to fund. In addition, if successful, a PFS project reduces the need for remediation services and associated budget pressures and, therefore, frees up additional resources that government can reinvest in priority programs. For example, an official from the Massachusetts Executive Office of Administration and Finance told us that any savings attributable to the Massachusetts Juvenile Justice PFS project will likely be reinvested in the same or similar programs. Further, the recipients of the prevention program are expected to benefit from improved outcomes, such as better education outcomes resulting from high-quality preschool. The ability to implement innovative programs by transferring risk. PFS is designed so that the government only pays for a program if its outcomes are achieved, thus shifting the risk of failure to investors. Page 28 GAO-15-646 Pay For Success Stakeholders said that PFS creates incentives to both scale prevention programs where strong evidence exists and to innovate where it does not exist. For example, the Massachusetts project expanded a proven program to reduce recidivism rates. The service provider had extensive experience working with young men in Massachusetts at risk of recidivating and had successfully reduced their incarceration rates by 33 percent. The Massachusetts project expanded the service to more than 900 young men in Boston and Springfield, areas where the service provider had extensive experience. If successful, Massachusetts will pay for the program and expects to benefit from improved social outcomes and cost savings in its budget. If the service provider is unsuccessful, Massachusetts does not pay. PFS also creates incentives to innovate and apply evidence-based approaches to new or different populations. In the New York City project the intervention was administered in a new environment, a jail, with a service provider with prior experience working in prisons, but without prior experience with the specific intervention. New York City and the project intermediary chose Moral Reconation Therapy for the intervention because of the available evidence base and its flexibility in implementation that could be adapted to a jail setting. The intermediary selected a provider to administer the service. The PFS structure allowed the project’s stakeholders to implement the intervention at Rikers Island for the first time while providing the city with assurance that investors will absorb financial loss if the intervention does not reduce participants’ recidivism. When the intervention did not reduce recidivism at the target rates, New York City did not pay for the program.33 A potential benefit of PFS is the opportunity for governments to implement innovative programs without an established evidence base by transferring risk to investors. In practice, investors told us they prefer to back programs that already have a rigorous evidence base because these programs have a known likelihood for success. In Maryland, the DLS concluded that PFS may limit the implementation of innovative programs and government’s opportunity to apply evidence-based approaches to new populations because investors would be most likely to back projects with well-established records of success in order to 33Vera Institute of Justice, Impact Evaluation of the Adolescent Behavioral Learning Experience (ABLE) Program at Rikers Island: Summary of Findings (July 2015). Page 29 GAO-15-646 Pay For Success minimize their risk. Therefore, PFS may exclude new service providers and program types because of the perceived risk. One of the organizations involved in the PFS projects in our review did not think that risks would be transferred from a government to investors. For example, in its review,34 the Maryland DLS concluded that in practice there are significant challenges to transferring risk to investors, such as identifying investors who are willing to invest in PFS projects and designing contract mechanisms to fully transfer risk. This was a factor in the DLS recommendation against using a PFS contract for a prisoner reentry program. The Maryland DLS stated that it may be difficult to design contract provisions to provide an enforcement mechanism to prevent investors from terminating the contract early. In addition, during the implementation of the project, the costs to administer the intervention may be greater than the investors’ upfront funding and it is not clear whether the government or the investors would be responsible for those additional costs. Collaboration across government agencies. Government stakeholders told us that many social services are “siloed” within governments. In other words, individual agencies, largely on their own, design, budget for, and implement narrowly targeted social services, such as job training courses or temporary accommodation for homeless individuals. These services are usually not integrated with others that could collectively help vulnerable individuals overcome complicated issues and are potentially not as effective as a more integrated approach. PFS projects target outcomes for populations or individuals rather than the output of specific services, which would be the responsibility of a specific agency. PFS projects bring together multiple government agencies to work collaboratively to find solutions for difficult social problems. For example, the New York project targets improved employment and recidivism outcomes for program participants and therefore brought together multiple state agencies to oversee their achievement. Officials from the New York Department of Labor and the Department of Corrections and Community Supervision are members of an executive steering committee that provides strategic direction and represents the project across agency boundaries. Similarly, a stakeholder in Massachusetts said that PFS can 34Maryland Department of Legislative Services, Office of Policy Analysis, Evaluating Social Impact Bonds as a New Reentry Financing Mechanism: A Case Study on Reentry Programming in Maryland (Annapolis, MD: January 2013). Page 30 GAO-15-646 Pay For Success Potential Benefits for Service Providers help break down silos because it can bring together multiple agencies to work together on an important social problem. Sustained focus for entrenched social problems. Government stakeholders told us that governments find it challenging to sustain focus on difficult social problems over multiple years. Governments generally operate on annual budget cycles, making it difficult to set resource allocations over multiple years. In addition, changes in political leadership may shift policy priorities and support for levels of funding or program continuation. PFS contracts include upfront agreements about investor funding levels over multi-year time frames, providing sufficient time to collect, track, and analyze data, and measure outcomes that take multiple years to achieve. A focus on outcomes and building an evidence base. Government stakeholders told us that governments sometimes are not focused on their programs’ outcomes because for social services they tend to contract for inputs instead of outcomes. As a result, governments have difficulty knowing whether a particular social program is having the desired effect on its target population. For PFS projects, governments contract for outcomes, and the contracts generally include a requirement that the program’s impact be rigorously evaluated, which allows governments to demonstrate the effect the program has as well as to add to the evidence base to inform future programs. Stakeholders told us that PFS offers benefits to service providers, including the following: Stable funding for operations. Stakeholders told us that service providers must often pursue multiple grant and fundraising efforts to fund their services and enter into contracts in which they are not reimbursed for the full cost of services provided. PFS provides an opportunity for stable funding spanning multiple years. Officials from the service providers in New York and Massachusetts told us that upfront, multiyear funding for their programs is one of the key reasons why their organizations were excited about participating in a PFS project. An official from Roca in Massachusetts said that without the consistent funding provided under PFS, the organization typically has had to piece funding together from multiple grants to fund its programs. Stakeholders also told us that PFS can be preferable for service providers compared to traditional performance-based contracts. In a traditional performance-based contract, the service provider is not paid all or a Page 31 GAO-15-646 Pay For Success Potential Benefits for Investors portion of its fees until it meets defined outcomes. In such a model, the service provider assumes financial risk that could damage it if it does not succeed, therefore potentially excluding new or small service providers. For example, UK government officials told us that in a performance-based contract, the service provider must cover the initial costs of delivering services, which is difficult for many nonprofit service providers because they do not have the capital available to provide services in advance of being paid. A benefit of PFS is that investors offer service providers full upfront funding, or cost reimbursement, for service provision while assuming most or all of the financial risk. Flexibility in implementing service delivery. Service providers identified two ways PFS projects can give service providers flexibility compared to a traditional government contract or grant. In some of our cases, service providers were using PFS to administer their proven approaches on a larger scale. For example, officials from the service provider in New York told us that participation in the PFS project has allowed the organization to implement its service delivery model, for which the organization believes it has a solid evidence base, on a larger scale rather than conform to government-determined service delivery requirements. PFS can also offer flexibility to service providers in their day-to-day operations because the government contracts for outcomes, rather than how the service provider spends the funding on the activities it undertakes. Service providers said they have more flexibility to tailor their approaches to meet the needs of the target population. Stakeholders told us that PFS also offers benefits to investors, including the following: Potential for return on investment. Stakeholders told us that investors benefit from potential financial returns. Investors conduct due diligence on projects and invest in projects where they believe there is potential for return on their investment. Investors and intermediaries noted that the level of potential returns for PFS projects has so far been below-market, compared to the level of risk. Opportunity for social investment. Stakeholders said that certain investors are interested in making investments with positive social impacts. Officials from the bank that helped to facilitate investments in the New York project told us that the individuals who invested in the project are interested in aligning their investments with their values and creating positive financial and social impacts. Social investors, such as nonprofit philanthropic organizations in the United States and UK, said PFS is a Page 32 GAO-15-646 Pay For Success tool to support social programs consistent with their missions. For example, an official from a philanthropic organization that invested in the Peterborough Project told us that the organization had previously provided grants to support services for short-sentenced prisoners and decided to invest in the project, in part, because it offered a sustainable solution to reducing recidivism among such prisoners. The organization reasoned that if the project turned out to be successful and could be shown to deliver cost savings to government, the UK government would be incentivized to directly fund services for short-sentenced prisoners in the future. In addition, social investors told us that because they will be repaid if a service provider meets outcome measures in a PFS project, they will have the opportunity to recycle the funds into other projects that align with their missions. Participating Organizations Sought to Manage Risks in the Feasibility Assessment, Design, and Implementation Phases of Complex PFS Contracts Governments, Service Providers, and Investors Recognize that PFS Projects Present Risk PFS is predicated on the idea that governments shift the risk associated with developing and implementing a prevention program to investors. Stakeholders told us that the various parties to PFS contracts face a range of risks that are inherent to contracts: • Financial risk. The risk that one or more parties to the contract could lose money by participating in a PFS project. For example, a PFS project may not generate savings that are greater than its costs, which can include management fees for an intermediary, evaluation costs, and investors’ contracted outcome payments. Critics of PFS have argued that, given the extra costs associated with intermediaries and investors, using PFS to implement a program is inherently more expensive than providing the prevention program as a direct government service. Because interest rates are low for government Page 33 GAO-15-646 Pay For Success borrowing, it could still cost less to borrow funds to pay for the service.35 However, some investors and intermediaries said they believe the complexity and associated costs of implementing PFS projects may decrease over time if the projects become more replicable. PFS proponents in general have argued that such direct government funding of new prevention programs has been difficult to do in the current budget environment. At the same time, investors risk losing their investment if the service provider fails to achieve the desired outcomes. Officials from a bank that has invested in multiple U.S. projects told us that the organization carefully considers the evidence of the intervention that the service provider will deliver in order to determine whether it is willing to take the financial risk associated with an investment and how that risk should be priced. In addition, organizations participating in PFS projects told us that governments, service providers, and investors have dedicated a significant amount of time and resources to developing PFS projects, at times without being compensated for their work. There is a risk that the parties are unable to agree on a project’s complex details and fail to launch the project, in which case participants lose the time and resources that they devoted to developing the project. Stakeholders identified other risks that could potentially result in increased financial risk for parties to the PFS contract: • Perverse incentives. The risk that outcome payment structures will create perverse incentives for stakeholders to take actions at the expense of the individuals receiving treatment in the project. For example, creaming is the risk that service providers focus on individuals that are easiest to serve to achieve the defined outcomes. As a result of the perverse incentive to focus on those that are easiest to serve, those that are most in need and most difficult to serve could receive fewer services or a diminished focus. If creaming occurs, the government is potentially paying outcome payments when the target population is not being served. 35Elizabeth Lower-Basch, Center for Law and Social Policy (CLASP), Social Impact Bonds: Overview and Considerations (Washington, D.C.: Mar. 7, 2014). Page 34 GAO-15-646 Pay For Success • Termination risk. The risk that one or more parties to the contract decide to terminate their participation in a PFS project, requiring the project to end earlier than initially planned. The early termination of a PFS project presents a risk that the target population may be disadvantaged or hurt by losing access to the intervention. Officials from an investor in multiple U.S. projects said that the organization understands that the early termination of a PFS project has the potential to negatively affect the programs’ target populations. Therefore, the organization worked with other parties during contract negotiations to create rules that would allow for the gradual termination of unsuccessful projects while taking into account the needs of current participants. A variant of termination risk is policy risk, or the risk that a government initiates a policy change that prevents a PFS project from operating as initially intended. A policy change could disrupt a service provider’s program delivery, putting the achievement of outcomes— and the investors’ investment—at risk. For example, the UK Ministry of Justice terminated the Peterborough Project 2 years early. The UK government implemented a new initiative called Transforming Rehabilitation that provides a similar set of services for all short- sentenced prisoners in the UK. Officials from the UK Ministry of Justice said maintaining the PFS program at Peterborough until 2017 for the third and final cohort as planned was not possible, as the majority of prisoners within that group would already be receiving 12 months of supervision and rehabilitation as a result of the wider reforms to probation. According to officials from the intermediary, it would not have been possible to maintain a control group for the evaluation of the last cohort in the project. The Ministry of Justice and other project participants agreed to transition arrangements for the project that, among other things, established that investors could receive outcome payments if the project evaluation shows recidivism across the first two cohorts was reduced by 7.5 percent and that the Ministry of Justice would pay termination costs. Officials from the intermediary for the project said that investors are pleased that the needs of short-sentenced prisoners are being met under Transforming Rehabilitation. However, because the Peterborough Project ended early, information and learning about the effectiveness of the program was not fully realized. • Performance risk. The risk that an intervention fails to achieve defined outcomes. A government does not make payments if defined outcomes are not achieved. However, if outcomes are not achieved, then the government has lost time developing the program and failed Page 35 GAO-15-646 Pay For Success to meet mission-related goals and objectives. Performance risk includes how the prevention program is operationalized, managed, and implemented. This risk can be heightened if the intervention is poorly structured or poorly tailored to the needs of a particular participant population, or if service delivery is poorly executed, which could lead to potential harm to program recipients. Investors also risk losing some or all of their investment. Stakeholders from the government and an investor in New York City told us that they considered the implications of potential performance risks in the project during its conception and development. For example, an official from the New York City government told us that the parties involved had extensive discussions, including about program outcomes and timelines. In addition, officials from an organization that invested in the project told us that they had to decide how likely, based on existing evidence, the program was to achieve the established outcome. • Appropriation risk. From the investors’ perspective, the risk that a government fails to appropriate funds necessary to make outcome payments. If a government does not appropriate necessary funds, investors could lose some or all of their investment, making them less likely to invest in PFS projects in the future. Investors’ perception of appropriations risk affects their decision on whether to participate in a PFS project and on how they view the risk of a project. For example, officials from a bank that has invested in multiple U.S. projects identified appropriations risk as one of the primary risks it sees for investors considering PFS projects. The officials told us that since investors assume substantial performance risk in the projects, they must have an assurance that, in the end, the government is going to make the outcome payments to which it initially committed. Stakeholders in our UK case illustrations did not view appropriations risk as a concern in the United Kingdom PFS market. • Reputational risk. The risk that a party’s reputation will be affected negatively by events that occur in a PFS project. Stakeholders in New York City told us that they are assuming reputational risk by participating in the Rikers Island project. For example, a former official from the New York City government told us that the city took a large reputational risk by launching the first PFS project in the United States, and the city is therefore very invested in its success. In addition, an official from one of the service providers working to implement the New York City project told us that the organization has assumed reputational risk by working as the service provider for the project. If the service provider fails to achieve outcomes for the Page 36 GAO-15-646 Pay For Success During Project Feasibility Assessments, Governments Reported Seeking to Maximize Cost Savings While Managing Financial and Performance Risk project, it could make it challenging to get future contracts. Furthermore, an official from the organization that is evaluating the New York City project told us that since there are individuals and organizations within the criminal justice sector who are skeptical of PFS or believe that it is a flawed concept, the organization is potentially risking its reputation within the criminal justice sector by participating in the project. To assess project feasibility, governments and other stakeholders identified and evaluated potential projects in an effort to maximize cost savings and also to manage potential financial and performance risk. Stakeholders from the selected case studies stated that, in the first phase of PFS project development, they evaluated the feasibility of using PFS to address a particular social problem within their jurisdiction. We found that in practice PFS contracts were complex and governments relied extensively on outside technical assistance to assess the feasibility of their projects. Attention to cost-benefit analysis. Governments estimate the potential effect the PFS project will have on future budgets to determine if the project is feasible. Using cost-benefit analysis and cost modeling,36 governments evaluate whether the prevention program is likely to generate cost savings, cost avoidance, or other benefits over the course of the project that exceed the project’s costs, which include the total amount of outcome payments and associated administrative costs. Based on this analysis, governments determine whether a project is feasible. For example, as noted earlier, the Maryland Department of Legislative Services determined that the state should not pursue a PFS project 36According to DOL officials, a cost-modeling methodology is different than a cost-benefit analysis; while cost modeling accounts for various forms of benefits, including cost savings, it may not account for all the expenditures and costs that would be included in a typical cost-benefit analysis. Page 37 GAO-15-646 Pay For Success because even relatively large reductions in recidivism among program participants were unlikely to result in the closure of a wing of a prison facility, the type of large, fixed-cost savings that it believed would be needed to cover the program’s costs. Governments’ cost-benefit analyses may also include indirect costs and social benefits. For example, New York assessed both the direct cost savings of reduced prison expenses to the government as well as quantifiable public sector benefits of savings from reduced crime resulting from the project. The New York project’s outcome payments are derived from these estimated public sector savings and benefits. According to a detailed summary of the project, the state estimated and included an amount for the cost savings from reduced crime for victims based on a study that estimated the intangible victim costs. To estimate these costs, the state used other costs such as medical costs and lost earnings for crime victims. The state also used a jury-compensation approach that used the money awarded to victims by juries to estimate the intangible victim costs of crime.37 Figure 10 shows the potential savings and benefits, as well as the outcome payment costs that New York identified for its PFS project. For example, a 10 percent reduction in recidivism would result in potential savings and benefits that equal outcome payment costs. As the rate of reduction in recidivism increases, the state’s potential savings and benefits (after making outcome payments) increase. 37State of New York, Center for Employment Opportunities, and Social Finance, Investing in What Works: “Pay for Success” in New York State Increasing Employment and Improving Public Safety: A Detailed Project Summary (March 2014). Page 38 GAO-15-646 Pay For Success Figure 10: Estimated Savings and Benefits and Outcome Payments at Various Reductions in Recidivism Rates Identification of entities that capture financial benefits and budget savings. Stakeholders said that in order to maximize cost savings to the entity who is sponsoring the PFS project, governments must know which government budget will capture and benefit from the savings. Stakeholders said that a barrier to potential PFS projects occurs when expected savings accrue to a different government entity than the one that is leading the project. This is referred to as the “wrong pockets problem.” If the savings resulting from a successful intervention by one government entity accrue to a different government entity, without agreement from the other government entity to contribute to outcome payments in proportion to the savings, the project may not be cost- beneficial to implement. For example, stakeholders in South Carolina have been developing a PFS project to fund a program designed to improve maternal health outcomes. Since potential savings generated by Page 39 GAO-15-646 Pay For Success the program would accrue to Medicaid, those savings would be split between the state and the federal government.38 Stakeholders told us that the state would find it challenging to agree to make all of the project’s potential outcome payments since it would only be capturing approximately 30 percent of the project’s savings, with the federal government capturing the rest. A stakeholder in New York City said the city chose to pursue the Rikers Island project specifically because the city bears all of its own criminal justice costs and therefore would not have to involve the state or federal government in developing the project or paying for the outcomes achieved. Access to, and capacity for, robust data collection and analytics. To determine project feasibility, governments must have access to complete and accurate cost and benefit data and must have the capacity to analyze it.39 However, stakeholders said that the availability of government data on social programs and the capacity to analyze the data is a challenge to the growth of PFS. Stakeholders said that governments need data systems in place that allow agencies to collect data and share it between agencies, as needed. However, in some cases, the necessary data can be difficult to obtain or share. For example, government stakeholders in Massachusetts told us that differences in the ways in which jurisdictions within the state maintained files on individuals on probation made it difficult to collect and share data on potential program participants at the beginning of the project’s implementation. Stakeholders said legal barriers can prevent governments from sharing data across multiple systems. For example, we previously reported that matching education and workforce data is challenging in states where collecting a Social 38States and the federal government share in the financing of the Medicaid program, with the federal government matching most state expenditures for Medicaid services on the basis of a statutory formula known as the Federal Medical Assistance Percentage (FMAP). The FMAP is calculated using a statutory formula based on the state’s per capita income (PCI) in relation to the national PCI: FMAP = 1.00 – 0.45 (State PCI / U.S. PCI) and may range from 50 to 83 percent. The federal government pays a larger portion of Medicaid expenditures in states with low PCI relative to the national average, and a smaller portion for states with higher PCIs. For more information see GAO, Medicaid: Assessment of Variation among States in Per-Enrollee Spending, GAO-14-456 (Washington, D.C.: July 16, 2014). 39Challenges with government capacity apply to multiple phases of PFS projects. Page 40 GAO-15-646 Pay For Success Security number in education data is prohibited by state law or agency policy.40 Stakeholders also expressed concerns that many governments do not have sufficient data collection and analysis capacity for PFS projects. Similarly, we have previously reported that it is critical to define the relevant types of information required for service acquisitions and then develop the appropriate data systems to collect and provide reliable data.41 Governments will need staff with expertise in data analysis and financial modeling to be able to conduct feasibility studies, identify future savings, evaluate service delivery, determine outcome payments, and negotiate PFS contracts. Government officials from both New York and New York City said that their government’s robust data systems facilitated their cost-benefit analyses. For example, according to a lessons learned report from the intermediary in the New York City Rikers Island project, the city’s strong financial planning structure allowed the city to gain a solid understanding of the amount of money that it could save based on changes in the size of its jail population. The city used this information to establish its willingness to pay for reductions in recidivism attributable to the intervention being funded by the PFS project. The cost information, together with historical criminal justice data, allowed the city to identify a specific reduction in recidivism that would make the PFS project beneficial for the city.42 To address challenges with government capacity, all of the U.S. PFS projects in our study included provisions for technical assistance from experts affiliated with universities or intermediary organizations to analyze project data and structure financial models needed to determine project feasibility and outcome payments. Sufficient evidence of an intervention’s success. In addition to cost- benefit analysis and access to data, stakeholders told us that during the feasibility phase it is important to consider the evidence of an intervention’s past success to help manage the performance risk of the project. While some stakeholders stressed the importance of sufficient 40GAO, Education and Workforce Data: Challenges in Matching Student and Worker Information Raise Concerns about Longitudinal Data Systems, GAO-15-27 (Washington, D.C.: Nov. 19, 2014). 41GAO-07-20. 42MDRC, Financing Promising Evidence-Based Programs: Early Lessons from the New York City Social Impact Bond (New York, NY: December 2013). Page 41 GAO-15-646 Pay For Success evidence of an intervention’s success, others noted that there is variation in the PFS field in the level of evidence and how it relates to the individual project. For example, evidence supporting an intervention applied to one population could provide the basis for testing the intervention with a different population. A PFS project could also be used to implement an innovative intervention with limited but promising evidence. Participating organizations—including governments, investors, and service providers— may differ in their tolerance of risk and uncertainty associated with the evidence of any intervention. Stakeholders from government, intermediaries, and investors said they also assessed the availability of service providers who could implement the intervention to determine whether the project would be feasible. Officials from an investor in multiple U.S. projects told us that the bank has looked to invest in PFS projects that feature a service provider that has a track record of success. Stakeholders said the availability of evidence-based social programs and service providers with the capacity to implement them is a challenge to growing the PFS field. Stakeholders expressed concerns that there is not a sufficient pipeline of nonprofit service providers whose interventions have been rigorously evaluated. We have previously reported on challenges nonprofits face related to funding indirect costs, such as program evaluation.43 In addition, stakeholders told us that in order to be successful in an established PFS project, service providers need experience implementing the evidence-based intervention, the capacity to implement the intervention on a larger scale, and the ability to collect and report data for evaluation. The enacting legislation for the Utah PFS Project includes a grant program to build the capacity of preschool providers to implement the high-quality preschool model, which is based on a prior evaluation, so these providers are able to participate in potential future PFS projects. Leadership support. Stakeholders said that support from top government leaders is essential to moving a PFS project forward to the design and implementation phases of the project. This support can provide the momentum that is necessary to get a PFS project off the ground and sustain the project through implementation. State officials in New York told us that PFS was part of Governor Andrew Cuomo’s policy 43GAO, Nonprofit Sector: Treatment and Reimbursement of Indirect Costs Vary among Grants, and Depend Significantly on Federal, State, and Local Government Practices, GAO-10-477 (Washington, D.C.: May 18, 2010). Page 42 GAO-15-646 Pay For Success During Contract Design, Parties Reported Defining Requirements to Manage Risk agenda during his gubernatorial campaign, as it aligned with his interest in taxpayer efficiency and a focus on outcomes. In addition, a report on the project stated that it was conceived when Governor Cuomo outlined a state priority to use PFS to reduce recidivism and increase employment among high-risk ex-prisoners.44 Furthermore, stakeholders in Utah told us that the long-term prospects of the project were boosted by the Board of the United Way of Salt Lake and the Mayor of Salt Lake County’s agreement, during the project’s proof of concept year, to dedicate funds to support the project’s outcome payments. Stakeholders also said that leadership from a central government agency can help maximize collaboration across government agencies. They said that it is important that a central agency coordinates across multiple departments in a situation where an intervention in one department will accrue future savings in a different department or where participation from multiple departments is needed to implement the project. We have previously reported that top leadership commitment can improve outcomes and set the direction, pace, and tone for implementing a significant policy change.45 Stakeholders also said it is critical to have a dedicated government official to manage the development of a PFS project. For example, the governments in Massachusetts, New York, and Utah had a full-time position in their budget offices to manage their PFS efforts. Stakeholders said contract negotiations for PFS projects are complex, lengthy, and resource intensive. Stakeholders from government, service providers, intermediaries, and investors told us that they required significant legal and technical capacity to support the development of these complex contracts because PFS contracts were new and existing templates were not available to use as a guide.46 While the same types of parties were present, the PFS projects in our study had different contractual arrangements. In some cases, the PFS contract was between 44State of New York, Center for Employment Opportunities, and Social Finance, Investing in What Works: “Pay for Success” in New York State: Increasing Employment and Improving Public Safety (March 2014). 45GAO, Highlights from a GAO Forum: Mergers and Transformation: Lessons Learned for a Department of Homeland Security and Other Federal Agencies, GAO-03-293SP (Washington, D.C.: Nov. 14, 2002). 46To help share knowledge on PFS, Massachusetts, New York, and Utah made their contracts publicly available after they were completed. Page 43 GAO-15-646 Pay For Success Defining Requirements in Outcome Payment Structures the government and the intermediary, which held additional contracts with investors and service providers. In other cases, the contract was between the government, intermediary, and service provider. The intermediary holds separate agreements with the investors in the project. While the contractual arrangements varied, stakeholders said that all parties were involved with negotiating the terms of the contracts. Parties to the PFS contracts in our study defined the requirements for the service provider, in part, by using performance targets for specific outcomes that were rewarded with outcome payments. These performance targets and associated payouts make up an outcome payment structure. See table 2 for examples of four different types of outcome payment structures we observed in the PFS projects we reviewed. For example, in the Utah High Quality Preschool project, the outcome payment structure was defined as a set payment for each student participant who avoided being enrolled in special education each year from kindergarten through sixth grade. Stakeholders from governments and service providers said that the outcome payment structure aligned the incentives of all parties to be invested in the service provider’s successful delivery of the prevention program. Further, as suggested in practices for complex service acquisitions, because the parties to the contract agree in advance to outcome measures and associated payments, there is less room for interpretation once the contract is being implemented.47 47GAO-07-20 and Brown, Potoski, and Van Slyke, Complex Contracting. Page 44 GAO-15-646 Pay For Success Table 2: Examples of Outcome Payment Structures from Selected Pay for Success Projects Performance measure    Target All or nothing: New York    Reduction in bed days on re-   8.5% reduction Outcome payment Below 8.5%: no payments 8.5% to more than 20%: $4.8 - $11.712 million $2,589.70, per student, per year £8,000 ($13,000) per child £23,000 ($37,000) per child £6,800 ($11,000) per child £15,800 ($26,000) per child Below 8%: no payment Above 8%: $85 per dayb There are 2 payment structures depending on the average number of hours worked: • If average hours worked is greater than or equal to 111 hours: $3,120, per person engaged in a transitional jobc • If average hours worked is less than 111 hours: $20 per hour engaged in a transitional jobd $6,000 per persone City Rikers Island Project Fee for outcome: Utah    Whether or not a student from   Student not admission compared to a historical group High Quality Preschool Project the testing cohort is enrolled in special education services after preschool each year kindergarten through 6th grade enrolled in special education services Fee for outcome: UK It’s 1. All About Me (IAAM) Register child with IAAM    Child registered adoption service provider Adoption Project 2. Placement with adoptive    Child placed family 3. Placement with adoptive    Child placed for 1 family for 1 year 4. Placement with adoptive    Child placed for 2 year Mix of all or nothing and 1. fee for outcome: New York Increasing Employment and Improving Public Safetya 2. family for 2 years Reduction in number of bed days spent in jail or prison compared to a control group years 8% reduction in bed days Start transitional job    Payment is conditional on achieving 8% reduction in bed days 3. Employment a year after    5% increase in release from prison or jail compared to a control group employment Source: GAO analysis of project documents. | GAO-15-646 aThe New York project will be implemented in two phases. Each phase will serve 1,000 individuals, measure the impact of the intervention on those individuals, and make performance-based payments based on that impact. Phase 1 and 2 outcome payments are different to account for inflation. The outcome payments presented in the table are for phase 1. bPhase 2 payout is $90.10 per day. cPhase 2 payout is $3,307 per person engaged in a transitional job. dPhase 2 payout is $21.20 per hour engaged in a transitional job. ePhase 2 payout is $6,360 per person. Page 45 GAO-15-646 Pay For Success Challenge of Securing Investors We have previously reported that a balanced set of performance goals and measures can help organizations address varied aspects of program performance.48 In some of the PFS cases in our study, the parties included a range of performance measures in the outcome payment structure to more effectively serve the target population. For example, the New York project included performance measures on both reducing recidivism and increasing employment for individuals who were recently released from prison. According to a detailed summary of the project, formerly incarcerated individuals face numerous challenges, including barriers to finding a job upon release.49 Stable employment can be an important factor in success upon release from prison. The outcome payment schedules also create financial incentives for investors and service providers to improve the outcomes for the target population. Stakeholders told us that multiple outcome measures can be a tool to incentivize service providers to address various participant needs. For example, the London Homelessness Project includes outcome measures to target the different causes of homelessness of individuals in the cohort. One of the five measures is related to supporting individuals’ progress toward employment, and another outcome measure is related to moving individuals into stable housing. Outcome payment structures with higher payouts at the end of the project create incentives for investors and service providers not to terminate the contract early because they will receive the majority of their return on investment at the end of the contract period. Stakeholders said that the rates of return offered by PFS projects likely will not meet market returns available on traditional investments with similar risk profiles, therefore reducing investor demand. Lower returns compared to traditional investments could make it more difficult for governments to attract sufficient capital for PFS projects. Investors assess the risks of a project to determine whether or not they will invest. Investors’ valuation of risk is primarily based on several factors related to managing their financial risk, including evidence of the intervention and 48GAO, Agency Performance Plans: Examples of Practices that Can Improve Usefulness to Decisionmakers, GGD/AIMD-99-69 (Washington, D.C.: Feb. 26, 1999). 49State of New York, Center for Employment Opportunities, and Social Finance, Investing in What Works: “Pay for Success” in New York State Increasing Employment and Improving Public Safety: A Detailed Project Summary (March 2014). Page 46 GAO-15-646 Pay For Success the ability of the service provider to implement it, the outcome payment structure, and the level of appropriations risk. For example, commercial investors prefer interventions backed by rigorous evidence and service providers who have a history of strong performance because it lowers their financial risk. Governments also manage their financial risk by determining the outcome payments and rate of return they are willing to pay to investors. In Utah, the legislation that authorizes the High Quality Preschool Project included a cap on the return for investors. A Utah legislator said the cap helps to ensure that the state captures more of the cost avoidance from reduced use of special education services. The legislature wanted to ensure they were being good government stewards because they would not be at the table negotiating PFS transactions. In other projects, the return on investment was negotiated on a case-by- case basis. For example, in the New York project, there is a cap on the total outcome payments the state can make over the life of the project. In the UK, government officials told us that some PFS project outcome payments and associated rates of return funded by the Department for Work and Pensions’ (DWP) Innovation Fund were not subject to negotiation. Instead, DWP developed a rate card, where it defined the maximum it is willing to pay for specific outcomes (see appendix IV for an example of a rate card). In the U.S. projects in our study, parties to the contracts used financing structures to address challenges in attracting investors, such as a tiered capital structure or a loan guarantee to secure the upfront capital needed for the project. Tiered capital structure. A tiered capital structure, where some investors take on more risk, can attract bigger pools of capital because it lowers risk for some investors. In three of our U.S. case illustrations a combination of commercial and philanthropic investors made up the capital structure. Stakeholders said that investors’ willingness to accept risk can vary and that philanthropic investors may be willing to accept more risk at a lower rate of return than commercial investors. For example, according to the intermediary in the Massachusetts Juvenile Justice project, to encourage investment the project capital structure was split between philanthropic and commercial investors. Commercial funding partners contributed senior debt, which will be paid out first; foundations contributed junior debt, which will be paid out following the senior debt. Philanthropic grants from foundations also served as first- loss capital, which will bear the first loss in the event that the service provider does not meet the outcome measures. According to the Page 47 GAO-15-646 Pay For Success During Project Design, Parties Reported Using PFS Contracts to Establish Governance Structures and Rules for Project Implementation to Manage Risk intermediary in the project, in the first-loss role philanthropic donors can combine and leverage their grant making with commercial capital. Loan guarantee. In a different model, a philanthropic investor can provide a loan guarantee to the primary investor to reduce its risk by assuming the loan if the service provider does not meet the outcome targets. For example, in the New York City Rikers Island Project, the city agreed to a maximum return of $2.1 million which was not high enough for Goldman Sachs to justify the level of risk it perceived regarding repayment of its $9.6 million loan. In an effort to balance the risk/reward trade-off, Bloomberg Philanthropies provided a $7.2 million grant to MDRC to act as a loan guarantee that would reduce Goldman Sachs’ capital at risk in the event that outcome targets were not achieved. An investor in the project further explained that although the service provider had a track record of success, its risk of failure in the NYC Rikers Island project was higher because it would be implementing a new curriculum in a difficult environment. The combination of the performance risk and its status as the first PFS project in the United States were the primary reasons for the loan guarantee. Governance rules that define interactions and the authority to make decisions after the contract has been signed can encourage cooperation and manage the risk of unexpected termination by any party. For the projects in our study, the contracts also defined how parties will interact during the implementation of the project. PFS contracts in our study included outcome payment structures, oversight structures, governance rules, and project evaluation mechanisms to manage risk. Oversight structures. PFS contracts included oversight structures to manage performance risk during contract implementation. All of the projects in our study had a governance structure, which defined how parties would make decisions during implementation of the contract, including how the project would be managed and overseen. For example, the New York project had a three-tier governance structure made up of the following: • An executive steering committee: This committee meets semi- annually to provide strategic direction, monitor compliance with the agreement, and review reports. Members include the Deputy Secretary for Civil Rights; the Deputy Secretary for Public Safety; the Commissioner of Department of Labor (DOL); the Commissioner of the Department of Corrections and Community Supervision (DOCCS); Page 48 GAO-15-646 Pay For Success CEO of the intermediary; and the Executive Director of the service provider. • A management committee: The committee meets monthly to monitor reports and program implementation and provides updates to the executive steering committee. Members include the Associate Commissioner of DOCCS; two Deputy Commissioners of DOCCS; the Deputy Commissioner of DOL; the Executive Director of the service provider’s regional office; the Director of DOCCS Research; a representative from the Harvard Kennedy School Social Impact Bond Technical Assistance Lab;50 and a director from the intermediary. • Working groups: Working groups meet to discuss operational and research issues that arise over the course of the project. They are composed of staff and subject-matter experts from DOCCS; DOCCS Research; DOL; the service provider; the Harvard Kennedy School Social Impact Bond Technical Assistance Lab; and the intermediary. Contract governance rules. Complex contracting practices suggest that including provisions for how decisions will be made when changes are needed during service delivery can help address implementation challenges. To manage various risks and inherent uncertainties, PFS contracts included governance rules to define how decisions would be made over the course of the project. PFS contracts included rules to define decision making after the contract was in place as well as provisions to address how parties to the contract could be replaced in the case of underperformance. For example, the New York contract included governance rules that defined when and how the service provider could be replaced. The contract specified that any replacement service provider is subject to approval from the New York Department of Labor and Department of Corrections and Community Supervision, among others. PFS contracts also defined the circumstances under which investors could decide to terminate the project based on the service provider’s performance. A Massachusetts official said that determining the provisions that would allow investors to terminate the contract was one of the most difficult parts of the negotiations. Because the PFS project was designed for the service provider’s model, Massachusetts wanted to 50The Harvard Kennedy School Social Impact Bond Technical Assistance Lab provides pro bono technical assistance to state and local governments, including New York, that are implementing PFS contracts. Page 49 GAO-15-646 Pay For Success support the model, which requires a number of years of participation to achieve outcomes. Massachusetts also wanted to ensure that program participants would not be left without services as a result of the program’s early termination. Investors did not want to continue their investment if the service provider was underperforming. In the contract, the parties agreed that the lead investor has a right to exit the project at the end of the second and third years if the service provider’s participant attrition rate exceeded defined thresholds. The contract also included provisions, such as the development of a remediation plan, designed to improve the service provider’s performance if it fell below a defined threshold. These provisions gave investors the opportunity to reduce their financial risk if the service provider was underperforming. Allowing for 2 years of implementation and outcome measurement before investors can exit the contract also helped mitigate the government’s performance risk because it allowed enough time for service providers to collect data to measure outcomes and make adjustments if necessary to improve performance. Termination provisions also defined what would happen if governments terminated the contract early. For example, in the NYC Rikers Island Project, the City of New York could terminate without cause. However, if the city terminated the contract before the entire evaluation cohort receives services, it would lose the pay-for-success features of the project and must reimburse the intermediary for program costs to date. Stakeholders said that building flexibility into contracts to allow for negotiations to make operational changes without amending the contract could help address implementation challenges. Some changes to the projects required updates to the PFS contract, which were costly and time intensive because all parties to the contract had to negotiate and agree to the changes. Stakeholders said that it is important to find a balance between specifically defining requirements and building sufficient flexibility into the contract to address challenges during program implementation. A Massachusetts official said the contract must be detailed enough to ensure that the program participants are protected and that risk is properly transferred from the government to other parties. However, the contract’s terms must also offer the parties a degree of flexibility to make changes as problems arise, because it is very time consuming and difficult to amend the contract. Government budgeting structure. Because appropriations risk was a key concern of investors, the U.S. cases in our study managed this risk by establishing different mechanisms to secure funds for outcome payments: Page 50 GAO-15-646 Pay For Success • Massachusetts enacted legislation authorizing the backing of PFS contract payments with the full faith and credit of the commonwealth and establishing a Social Innovation Financing Trust Fund into which annual appropriations may be made. • New York City, if the program had been successful, would have made outcome payments as it would for any other contractual obligation. There were two evaluation points that would trigger a possible payment. The final payment was scheduled to occur 1 year after the final results, giving the city more time to realize savings. • New York pledged to obtain appropriations to make outcome payments. The contract includes provisions for an orderly wind down if the state does not renew the appropriation. • Utah enacted legislation that authorizes a School Readiness Restricted Account which consists of funds appropriated by the legislature, federal grant funds, and private donations. Funds in this account may be used for contracts with independent evaluators, future outcome payments, and grant awards. Page 51 GAO-15-646 Pay For Success Random Control Trial  NewYorkIncreasingEmploymentand Improving Public Safety  MassachusettsJuvenileJustice Quasi-Experimental Design  NewYorkCityRikersIsland  PeterboroughPrison Process to Verify Outcomes  UtahHighQualityPreschool  LondonHomelessness  UKIt’sAllAboutMeAdoption Source: GAOsummaryofprojectdocuments Evaluation Methods in Case Illustrations Project evaluation mechanisms. The PFS case studies we examined used evaluations to manage the governments’ various risks and to ensure the defined outcomes are achieved. In each case, outcome payments were contingent upon the evaluator certifying that the agreed-upon outcomes were met. Four of the evaluations in our case illustrations were impact evaluations, such as a random control trial51 (RCT) or a quasi- experimental design,52 where outcomes for individuals receiving the intervention were compared to those from a similar control group. Three of the evaluations used a process to verify that outcomes were being achieved for the target population. For example, in the Utah project’s evaluation, the evaluator tests each student participating in the high- quality preschool program at the beginning and end of the school year. The students who score two standard deviations below the mean on the pre-test are tracked from kindergarten through sixth grade to verify whether they are enrolled in special education. The evaluation was derived from an earlier analysis that showed a decrease in special education use among students attending the high-quality preschool program. In the London Homelessness project, service providers submit evidence for some of the measures to the Greater London Authority, which is responsible for verifying the attainment of those measures. For example, for the outcome measure on helping clients into stable housing, providers submit a form from an individual’s landlord, which the Greater London Authority verifies and determines whether to make the associated outcome payment. Stakeholders said that evaluations can manage potential perverse incentives, such as creaming. Some projects in our study included a mechanism to address creaming in the evaluation. For example, the Utah project on early childhood education addressed the risk of creaming by only including the children who score two standard deviations below the mean on the pre-test in the evaluation for outcome payments. 51Random control trials compare the outcomes for groups that were randomly assigned either to the treatment or to a nonparticipating control group before the intervention, in an effort to control for any systematic difference between the groups that could account for a difference in their outcomes. 52Quasi-experimental designs resemble randomized experiments in comparing the outcomes for treatment and control groups, except that individuals are not assigned to those groups randomly. Instead, unserved members of the targeted population are selected to serve as a control group that resembles the treatment group as much as possible on variables related to the desired outcome. Page 52 GAO-15-646 Pay For Success Stakeholders said that evaluations can also help manage perverse incentives for stakeholders to take actions at the expense of the individuals receiving treatment in the project. For example, because outcome payments in the Utah project are made based on students avoiding special education, there is potentially an incentive to withhold special education services to increase outcome payments. The Utah project manages this risk by keeping the identity of the students in the evaluation cohort confidential, so the teachers who determine special education needs are not aware of a student’s status in the project. In another example, a report on the London Homelessness Project stated that well-defined target populations can reduce the risk of creaming because the service provider cannot manipulate the population.53 This project targets 830 entrenched homeless individuals who had been seen sleeping rough54 or who have stayed in a London rough sleeping hostel in the last 3 months, and who had been rough sleeping at least six times over the last 2 years. Stakeholders said that there are challenges to implementing RCT evaluations for PFS projects. We have previously reported that well- conducted randomized experiments are best suited for assessing effectiveness when multiple causal influences create uncertainty about what caused results.55 Such experiments are often expensive because they are among the most rigorous forms of evaluation, as well as difficult or sometimes impossible to carry out. For example, a risk of RCTs is that the evaluation may be inconclusive if the control group cannot be maintained and because a large sample size is required to yield statistically significant results. One stakeholder said that a RCT can increase the reputational risk to service providers if it is poorly designed and misses the real outcomes of the program. To manage the risk of these challenges, stakeholders from several projects identified the need for an alternative methodology in case the evaluation could not be implemented due to unanticipated implementation challenges. For example, in the NYC Rikers Island Project, the initial evaluation plan was 53United Kingdom Department for Communities and Local Government, Qualitative Evaluation of the London Homelessness Social Impact Bond: First Interim Report (London, United Kingdom: September 2014). 54Rough sleeping is a term used in the UK to refer to individuals who sleep outside due to homelessness 55GAO-10-30. Page 53 GAO-15-646 Pay For Success Ongoing PFS Contracts Show Benefits of Data- Driven Performance Management, but Final Outcomes Are Not Yet Known based on a random control trial, but in the jail environment it was not possible to keep the control and treatment groups separate. Therefore, project stakeholders designed an alternative methodology, a quasi- experimental design, which provided a sufficient degree of confidence that the outcomes achieved could be attributed to the program.56 Stakeholders also addressed potential ethical concerns associated with RCTs, such as withholding the service intervention from a population that could potentially benefit from the services in order to maintain a control group for an impact evaluation. While there could be ethical concerns associated with using a control group for a random control trial evaluation of a service, limited availability of funding often prevents treating the entire population. Therefore, stakeholders indicated they could possibly draw from patients on a wait list for the program to serve as a control group. The Massachusetts project addressed ethical concerns with RCTs by including a “no denial of service policy” if an individual in the nonparticipating control group asked to participate in the program. This design required a significant increase in the evaluation’s sample size to address potential contamination between the treatment group and the control group in the sample. During the third phase of a PFS project, service providers implement the prevention program. To manage performance risk, participating organizations provide oversight, monitor performance, and evaluate the extent to which outcomes are achieved, based on the structures and governance roles established in PFS contracts. 56In the quasi-experimental evaluation for the NYC Rikers Island project, the evaluator compared the outcomes for a cohort of adolescents incarcerated at Rikers during calendar year 2013 to a historical group that did not receive the program in order to determine the impact of the intervention. Page 54 GAO-15-646 Pay For Success Implement Prevention Program Using Data-Driven Performance Management Actively managing performance to achieve outcomes during contract implementation is vital to achieving results. While the structures and membership varied, stakeholders said PFS oversight bodies regularly reviewed performance data during service delivery. According to stakeholders, intermediaries and investors can bring performance management expertise to service providers and provide a rigorous focus on performance management and accountability. Officials from the intermediary for the New York City project said they continued to monitor the implementation of the service delivery using metrics that predict whether the project will be successful. They said that these metrics and targets allowed them to troubleshoot project implementation and intervene early to modify elements of the program that were not working. For example, there is a community service element to the Moral Reconation Therapy program that was modified to work in the Rikers Island jail environment. In another example, officials from the intermediary for the Peterborough Project said they used data analytics and provided program management support to help improve service delivery. In 2014 the UK Department of Work and Pensions evaluated its Innovation Fund PFS projects and found that for its first 10 PFS projects the strong focus on monitoring and managing project performance built capacity among service providers and helped to increase performance levels and improve efficiency of service delivery.57 Social service providers often collect and report input and output data in response to multiple grant requirements, but these data may not always be useful for monitoring the performance of a PFS project in achieving outcomes. Service providers said the PFS model may require new methods to collect and use performance information that shows the program is being implemented as planned and with fidelity to the model that was shown to be successful. Some service providers may also need to invest in building their capacity to collect and report outcome data to participate in PFS projects. An official from one service provider noted that her organization invested in data entry and data analyst positions and has a team that collects, analyzes, and processes data that it submits to the intermediary. 57Andrew Thomas, Rita Griffiths, and Alison Pemberton, The Innovation Fund pilots qualitative evaluation: Early Implementation Findings, United Kingdom Department of Public Works (London, UK: 2014). Page 55 GAO-15-646 Pay For Success Evaluate Outcomes and Next Steps Projects in our study faced unanticipated challenges in delivering services and stakeholders said oversight mechanisms helped to identify issues quickly and facilitated decision making to address problems. For example, stakeholders in our study reported difficulty in identifying cases for referrals to the service provider. Maintaining the planned level of referrals was important because the evaluation and outcome payments were dependent on the level of participation in the project. An official from the service provider in the New York project said that approximately 3 to 4 months into program implementation, there was a dip in the number of potential participants referred to the program. He said the cause was that the service provider was seeking referrals from parole bureaus where it had traditionally drawn few participants in order to maintain the project’s randomized referral and evaluation plan. However, he said this strategy depressed the overall number of referrals. As a result, they decided to expand the number of parole bureaus that could serve as a source of referrals. Stakeholders said that pilot periods for service delivery before the evaluation starts could reduce unanticipated challenges during PFS implementation and reduce risks that service providers will not meet outcome targets. According to a report on the New York City project,58 it may be beneficial to pilot-test a program at full scale and only evaluate it after it achieves a steady state of operation. On the other hand, pilot programs require time and money, both of which are likely to be constrained for PFS projects. The New York City project included a pilot test conducted at half scale. Stakeholders said that although the experience was valuable, certain issues came to light only after the program expanded to full scale. A pilot project can also be used to build evidence on an intervention to develop a future PFS project. For example, the Asthma Impact Model for Fresno is implementing a pilot project from spring 2013 through early 2016. The pilot project includes a random control trial evaluation on the intervention to demonstrate the outcomes of a home-based asthma management program. If successful, the results of the pilot will be used to develop a PFS strategy. Following program implementation, the parties to a PFS project must evaluate whether the service provider achieved outcome measures and 58MDRC, Financing Promising Evidence-Based Programs: Early Lessons from the New York City Social Impact Bond (New York, NY: 2013). Page 56 GAO-15-646 Pay For Success determine whether the government must make outcome payments based on the results. PFS contracts we examined spanned 3 to 10 years and one project—New York City—had been completed as of the time of our review. Results for the New York City project were reported in July 2015 and indicated that the program did not reduce recidivism among adolescents at Rikers Island and therefore did not meet the predetermined threshold of success: a 10 percent reduction in recidivism. As a result, New York City did not pay for the program and the intervention was discontinued as of August 31, 2015. Results of the evaluation of the initial cohort of the Peterborough Project were reported in August 2014 and indicated that reoffending rates for participants were reduced by 8.39 percent compared to a control group. A reduction of at least 10 percent within the cohort was necessary to trigger an outcome payment to investors, so no payment was made. However, the project’s terms state that outcome payments will be made if reoffending rates are reduced for each individual cohort by 10 percent or across all cohorts by 7.5 percent. Therefore, if the evaluation to be conducted following the second cohort finds that reoffending rates across both cohorts were reduced by at least 7.5 percent, the Ministry of Justice will make outcome payments to investors. We asked participating organizations what will happen to their programs at the end of their PFS contracts. Some told us that governments may consider funding the service delivery directly. In another case, the It’s All About Me adoption project has been designed so that if local governments use the service in large numbers over 10 years, the project will amass sufficient reserves after repaying investors to self-finance the service in the future. While PFS financial models were designed based on future government savings, some government stakeholders indicated they were not planning to reduce future agency budgets if PFS outcomes are achieved, but will reallocate the savings to address other unmet needs in the same agency budget. Page 57 GAO-15-646 Pay For Success The Federal Government Could Play a Role in Funding, Building Capacity for, and Managing Investor Risk in PFS, but a Formal Mechanism to Collaborate and Share Lessons Learned Does Not Exist The Federal Government Could Play Several Roles to Address PFS Challenges at the State and Local Level While the effectiveness of the PFS model remains to be seen, stakeholders cited three potential roles the federal government could play to help address challenges at the state and local levels of government and further develop the PFS field: providing outcome payments, building capacity, and providing loan guarantees. Providing outcome payments. State and federal officials and other stakeholders said the federal government could provide outcome payments in PFS projects to address the wrong pockets problem, which could arise when an intervention results in savings to the federal government as well as the government commissioning the project. This issue makes it challenging for state and local governments to have sufficient funds available to make outcome payments. In our case studies, some officials said that a key consideration when assessing project feasibility was whether cost savings to their jurisdiction were sufficient to justify the cost of implementing the project when other jurisdictions, such as the federal government, shared in the costs and savings as well. To date, the federal government has not become a formal party to any PFS contract in which the federal government would share in the project’s savings. However, the federal government has provided grants to support outcome payments for two of the case studies we examined. To assist Page 58 GAO-15-646 Pay For Success New York and Massachusetts with their recidivism PFS projects, DOL awarded nearly $24 million in grant funding to support outcome payments for these projects.59 A stakeholder said that the DOL grant catalyzed the New York project, allowing them to fund outcome payments for the first cohort of 1,000 participants. Having support for outcome payments helped Massachusetts officials increase the number of participants in the project, thus allowing for a stronger evaluation of the program’s outcomes. The UK government also took steps to mitigate challenges related to the wrong pockets issue. For example, the £20 million (approximately $31 million) Social Outcomes Fund, managed by the Cabinet Office, was created to address challenges related to public sector spending silos that occur between agencies in central and local governments trying to address complex social issues. The Social Outcomes Fund provides outcome payments for government departments, local authorities, and other organizations to implement innovative projects to address social problems where savings could be spread among several government agencies. This approach addresses the difficulty faced by a single office attempting to justify the cost of implementing the program. The federal government could also commission PFS projects and provide outcome payments for programs where it would capture all of the savings, such as programs targeting Social Security Disability Insurance. Similar to state and local governments, as an outcome payor the federal government could face many of the same potential benefits, risks, and decisions regarding project feasibility identified by stakeholders in our study. Playing the role of outcome payor in PFS projects could provide benefits from potential future savings to the federal government if outcomes are achieved. For example, a stakeholder said a potential area where the federal government could play a role as outcome payor is in the provision of nonmedical asthma management services including in- home education, environmental assessment, and remediation such as 59DOL officials noted that the projects funded under DOL’s PFS solicitation were required to contract with an independent validator to assess whether the project met its outcome targets. As part of the validation methodology, grantees will credibly demonstrate that outcome targets are achieved due to the intervention by defining the outcome target relative to a well-defined comparison population or control group. DOL has also engaged an independent evaluator (not the independent validator associated with the partnership) to assess the implementation, outcomes, and benefits of projects funded under the PFS solicitation. Page 59 GAO-15-646 Pay For Success mold removal, which could potentially reduce the number of asthma attacks and emergency visits by Medicaid recipients. Reducing emergency room visits paid by the Medicaid program could yield savings to both the state and federal governments. However, as an outcome payor, federal agencies would need to manage financial and performance risks similar to state and local governments. Federal agencies considering entering a PFS contract as an outcome payor would need to assess the feasibility of individual projects, including the factors stakeholders in our study identified, such as evidence on the intervention, cost-benefit analysis of the project, and the commitment of government leadership. Another potential challenge for federal agencies to consider is whether their existing grant program authority can accommodate outcome payments. PFS projects can span 3 to 10 years, while grant program authorities may limit the duration of a grant project. For example, DOJ officials told us that the reentry grant program authorities under the Second Chance Act limited the grant performance period to 12 months. This restriction would have prohibited the use of a PFS project as no outcomes could be achieved in such a short project period. However, this time limit was later eliminated for grants funded by the appropriation made available for DOJ’s PFS efforts for fiscal years 2014 and 2015.60 Another issue for agencies to consider is the period of availability of the funds and what happens to funds obligated for outcome payments if they are not made because the service provider does not meet the outcome targets. DOL officials told us that the grant funds obligated for outcome payments will be deobligated and unavailable for new obligations if outcome targets are not met in the New York or Massachusetts PFS projects. In another example, CNCS’s fiscal years 2014 and 2015 appropriations include authority that allows CNCS to reobligate funds to other authorized projects if outcome targets are not met.61 Building capacity for PFS projects. According to stakeholders, the federal government could make grants for feasibility studies and project 60Consolidated Appropriations Act, 2014, Pub. L. No. 113-76, div. B, title II, 128 Stat. 5, 63 (Jan. 17, 2014) and Consolidated and Further Continuing Appropriations Act, 2015, Pub. L. No. 113-235, div. B, title II, 128 Stat. 2130, 2193-94 (Dec. 16, 2014). 61Pub. L. No. 113-76, div. H, title IV, 128 Stat. 5, 401 and Pub. L. No. 113-235, div. G, title IV, 128 Stat. 2130, 2506-07. Page 60 GAO-15-646 Pay For Success development to build the capacity of state and local governments, intermediaries, and service providers, and offset administrative costs. In fiscal year 2014, CNCS awarded $12 million in grants to eight intermediaries to help governments and nonprofits build their capacity to implement and structure PFS projects.62 As of July 2015, the initial eight intermediaries had made awards to a total of 48 state and local governments and nonprofit organizations to fund technical assistance on PFS project development. According to CNCS, the grants address the limited availability of funding for planning, feasibility studies, and deal structuring, which has constrained growth in the field. In another example, DOJ and HUD are jointly pursuing a PFS demonstration project on permanent supportive housing for the prisoner reentry population.63 The project will make grants to intermediary organizations that will work with state and local governments and service providers to implement PFS projects and will include an evaluation to determine if this model of delivering supportive housing can achieve savings by decreasing government spending on corrections systems, homeless services, Medicaid, and crisis care services, some of which are federally funded. The UK central government addressed capacity issues by creating organizations to manage grant making and technical assistance for organizations involved in PFS projects. Through its Commissioning Better Outcomes Fund, the Big Lottery Fund64 provides up to £6 million (approximately $9 million) in grants (up to a maximum of £150,000 (approximately $229,000) per project) for technical assistance to assess 62The award recipients for the 2014 CNCS PFS grants are: (1) Corporation for Supportive Housing; (2) Green & Healthy Homes Initiative; (3) Harvard Kennedy School Social Impact Bond Lab; (4) Institute for Child Success, Inc.; (5) National Council on Crime and Delinquency; (6) Nonprofit Finance Fund; (7) Third Sector Capitol Partners, Inc.; and (8) University of Utah Policy Innovation Lab. Recipients are required to select subgrantees in a competitive process. 63Under DOJ’s fiscal year 2014 appropriation, the department was authorized to use up to $5 million (of its funding available for PFS projects) for programs implementing the Permanent Supportive Housing Model. DOJ entered into an interagency agreement with HUD to carry out a permanent supportive housing demonstration project with these funds. In fiscal year 2015, an additional $5 million of funding was authorized for PFS projects implementing the Permanent Supportive Housing model. 64The UK’s Big Lottery Fund is a government sponsored organization that is responsible for distributing a percentage of all funds raised by the National Lottery to projects supporting health, education, environment, and charitable purposes, from early years intervention to commemorative travel funding for World War II veterans. Page 61 GAO-15-646 Pay For Success the feasibility of PFS projects. It also provides outcome payments to encourage and sustain government engagement. The Better Outcomes Fund seeks to increase the capacity of service providers, enhance understanding of how to organize and implement a successful PFS project, and increase the knowledge base of local government officials working with PFS projects in the field. A stakeholder from the Centre for Social Impact Bonds said these grants have helped improve the technical and financial capacity of key players in PFS projects. State and federal officials also said that the federal government is uniquely positioned to help build capacity by encouraging knowledge sharing among parties that are implementing PFS projects. For example, to share knowledge with its grantees, DOL leads a working group for the state and local government recipients of PFS grant awards in an effort to increase knowledge and share best practices. DOL officials said that the monthly meetings have included speakers with PFS expertise, such as officials from Social Finance, the Peterborough Project, and other federal agencies. In addition, the White House Office of Social Innovation and Civic Participation co-hosted, with the Arnold Foundation and the Nonprofit Finance Fund, a series of PFS Summits in late 2014 and early 2015 to highlight and advance regional activity underway across the country, build the longer term Pay for Success infrastructure to catalyze future projects, and facilitate an open, transparent dialogue among public, private, and philanthropic stakeholders. Officials from one state government said that knowledge sharing by the federal agencies has been helpful to the state as it develops and implements PFS projects. UK government officials also said that knowledge sharing was an important component of the growth of PFS. As part of the annual reporting requirements for PFS grants, the Big Lottery Fund requires that grantees inform officials of plans to communicate information on their learning and experience as the project progresses. Where the Big Lottery Fund supports outcome payments, projects are also required to evaluate their own performance, contribute to an independent program evaluation, and support dissemination of learning from these evaluations. In the UK, officials from the Centre for Social Impact Bonds said the Centre has also addressed knowledge sharing issues by coordinating meetings with other government organizations, holding workshops, networking, and making connections between interested parties. Providing loan guarantees. A challenge to PFS projects that could limit growth is obtaining capital from investors. Investors said that the federal government could provide loan guarantees or credit enhancements for Page 62 GAO-15-646 Pay For Success There Is No Formal Mechanism for Federal Agencies to Collaborate on PFS PFS projects to increase the availability of capital by reducing risk for investors. Stakeholders said that having the backing of the federal government would provide greater incentives for investment in PFS projects and help establish the market. For example, Bloomberg Philanthropies provided a $7.2 million loan guarantee for the New York City Riker’s Island project to reduce some of the risk to investors of losing their investment if outcome targets were not achieved. However, some stakeholders said a government loan guarantee would negate a primary benefit of PFS to transfer the risk of program implementation away from the government. Such guarantees provided by the federal government would potentially leave the federal government liable for paying for a project that did not achieve agreed-upon outcomes. In our prior work, we stated that collaborative mechanisms, such as interagency groups or collaboration technology, can be used to develop policies, implement programs, and share information.65 The agencies in our study participated in an interagency PFS working group that met between 2012 and 2014 to share information on agency efforts related to PFS. Federal officials said the working group is no longer meeting because of staff turnover in leadership positions. Officials who participated in the working group said it was beneficial and furthered their agencies’ efforts related to PFS. For example, a federal agency official said the PFS working group helped officials network with their peers and collaborate on specific efforts. When CNCS developed its fiscal year 2014 PFS grant notice of funding availability, it consulted with other federal agency officials with PFS expertise. Though officials found these meetings useful, they were not maintained with a regular schedule and information resulting from meetings was not documented or broadly disseminated. In May 2015, OMB officials noted that the White House and federal agency officials are working to restart interagency meetings on PFS. A mechanism for agencies to share knowledge and information on PFS could give agencies interested in potential PFS projects access to leading practices and current information on how federal participation is evolving. 65GAO, Managing for Results: Implementation Approaches Used to Enhance Collaboration in Interagency Groups, GAO-14-220 (Washington, D.C.: Feb. 14, 2014) and Managing for Results: Key Considerations for Implementing Interagency Collaborative Mechanisms, GAO-12-1022 (Washington, D.C.: Sept. 27, 2012). Page 63 GAO-15-646 Pay For Success Conclusions In addition, a collaborative mechanism would allow agencies to share information on more technical aspects of PFS, such as practices on how to evaluate savings that accrue to federal agencies as the result of successful PFS efforts. Officials from one federal agency said that information about the various roles that federal agencies have played in the PFS field would be informative, given that the approach to supporting PFS has varied by agency. Having a central mechanism for sharing information on PFS would also decrease the risk of duplicative efforts in projects that affect multiple agencies. The PFS model is evolving as one among many tools for policymakers to consider when seeking to provide solutions for some of the nation’s most vexing social problems. According to stakeholders, PFS allows governments to shift the focus of social service interventions from costly remediation to lower cost prevention, while simultaneously mitigating risks associated with scale-up by funding projects with private investment. For service providers, the PFS model provides a source of upfront capital for operations, as well as an opportunity to scale-up promising programs and refine their capacity for sophisticated data collection and analysis. As organizations consider pursuing PFS contracts, they will need to think about how they will manage the risks inherent in PFS contracting. For example, a government’s costs of implementing a PFS project, which can include investor returns, management fees, and evaluation costs, can be high, so the government must decide whether potential benefits outweigh these costs. Flexibilities built into PFS contracts can help all parties to the project manage risks by allowing them to make changes as implementation and evaluation methodologies evolve. In addition, the parties to a project will need to manage the potential risks that unintended consequences will negatively affect the population receiving treatment. Managing these risks requires rigorous design of project implementation and evaluation methodology to minimize perverse incentives for organizations to focus treatment on the easiest cases while avoiding the more challenging ones that could negatively affect outcomes. PFS is most likely an appropriate tool for prevention programs that address entrenched social problems. It is important to consider key project feasibility factors, such as the availability of sufficient evidence on an intervention’s effectiveness and the capacity of PFS stakeholders to collect and analyze the data required by these complex projects. The PFS model is evolving, and early programs are just beginning to evaluate results. It is too soon to tell if governments will realize the Page 64 GAO-15-646 Pay For Success Recommendation for Executive Action benefits that induced them to try PFS. For example, a potential benefit of PFS is that it allows a government to shift the risk of implementing an innovative program to investors. In practice, investors whose return on investment is contingent on positive results may prefer projects that are based on rigorous evidence of success and may avoid innovative approaches that have not been rigorously tested. If this potential flight to programs with a strong evidence base turns out to become reality, it may not make sense for governments to rely on PFS projects. Instead, they may consider funding these types of programs directly, through traditional performance-based contracts that incorporate features of PFS projects that reduce the government’s risk, such as independent evaluation and governance rules that allow for strong management and oversight. By undertaking a PFS project to implement a program that is known to be successful, a government could be taking on extraneous costs for little or no benefit. As the PFS field continues to grow and as Congress and federal agencies consider new investments in these projects, important considerations remain about the appropriate role for the federal government. Similar to state and local governments, the federal government faces potential benefits and challenges stemming from participation in PFS programs. As federal agencies consider expanding their involvement in PFS, it becomes increasingly important for officials at all levels of government to collaborate to share knowledge and experiences. OMB led an informal working group on PFS for a limited period. Despite the stated usefulness of this group, notes and findings from the meetings were not shared among agencies, and the meetings were eventually discontinued. Having a mechanism for federal collaboration on PFS projects as the field grows would allow agencies to leverage the experience of early federal actors in the PFS field and would decrease the potential for missteps in developing projects due to information gaps and failure to learn from experience with this evolving tool of government. To identify and broadly disseminate information on leading practices and lessons learned, the Director of OMB should establish a formal means for federal agencies to collaborate on PFS. This could include creating a formal working group and providing collaboration technologies, such as shared databases or web portals. Page 65 GAO-15-646 Pay For Success Agency Comments and Our Evaluation We provided a draft of this report to the Director of the Office of Management and Budget, the Secretaries of the Treasury, Labor, Housing and Urban Development, Justice, Health and Human Services, and Education, and the Chief Operating Officer of the Corporation for National and Community Service. The Office of Management and Budget (OMB) provided written comments concurring with our recommendation and is working with agencies to explore options for continued collaboration on Pay for Success. The response from OMB is reproduced in appendix V. The Departments of Labor, Housing and Urban Development, Health and Human Services, Education, and the Corporation for National and Community Service provided technical comments that were incorporated into the draft as appropriate. The Departments of Treasury and Justice had no comments on the draft report. We are sending copies of this report to the Director of OMB, the Secretaries of the Treasury, Labor, Housing and Urban Development, Justice, Health and Human Services, and Education, and the Chief Executive Officer of CNCS. If you or your staff have any questions about this report, please contact me at (202) 512-6806 or sagerm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix VI. Sincerely yours, Michelle Sager Director Strategic Issues Page 66 GAO-15-646 Pay For Success Appendix I: Objectives, Scope, and Methodology Appendix I: Objectives, Scope, and Methodology This report responds to a request that we examine Pay for Success (PFS) to help Congress gain a better understanding of how the mechanism has been defined in theory and practice and what is known about its implementation and results to date. We examined (1) how selected PFS projects have been structured and what potential benefits these projects can provide; (2) how selected PFS contracts have been structured to address potential project risks; and (3) the potential roles for the federal government’s involvement in PFS projects. To address these objectives, we began by identifying and reviewing relevant literature on PFS, including work produced by public sector, nonprofit, academic, and private sector organizations. We also interviewed individuals with public management expertise who had examined PFS and published work on or related to PFS. We identified some of these individuals through our literature review, while others were referred to us by other experts. These individuals were Rob Duggar of ReadyNation; Mildred Warner of Cornell University; Leonard Gilroy of the Reason Foundation; John Roman of the Urban Institute; Jitinder Kohli of Deloitte Consulting LLP; and Shar Habibi of In the Public Interest. During these interviews we asked the interviewees, among other things, about the potential benefits and challenges of PFS; the advantages and disadvantages that PFS has compared to other program funding mechanisms; and the potential roles that the federal government could play in PFS projects. Based on our literature review and our interviews with individuals knowledgeable about PFS, we identified jurisdictions in the United States and abroad that were implementing, planning to implement, or studying the potential of PFS projects. We then selected seven U.S. and three United Kingdom (UK) case illustrations to review in depth. We selected cases from the UK in order to capture the country’s experience as the first to develop and implement PFS projects as well as because it is the location with the longest experience with this financing mechanism. We selected four PFS projects that were being implemented in the United States at the time we began our engagement in July 2014. Those projects were the Massachusetts Juvenile Justice PFS Initiative; the New York City Rikers Island Project; the New York PFS project: Increasing Employment and Improving Public Safety; and the Utah High Quality Preschool Program. Because those cases focused heavily on reducing recidivism and improving early childhood education, we selected two additional U.S. projects that were in development to reflect an additional policy area (improving health outcomes). Those projects were the South Carolina Maternal Health PFS Initiative and the Asthma Impact Model for Page 67 GAO-15-646 Pay For Success Appendix I: Objectives, Scope, and Methodology Fresno (AIM4Fresno). Finally, we selected one case in which a state considered but ultimately decided not to move forward with a PFS project. That case was Maryland’s review of a potential PFS initiative focused on reducing recidivism. For the UK projects, we selected three projects that were also in implementation and varied in issue area, structure, and the participating organization that conceived them. Those cases were the Peterborough Prison Project; the London Homelessness Project; and the It’s All About Me Adoption Project, on reducing adoption wait times. To obtain information about each of these case illustrations, we conducted site visits to Boston, Massachusetts; New York City, New York; Albany, New York; Annapolis, Maryland; Salt Lake City, Utah; and London, United Kingdom. During these visits, we requested and reviewed documents relevant to the case illustrations, including feasibility studies, contracts, and evaluation reports. We conducted in-depth interviews with representatives of organizations participating in the PFS projects in these locations. For each case, as appropriate, we interviewed representatives from government entities, intermediaries, service providers, investors, evaluators, and validators. We conducted some of the interviews by telephone before or after site visits due to scheduling conflicts. During the interviews, we asked the interviewees, among other things, about the potential benefits and risks of PFS; the advantages and disadvantages that PFS has compared to other program funding mechanisms; the specific roles their organizations play in PFS projects; the PFS project contract negotiations, including the establishment of outcome payment and governance structures; and the potential roles that the U.S. federal government or UK national government could play in PFS projects. The interviewees for our case illustrations were representatives from: Bank of America Merrill Lynch; Bates Wells Braithwaite; Big Lottery Fund; Big Society Capital; Bloomberg Philanthropies; The Boston Foundation; Bridges Ventures; Center for Employment Opportunities; Chesapeake Research Associates; Collective Health; Esmee Fairbairn Foundation; Goldman Sachs; Greater London Authority; Granite School District Preschool Services; Harvard University’s Social Impact Bond Technical Assistance Lab; Imprint Capital; Maryland Department of Legislative Services; Massachusetts Executive Office of Administration and Finance; MDRC; New Profit Inc.; New York City Mayor’s Office; New York Department of Corrections and Community Supervision; New York Division of the Budget; The Osborne Association; Public Consulting Group; Roca Inc.; The Robin Hood Foundation; Rockefeller Foundation; Salt Lake County Mayor’s Office; Sibayltics LLC; Social Finance (UK); Social Finance (U.S.); St. Giles Trust; Thames Reach; Third Sector Page 68 GAO-15-646 Pay For Success Appendix I: Objectives, Scope, and Methodology Capital Partners; UK Cabinet Office; UK Department for Communities and Local Government; UK Ministry of Justice; United Way of Salt Lake; Utah Governor’s Office of Management and Budget; Utah House of Representatives; Utah School Readiness Board; Utah State University’s Early Intervention Research Institute; Vera Institute of Justice; and Voices for Utah Children. To assess the PFS contract structure, we reviewed our prior reports and contracting literature to identify contracting practices from the federal experiences relevant to PFS contracts, which are complex service acquisitions. We compared the contracting practices to the PFS contracts in our study to establish their relevance to the design and implementation of PFS contracts. Because the scope of our review was to assess the potential benefits and risks of PFS and the structure of PFS projects, we did not assess whether the outcomes established for specific projects and any cost savings that were identified with their attainment were appropriate, ambitious, or met other dimensions of quality. In addition, since most of the case illustrations we reviewed were in early stages of implementation, we did not request any performance information related to them and did not assess the likelihood of the projects successfully achieving outcomes. To obtain information about federal agencies’ involvement in PFS, we reviewed relevant documentation, including the President’s Budget for fiscal years 2012 through 2016. Through this review, we identified eight federal agencies that were involved in an informal working group and played different roles vis-à-vis PFS. The Office of Management and Budget, working with the White House Office of Social Innovation and Civic Participation, has provided informal guidance on PFS efforts, and participated in the informal working group. The Department of the Treasury participated in the informal working group. The Departments of Labor (DOL), Housing and Urban Development (HUD), and Justice (DOJ), and the Corporation for National and Community Service (CNCS) have funded or plan to fund PFS projects. The Departments of Health and Human Services (HHS) and Education (Education) have been exploring opportunities to pursue the PFS concept. We interviewed officials from each of these agencies about their involvement in PFS, and asked them, among other things, about their strategic approaches for using PFS; the advantages and disadvantages that PFS presents compared to other financing tools; the amount of funding they allocated to PFS; interagency collaboration on PFS; and early lessons learned from their involvement in PFS. Page 69 GAO-15-646 Pay For Success Appendix I: Objectives, Scope, and Methodology Furthermore, of the eight agencies involved in the informal working group on PFS, we interviewed officials and reviewed budget documents from six agencies that typically fund social programs—CNCS, DOJ, DOL, Education, HHS, and HUD—to identify illustrative examples of how these agencies have used evidence-based approaches in their decision making more generally. We conducted our work from July 2014 to September 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Page 70 GAO-15-646 Pay For Success Appendix II: Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies Appendix II: Illustrative Examples of Evidence- Based Programs from Selected Federal Agencies To provide examples of how federal agencies are using evidence to inform decision making, we asked the agencies in our study to provide us with examples of existing programs or program proposals that were consistent with Office of Management and Budget (OMB) guidance on areas in which agencies could improve their use of evidence. Table 3: Illustrative Examples of Evidence-Based Programs from Selected Federal Agencies Example Social Innovation Fund (SIF) Justice Reinvestment Initiative (JRI) Reemployment and Eligibility Assessments and Reemployment Services (REA/RES) -Demonstration and Evaluation -Expanded REA/RES in all states Investing in Innovation Fund (i3) Agency Corporation for National and Community Service (CNCS) Department of Justice (DOJ) Department of Labor (DOL) Department of Education (Education) Description Since fiscal year 2010, the SIF has combined public and private resources to grow the impact of innovative, community-based solutions that have compelling evidence of improving the lives of people in low-income communities throughout the United States. The SIF integrates rigorous evaluation into its program model. Since fiscal year 2010, JRI has provided targeted technical assistance and grant funding to help state, local, and tribal governments. The goals of the initiative are to use evidence to • analyze data on criminal justice systems to identify what factors are driving prison and jail population growth; • develop strategies to control costs, improve public safety, reduce unnecessary confinement, and improve reentry programs; and • provide implementation grants to encourage adoption of significant policy and legislative changes resulting from JRI-supported efforts. The JRI also supports the work of the blue ribbon Charles Colson Task Force on Federal Corrections, which is charged with finding practical, data-driven approaches to addressing overcrowding in federal prison The REA/RES program seeks to reduce improper unemployment insurance payments and provide comprehensive services that assist participants with reemployment. Under the program, participants are provided referrals to services that facilitate job searches, including resume writing, interview techniques, and other job placement activities. In fiscal year 2015, DOL expanded the reemployment services model nationwide based on strong evidence of combined reemployment services and eligibility assessments for the Unemployment Insurance claimants. DOL meanwhile is further testing specific strategies in demonstration replications with rigorous evaluation. The Investing in Innovation Fund, established by the American Recovery and Reinvestment Act of 2009, provides funding to support (1) local educational agencies (LEA) or (2) nonprofit organizations in partnership with one or more LEAs or a consortium of schools. The purpose of this program is to support innovative and proven approaches to address persistent education challenges while also building knowledge of what works in key areas such as implementing college- and career- ready standards, using data to inform instruction and personalize learning, and improving low-performing schools. The i3 program uses a tiered-evidence framework to direct larger awards to projects with the strongest evidence base and support promising projects that are willing to undergo rigorous evaluation. Officials stated that the funding requested for 2016 would allow the department to continue fostering educational innovation while developing an evidence base in areas of high need and generating private-sector investment to complement the federal investment. Page 71 GAO-15-646 Pay For Success Example Promoting Readiness of Minors in Supplemental Security Income (PROMISE) Performance Partnership Pilots Disconnected Youth (P3) Maternal, Infant, and Early Childhood Home Visiting (MIECHV) Program Teen Pregnancy Prevention Appendix II: Illustrative Examples of Evidence- Based Programs from Selected Federal Agencies Description The PROMISE program is a crosscutting joint initiative funded by Education and SSA. In fiscal year 2013, Education awarded 5-year demonstration projects under the PROMISE program to develop, test and evaluate state-level interventions to improve outcomes for children with disabilities and their families and to reduce long-term reliance on the Supplemental Security Income (SSI) program and other public assistance. The projects are charged with establishing partnerships with state and local agencies and other entities to improve interagency collaboration in carrying out interventions and in developing innovative methods of providing services and supports. Awards were made to five states and one consortium of six states. P3 enables state, local, and tribal pilot sites to test innovative, outcome-focused strategies to achieve significant improvements in educational, employment, and other key outcomes for disconnected youth. Pilots must participate in a federally sponsored national evaluation of P3, which will consist of the analysis of participant characteristics and outcomes, an implementation analysis at all sites, and rigorous impact evaluations of promising interventions in selected sites. In fiscal years 2010-2015, the programs provided funding to support evidence-based models such as curricula designed to be administered to adolescents in school settings and youth-based leadership activities) to prevent teen pregnancy, sexually- transmitted infections, and associated sexual risk behavior. This initiative encompasses programs administered by the Administration for Children and Families, the Centers for Disease Control and Prevention, and the Office of the Assistant Secretary for Health. In fiscal years 2012-2015, the program provided funding to support evidence-based models to prevent teenage pregnancy, such as curricula designed to be administered to adolescents in school settings and youth-based leadership activities. This initiative encompasses programs administered by the Administration for Children and Families, the Centers for Disease Control and Prevention, and the Office of the Assistant Secretary for Health. Agency Education, Social Security Administration (SSA), DOL, Department of Health and Human Services (HHS) Education, DOJ, DOL, HHS, Institute of Museum and Library Services (IMLS), CNCS HHS HHS Source: GAO analysis of information provided by selected agencies. | GAO-15-646 Page 72 GAO-15-646 Pay For Success Appendix III: Overview of PFS Projects in Our Study Appendix III: Overview of PFS Projects in Our Study Table 4: Overview of U.S. PFS Projects in Our Study New York City (NYC) Rikers Island Project Massachusetts (MA) Juvenile Justice PFS Initiative New York (NY): Increasing Employment and Improving Public Safety Utah High Quality Preschool Program Policy area    Recidivism  Recidivism and employment Project time    Beginning in September and  Oct. 2013-Dec. 2019 Recidivism and employment    Early childhood education Oct. 2013- March 2019    School year 2013-2014 – School year 2017- frame Investment running 3, 4, or 6 yearsa 2018 $7 million, including a $4.6 million loan from Goldman Sachs and a $2.4 million subordinate loanfromJ.B.Pritzker Foundation. $9.6 million loan from Goldman Sachs. 75 percent of the loan is guaranteed by Bloomberg Philanthropies. $16.11 million, including a $8 million loan from Goldman Sachs and $8.11 million in funding from various philanthropicorganizations. $13.5 million equity investment from 40+ investors and foundations. 10 percent of the investmentisguaranteedby the Rockefeller Foundation. Size of target    Approximately 3,000  929 individuals 2,000 individuals    Approximately 3,500 individuals population Stakeholders • individuals per year 2 government agencies (NYC Mayor’s Office and Department of Corrections) • Intermediary (MDRC) • 2 service providers (Osborne Association and Friends of Island Academy) • 1 private-sector investor (Goldman Sachs) and 1 philanthropic loan guarantor (Bloomberg Philanthropies) • Evaluator (Vera Institute of Justice) • 12 government • agencies; (U.S. Department of Labor, MA Executive Office of Administration and Finance, MA Office of the Commissioner of Probation, and MA Department of Youth Services, MA Department of Corrections; the MA • Parole Board; the Suffolk County Sheriff’s • Department; the Middlesex County Sheriff’s Department; the Hampden County Sheriff’s Department; and the • Essex County Sheriff’s Department. Also, data for evaluation and validation come from the Executive Office for Labor and Workforce Development (wage data) and the Department of Criminal Justice Information Services (criminal records) 5 government offices • (U.S. Department of Labor, NY Governor’s Office, the NY Division of the Budget, NY Department of Labor, • and NY Department of Corrections and Community • Supervision); Intermediary (Social Finance) • Service provider (Center for Employment Opportunities) Over 40 private investors) and 3 • philanthropic investors (Robin Hood Foundation, Rockefeller Foundation, and Laura and John Arnold Foundation) 2 government offices (Salt Lake County and the Utah School Readiness Board) Intermediary (United W ay of Lake) 6 service providers 1 private-sector investor (Goldman Sachs) and 1 philanthropic investor (J.B. Pritzker Foundation) Evaluator (Utah State University) Salt Page 73 GAO-15-646 Pay For Success Appendix III: Overview of PFS Projects in Our Study New York City (NYC) Rikers Island Project Massachusetts (MA) Juvenile Justice PFS Initiative New York (NY): Increasing Employment and Improving Public Safety Utah High Quality Preschool Program • Intermediary (Third • Sector Capital Partners) • Service provider (Roca) • 1 private-sector investor (Goldman Sachs) and 5 philanthropic investors • Fiscal services providerb (New Profit) • • Evaluator (Urban Institute) • Validator (Public Consulting Group) Evaluator (NY Department of Labor Research and NY Department of Corrections and Community Supervision Research) Validator (Chesapeake Research Associates) Number of    1  3 outcome metrics Evaluation    Quasi-experimental  Random control trial evaluation 3    1 Random control trial    Review of evidence for method evaluation evaluation the achievement of specific outcomes Range of    $0-$11.7 million  $0-$28 million outcome payments First    Summer 2015  Job readiness payment: May $0-$21.5 million    Dependent on numerous variables Early 2017    Fall 2015 The state is renewing a 2-    Legislation created a outcome payment due 2015 Recidivism and employment payments: Fall 2017 Government    No new budget structure for  The state requests an annual budget structure appropriation for outcome payments and backs future payments with its full faith and credit. year appropriation every year. special account into which an appropriation to cover outcome payments was placed. outcome payments. Oversight    Intermediary was responsible  Provided by operating and Provided by the executive    Provided by the Utah structure for project management, including making adjustments to service delivery and coordinating with and reporting to other program partners. oversight committees, composed of the project’s stakeholders. steering committee, which provides strategic guidance, the management committee, which provides operational management, and working groups, which provide implementation support. School Readiness Board, which receives and reviews regular operational reporting from the intermediary. Source: GAO analysis of project documents. | GAO-15-646 aIn July 2015, the evaluator determined that the program did not lead to reductions in recidivism rates for participants in its 1-year assessment of the project. The New York City government did not make outcome payments to investors and the program was discontinued as of August 31, 2015. bThe fiscal services provider in this project is responsible for processing and overseeing payments to the intermediary and service provider. Page 74 GAO-15-646 Pay For Success Appendix III: Overview of PFS Projects in Our Study Table 5: Overview of United Kingdom (UK) PFS Projects in Our Study Policy area Project time frame Investment Project size Stakeholders Peterborough Prison Project Recidivism September 2010-August 2017a £5million ($8 million) 3,000 individuals • 2 national-level outcome payors (Ministry of Justice and Big Lottery Fund) • Intermediary (Social Finance) • Consortium of service providers (One Service) • Multiple private and philanthropic investors • Assessors (Qinetiq and University of Leicester and Greenwich) 1 Quasi-experimental evaluation – propensity score matching £0-£8 million ($0-$12 million) August 2014b Funding for outcome payments was provided by the Ministry of Justice and the Big Lottery Fund. Intermediary has responsibility for meeting with and reporting to the other stakeholders. London Homelessness Project Homelessness Begun in November 2012 and running for three years Information not publicly available 830 individuals • 1 national-level government organization (Department for Communities and Local Government) • 1 local-level government organization (Greater London Authority) • 2 service providers (Thames Reach and St. Mungo’s Broadway) • Intermediary (for St. Mungo’s Broadway: Triodos Bank) • Multiple investors 5 Greater London Authority reviews evidence for the achievement of specific outcomes £3-£5 million ($5 million-$8 million) Payments made quarterly throughout the project The UK Department for Communities and Local Government provided grant funding of up to £5 million ($8 million) to the Greater London Authority to commission the project and make outcome payments. Two groups composed of various stakeholders provide strategic and operational management. There is also regular government monitoring of service providers. It’s All About Me Project Adoption Beginning in October 2013 and continuing for 10 years £3 million ($5 million) 140 children per year • Multiple local-level governments (UK Local Authorities) • 1 contract management organization (It’s All About Me Service Company) • Multiple service providers • 2 Investor organizations (Bridges Ventures and Big Society Capital) 4 Review of government documents that show achievement of specific outcomes Up to £53,600 per child ($0 - $87,000) On registration of the child in the project Local authorities enter into a contract with a voluntary adoption agency The Board of the IAAM Service Company, composed of representatives of the developers, service providers, and investors, manages oversight. Number of outcome metrics Method for determining outcome achievement Range of outcome payments First outcome payment due Budgeting structure for outcome payments Oversight structure Source: GAO analysis of project documents. | GAO-15-646 aThe project was intended to include three cohorts of prisoners, each closing after 2 years or 1,000 prisoners had been discharged. As a result, the project was expected to include six years of prisoner intake, with service provision continuing for seven years. The period for data analysis and verification would take the entire length of the contract to eight years. However, the UK Ministry of Justice Page 75 GAO-15-646 Pay For Success Appendix III: Overview of PFS Projects in Our Study decided to terminate services for the third cohort after initiating a new criminal justice policy change, shortening the length of the project. bThe UK Ministry of Justice and Big Lottery Fund were due to make outcome payments if the evaluation of outcomes for the first cohort of prisoners showed a 10 percent or larger decrease in recidivism. However, in August 2014, the assessors found that recidivism among prisoners was reduced by 8.39 percent, and therefore no outcome payment was made. Page 76 GAO-15-646 Pay For Success Appendix IV: Example of United Kingdom Rate Card for Social Impact Bonds Appendix IV: Example of United Kingdom Rate Card for Social Impact Bonds Government officials from the United Kingdom (UK) told us that the rate of return for Social Impact Bonds funded with the Department for Work and Pensions’ (DWP) Innovation Fund are not subject to individual negotiations. The Innovation Fund is a pilot initiative that has commissioned Social Impact Bonds to help young people who are disadvantaged participate and succeed in education or training, therefore improving their employability and reducing their dependency on benefits. The Innovation Fund is 100 percent outcome based. Investors or intermediaries work with service providers to bid on Social Impact Bond projects based on the predetermined outcome payments published in DWP’s rate card (see figure 11). For example, if a service provider meets an outcome on an individual’s sustained employment, DWP pays £2,000 ($3,000). DWP makes awards for Social Impact Bond projects through a competitive bidding process. DWP will pay the investors or intermediaries solely on the basis of outcomes achieved. The service providers receive funding from the investors or intermediaries to cover their delivery costs, so they do not have to take the risk of depending on outcome payments. Page 77 GAO-15-646 Pay For Success Appendix IV: Example of United Kingdom Rate Card for Social Impact Bonds Figure 11: United Kingdom Department for Works and Pensions (DWP) Rate Card Note: National Vocational Qualifications (NVQ) are occupational certifications in the UK. The standards are statements of performance that describe what competent people in a particular occupation are expected to be able to do and cover all the main aspects of an occupation, including current best practices, the ability to adapt to future requirements, and the knowledge and understanding that underpin competent performance. Page 78 GAO-15-646 Pay For Success Appendix V: Comments from the Office of Management and Budget Appendix V: Comments from the Office of Management and Budget Page 79 GAO-15-646 Pay For Success Appendix VI: GAO Contact and Staff Acknowledgments Appendix VI: GAO Contact and Staff Acknowledgments GAO Contact Staff Acknowledgments Michelle Sager, (202) 512-6806 or sagerm@gao.gov In addition to the above contact, Elizabeth Curda (Acting Director) and Barbara Lancaster supervised this review and the development of the resulting report. Virginia Chanley, Karin Fangman, Sherrice Kerns, Donna Miller, and Daniel Webb made significant contributions to this report. In addition, Joy Booth, Alicia Cackley, Jeffrey DeMarco, Deirdre Duffy, Gale Harris, Katherine Iritani, Shelby Kain, Adam Miles, Susan Offutt, Erin Saunders-Rath, Paul Schmidt, and William Woods made key contributions. (451107) Page 80 GAO-15-646 Pay For Success GAO’s Mission Obtaining Copies of GAO Reports and Testimony Order by Phone Connect with GAO To Report Fraud, Waste, and Abuse in Federal Programs Congressional Relations Public Affairs The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability. The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (http://www.gao.gov). Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence. To have GAO e-mail you a list of newly posted products, go to http://www.gao.gov and select “E-mail Updates.” The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white. Pricing and ordering information is posted on GAO’s website, http://www.gao.gov/ordering.htm. Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537. Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order. Call for additional information. Connect with GAO on Facebook, Flickr, Twitter, and YouTube. Subscribe to our RSS Feeds or E-mail Updates. Listen to our Podcasts and read The Watchblog. Visit GAO on the web at www.gao.gov. Contact: Website: http://www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov Automated answering system: (800) 424-5454 or (202) 512-7470 Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512- 4400, U.S. Government Accountability Office, 441 G Street NW, Room 7125, Washington, DC 20548 Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149 Washington, DC 20548 Please Print on Recycled Paper.