Pay For Success: A Roadmap For Implementation in Minnesota

1 Pay for Success: A Roadmap for Implementation in Minnesota Capstone Paper In Partial Fulfillment of the Master Degree Requirements The Hubert H. Humphrey School of Public Affairs The University of Minnesota Aditi Kadam Jared Swanson Kyle Kretschmann Min Lee Nishank Varshney May 9, 2018 May 8, 2018 May 9, 2018 Date of oral presentation Approval date of final paper Mr. Dave Pinto Prof. Judy Temple Mr. Rob Grunewald Name and Title of Capstone Instructor Typed Name & Title, Client 2 ACKNOWLEDGEMENTS There are many individuals and organizations that provided help in various forms during the writing of this report. However, we would like to start by thanking the 3 individuals who have been with us throughout this entire process. We thank our wonderful instructor, Professor Judy Temple, for not only providing our team with incredible guidance during the writing of this report, but also for helping many of us during our graduate careers. We thank Representative Dave Pinto for his constant and enthusiastic support despite his busy schedule. Finally, this report would not have come this far without the involvement of Rob Grunewald. We are indebted to Mr. Grunewald for his generosity, insight, and overall support. The following individuals and their organizations aided our team when we needed targeted help: Kevin Gerdes, Mary Lou Garza, Susan Viker, and Terri Barreiro of the Humphrey School of Public Affairs; Brian Paulson of the Pohlad Foundation; Marcie Jefferys of Ramsey County; Lisa Backer of the Minnesota Department of Education; Richard Chase of the Wilder Foundation; Rachel Hardeman of the School of Public Health at the University of Minnesota; Margaret Kelly of the Minnesota Management and Budget Department; Dave Greeman of the Minnesota Department of Human Services; and Linda Li of Social Finance. We cannot thank them all enough for their time and help. We are also really thankful to Andrew Harris for his help in proof-reading the report. Finally, we would be remiss if we did not thank our friends and family. It was their day-to-day support and love that got us all to this point. If we ever make the world a better place, it will be because of them. 3 Table of Contents TABLE OF CONTENTS...................................................................................................................................... 3 EXECUTIVE SUMMARY................................................................................................................................... 4 PART I: PAY FOR SUCCESS – AN OVERVIEW........................................................................................... 5 INTRODUCTION.................................................................................................................................................... 5 HOW PFS WORKS................................................................................................................................................ 5 COST CONSIDERATIONS FOR A PAY FOR SUCCESS PROJECT............................................................................... 7 RISKS ASSOCIATED WITH THE PFS MODEL ........................................................................................................ 9 LIMITATIONS OF PAY FOR SUCCESS.................................................................................................................... 9 PART II: LESSONS FROM PAY FOR SUCCESS PROJECTS................................................................... 10 THE INNOVATION IN MINNESOTA PFS MODEL................................................................................................. 10 BARRIERS TO SUCCESS OF PFS IN MINNESOTA ................................................................................................ 11 THE CURRENT SITUATION OF PFS IN MINNESOTA ............................................................................................ 12 LESSONS LEARNED FROM PAY FOR SUCCESS LEGISLATIONS OUTSIDE MINNESOTA ....................................... 13 PAST & ONGOING PAY FOR SUCCESS PROJECTS............................................................................................... 14 PART III: HOW TO MAKE PAY FOR SUCCESS WORK IN MINNESOTA........................................... 16 FOUNDATIONAL ASPECTS ................................................................................................................................. 16 Statutes.......................................................................................................................................................... 16 Financial Mechanisms for a Pay for Success Model .................................................................................... 17 Timeline for a PFS Project............................................................................................................................ 20 ROADMAP FOR IMPLEMENTATION IN MINNESOTA............................................................................................ 20 Step 1 — Convening of Stakeholders ............................................................................................................ 21 Step 2 — Feasibility Study ............................................................................................................................ 22 Step 3 — Request for Information................................................................................................................. 22 Step 4 — Contracts and Risk Mitigation....................................................................................................... 23 Step 5 — Implementation .............................................................................................................................. 25 Step 6 — Evaluation...................................................................................................................................... 25 Step 7 — Assessing the Success of the PFS program.................................................................................... 26 RECOMMENDATIONS FOR INTERVENTIONS AND PROVIDERS IN MINNESOTA................................................... 26 Early Childhood Education........................................................................................................................... 27 Childhood Asthma......................................................................................................................................... 28 Child Dental Care ......................................................................................................................................... 29 Doula Care.................................................................................................................................................... 30 Homelessness ................................................................................................................................................ 31 CONCLUSION.................................................................................................................................................... 33 GLOSSARY......................................................................................................................................................... 34 APPENDIX.......................................................................................................................................................... 35 APPENDIX A – STAKEHOLDERS AND THEIR ROLE IN A PFS MODEL ................................................................. 35 APPENDIX B - RISKS ASSOCIATED WITH A PFS MODEL.................................................................................... 38 APPENDIX C – SUMMARY OF SELECTED PAY FOR SUCCESS LEGISLATIONS .................................................... 41 APPENDIX D - PAST & CURRENT PAY FOR SUCCESS PROJECTS ....................................................................... 43 REFERENCES.................................................................................................................................................... 49 4 EXECUTIVE SUMMARY Pay for Success (PFS) is a promising financing model that encourages investment in programs that produce improved social outcomes resulting in future cost-savings for the government. In a PFS project, investors provide initial capital to scale-up effective social programs and the government pays back the investors only if the desired outcomes are achieved. Minnesota emerged as a pioneer in this field, being the first U.S. state to enact legislation authorizing a Pay for Performance pilot in 2011, even before the first PFS program was launched in New York. However, despite having the legislation in effect for more than 6 years now, no PFS project has been implemented in Minnesota. Meanwhile, over 20 PFS programs have been launched in other states such as Illinois, Ohio, Colorado, and South Carolina among others, some of which have also seen their first success payments made out to the investors. This report describes the PFS financing mechanism and provides a set of recommendations for the state government and other stakeholders to advance the implementation of PFS projects in Minnesota recognizing the roadblocks that stalled implementation of the state Pay for Performance Act. We present a set of steps through which PFS funding can be approached in Minnesota and provide a list of program areas where PFS projects can be launched. The report also highlights the legislative action that could move PFS projects ahead in the state of Minnesota and discusses some ways to move forward in the absence of legislative involvement. 5 PART I: PAY FOR SUCCESS – AN OVERVIEW Introduction Interest in finding alternative ways to fund social programs has led to an increased focus on social impact financing or the PFS model. It is a financing mechanism where investors provide initial capital to expand effective social programs and the government repays the investors only if the desired outcomes are met. Each year, governments spend hundreds of billions of dollars addressing social problems (Liebman and Sellman 2013). Even though there is credible evidence that early investment in preventive programs in areas such as early childhood education and healthcare can help avoid spending on remedial programming later, thus saving money for the governments, they end up spending more on remediation than prevention (Temple and Reynolds, 2007; Kozhimannil et al., 2013). One reason it is difficult for governments to expand innovative preventive programs is that funding attempts at innovation come with both financial and political risk. At the same time, tight budgets lead to underinvestment in preventive programs, even when doing so would lead to savings in the future (Liebman & Sellman, 2013). The idea behind PFS is to create an investment opportunity in these program areas and use the realized savings to pay back investors. How PFS Works The PFS model has the potential to improve social outcomes, overcome barriers to social innovation, and encourage investments in cost-saving preventive services (Liebman & Sellman, 2013). Under this model, public funds are distributed to proven interventions and the risk of program failure is transferred from the public to the private sector. The stakeholders involved in a common PFS program are the government, private investors, an intermediary, a service provider, and an independent evaluator (For a detailed overview about each stakeholder and their role, see Appendix A). As shown in the figure below, the government enters into 6 a Pay for Success contract with the intermediary. The intermediary raises capital from the private investors which is then used to expand and scale effective services. Minimum performance targets, which are tied to government cost savings may be set prior to the services being delivered. If these targets are not met, the government does not pay and the investors lose their investment. However, if the targets are met, the government repays the investors. The repayment is usually a stepwise increase in payment for performance that exceeds the minimum target, with the highest performance generating the highest rate of return for the investors. Measuring attainment of the performance targets is a crucial step in the PFS model and can be challenging. The evaluation determines whether the investors are paid, and if so, how much they will be paid. Factors such as sample size, time horizon, and program design can affect the difficulty in obtaining a statistically valid measure of performance. Because of this, programs funded through PFS Private Investors & Philanthropic Funders Intermediary Government Service Providers 1. Investment 5. Principal + ROI 2. Working Capital 3. Outcomes & Savings 4. Performance based payments Fig. 1: Pay for Success Model Adapted from: South Carolina Healthy Connections SIB Model 7 are typically ones that have been implemented and evaluated in the past. The PFS model allows such programs to be implemented in new places and scaled up to serve more people. The PFS model might appeal to all stakeholders involved due to what it offers for each of them. Through PFS, governments are able to encourage innovation in social programs and shift the risk of this innovation from taxpayers to private investors. At the same time, they are able to save money by only funding proven preventive programs (Liebman & Sellman, 2013). Service providers are attracted to this model because it provides a source of multi-year funding while building a relationship with the government that may continue to provide funding if the organization is able to prove they are successful. Finally, investors are attracted to the model because they see an opportunity to promote social good while earning a reasonable rate of return. Some of the features of PFS projects can vary on parameters such as investor composition, risk bearing models, and repayment conditions (Nonprofit Finance Fund, 2018). In the South Carolina’s Nurse-Family Partnership project, the senior investor is a philanthropic organization that has agreed to reinvest the success payments to extend the service period of the program instead of claiming it as a profit. The PFS project at Rikers Island in New York City had Goldman Sachs as the private investor, but Bloomberg Philanthropies provided a guaranteed grant as a backstop to Goldman’s investment (Burand, 2013). Thus, Goldman Sachs was protected from large losses in the project, reducing the risk on their investment. The risk of investment can also be shared between the private investors and the intermediary or service provider. In this model, service providers may cover some of the upfront costs of providing services and only recoup these costs if the targets are met. Cost Considerations for a Pay for Success Project For any PFS contract to generate returns for the investment, it has to not only justify the cost of service provision but also cover the overhead charges associated with the program. This means that the benefits from a PFS project have to be lucrative enough to cover the cost of providing services, the 8 cost incurred by intermediaries to run the project, the cost of evaluating the program, and the costs for setting up the legal contracts. All of these costs included, a typical PFS project runs in millions of dollars. Current project costs range from $2.37 million in Ventura County to $30 million in South Carolina (Nonprofit Finance Fund Project Tables, 2018). The current statutes in Minnesota allow the offering of Pay for Performance bonds up to $10 million. Since the payment to investors is tied to the future prevented expenditure by the government, a rigorous impact evaluation is required to attribute the cost savings to the program. While an evaluation consisting of a randomized control trial is considered the most effective method, in some cases where it might not be feasible, other methods (e.g. propensity score matching) can be employed. These evaluation methods add a significant cost to the project, with a rigorous and reliable evaluation costing around 10% of the overall program budget. The costs of such evaluations, however, can be reduced by using outcomes measures already collected in administrative data sets. Negotiation of legal contracts also adds significant costs to the Pay for Success projects with the legal costs adding up for all parties. Because PFS is in many aspects still in its infancy, there is little reproducibility across projects, requiring contracts with different due diligence processes for each project. However, as PFS grows and contracts become more standardized, contract negotiation costs may decrease. Existing contracts are available online through the Nonprofit Finance Fund's Pay for Success Learning Hub. In some instances, as in Chicago's PFS project, contract negotiation expenses have been covered by philanthropic organizations in order to reduce the cost burden on involved parties and to encourage the implementation of the PFS projects (Nonprofit Finance Fund Project Tables, 2018). Covering these costs is another way for philanthropic organizations to be involved in PFS projects. 9 Risks Associated with the PFS Model One of the rationales behind the emergence of the PFS model is to transfer the financial risks of program performance from the taxpayer to other parties. Most investment models require risk-bearing and risk-sharing through diversification and this is also true of the PFS model. In her paper, Burand (2013) identifies the following categories of risks: intervention model, execution, intermediary, political, financial, and reputational. The assignment of these risks to agents which have the highest capacity to mitigate them is the ideal way to tackle these risks. Two of the more important risks arise from the possibility that the provided service may not generate predicted cost savings or that the government intended to be the payor may renege on the agreement. These risks and others are described in detail in Appendix B. Limitations of Pay for Success While the PFS model seems attractive and beneficial to all the stakeholders and society in general, there are certain criticisms and limitations of the model. Many early adopters theorized that the PFS model would pave way for innovative interventions, but in practice, it has only scaled up proven interventions due to the risks involved in pioneering new projects. The mechanism of PFS is complicated and therefore, the complexities may leave many stakeholders feeling unsure about how this funding mechanism works. Additionally, this model involves significant overhead costs associated with the intermediary and the evaluation that could be avoided if the government were to directly fund the programs. Another criticism of PFS is that projects with easier-to-measure outcomes are usually chosen over projects with outcomes that may be very important but require a longer time horizon to see the most important effects. Additionally, because of difficulty in contracting with multiple governments that may all potentially serve as payors, the chosen projects are typically ones that benefit one level or agency of government, restricting a number of potential programs from getting funded. 10 PART II: Lessons from Pay for Success Projects In 2011, Minnesota attempted PFS with the passing of HF681/SF434, which authorized the creation of a PFS pilot project (Temple et al., 2015). The bill passed as Section 27 16A.94 as part of an omnibus bill, was signed in to law in July 2011. This approved up to $10 million in bonding for nonprofit service providers who could meet performance and value targets. Bonding was favored as the basis for funding because the state of Minnesota faced a $6.2 billion deficit for the 2012-2013 biennium and did not have funds to directly allocate to a sinking fund (Minnesota Senate Fiscal Staff, 2010). The Act gave implementation authority to the Department of Management and Budget (MMB) who created the Pay for Performance Oversight Committee. The committee received proposals from agencies towards potential cost savings from social programs with a goal to innovate within the sphere of PFS financing and increase the funding of preventive programming. The use of bonds to fund PFS contracts made the legislation easier to pass as part of a bonding bill but created higher costs for future projects due to project’s returns needing to cover bond payments in addition to intermediary and evaluator costs. To avoid costs from bond payments, the next efforts in PFS in Minnesota may involve amending the funding source of this act. The Innovation in Minnesota PFS Model Minnesota was the first state to pass legislation authorizing the use of state bonds under the name of human capital performance bonds (Temple et al., 2015). The state attempted to innovate by being the first to use fixed-rate, fixed-term bonds as proposed by Rothschild (2013). Although no intermediary was mentioned in the Pay for Performance Act, as implementation proceeded, the plans involved shifting responsibilities for raising working capital and bearing risk, towards intermediaries 11 and service providers. In the Minnesota model, as the investors buy bonds, the proceeds are set aside in a state fund. Services providers do not receive initial funding at the time of service expansion and are not repaid until the evaluation is completed years later. To receive payments, service providers must show that they are creating sufficient cost savings to the state. Further, pinpointing where the cost savings occur can be difficult. Securing working capital remains the responsibility of the service provider and the selected intermediary. In this version of PFS, private investors technically are not directly investing in interventions but rather in bonds that may eventually pay for service provision. Barriers to Success of PFS in Minnesota A number of interrelated issues have influenced the lack of adoption of Pay for Performance or human capital performance bonds: • There was a political failure to find a passionate champion, a leader who brings together all stakeholders together and drives the program forward, during the process of implementation. • The requirement for raising working capital and the risk associated with not meeting success targets lay with the intermediary and service providers, who lacked the capacity to bear this risk. • The bill lacked support from service providers, due in part to fears of perverse incentives. • The legislation passed included provisions about the timing of compensation for investors and intermediaries that interfered with contract negotiation. • The state agency charged with overseeing pilot wanted to satisfy political considerations that would not necessarily maximize the likelihood of the pilot's success. While the human capital performance bond model was the idea of an entrepreneurial private citizen and nonprofit service provider, once the legislature approved the pilot there was no public or nonprofit leader able to push the process forward with sufficient inertia to clear the concerns about high administrative costs and large legal fees required to create a PFS contract. 12 According to numerous interviews with area informants, the “wrong pocket problem” (for definition, see Glossary) imposed constraints on the set of potential interventions. The two pilot projects that were planned but were not implemented were the EMPLOY program (Duwe, 2015) for recently released prisoners and a supportive housing pilot backed by the Corporation for Supportive Housing. Ultimately the chosen intermediary for the EMPLOY program (the Greater Twin Cities United Way) backed away from the implementation and no new intermediary was selected. Concerns expressed by the intermediary included its willingness to bear risk as well as the insistence by the state agency overseeing the pilot that for political reasons the services delivered should have a presence throughout the state when the intermediary wanted to choose service delivery sites to maximize the chance of success for the pilot. Without a pilot program, PFS has stalled in Minnesota due to a lack of proof of concept as well as uncertainty about the status of the 2011 Act. The current situation of PFS in Minnesota The agency that may be the furthest along in terms of planning for the implementation of a PFS project is the Minnesota Department of Education (MDE). In late 2016, the U.S. Department of Education awarded feasibility grants to Minnesota and 8 other states for studying the use of PFS for preschool programming. Minnesota's study involved examining the feasibility of using PFS to improve the quality of newly state-funded Voluntary Pre-Kindergarten (VPK) program that expands access to school-based preschool programs serving high poverty populations. The PFS study currently underway involves examining the use of the Pyramid Model to improve the quality of the VPK program. The Pyramid Model would be incorporated as a professional development framework, training teachers on how to support social-emotional competence in young children. The hope is that having teachers trained with the Pyramid Model will reduce behavioral problems in the classroom, ultimately producing potentially monetizable outcomes (e.g. lower teacher turnover, lower rates of special education placement, and higher rates of kindergarten readiness). 13 A $400 thousand grant from the U.S. Department of Education has enabled MDE to partner with SRI International, an independent consulting firm, to explore the practicability of using PFS (United States Department of Education/Health, 2016). The study involves preliminary data analyses, a prospective cost-benefit analysis, identification of potential PFS partners (payors, investors, etc.), and development of the evaluation methodology. The feasibility study began in early 2017 and is expected to be complete by late 2018. Pending the results, MDE hopes to begin securing funding from private investors to expand the use of the Pyramid Model using PFS. As this report is being written, interviewees point toward ongoing conversations on using PFS, including a plan for reducing county justice system involvement for at-risk youth in the Saint Paul Public Schools as well as plans in Ramsey County exploring use of PFS for supportive housing. Lessons Learned from Pay for Success Legislations outside Minnesota Starting in 2012 with New York City, several state and local governments have launched PFS initiatives with considerable success. Some states passed legislation to facilitate PFS funding of social, health, or human service projects. These legislative actions allow for the creation of PFS contracts, establish how state or local governments can set aside cost savings in order to repay the investors, designate who authorizes the contracts, establish a cap for funds, and create a flow of funds (Nonprofit Finance Fund Project Tables, 2018). Thus far eleven states and the District of Columbia have successfully enacted PFS legislation that provides a framework for project approval and the required funding. An additional five states including Maine and Delaware have taken a separate approach of passing legislation allowing for the study of PFS projects to be recommended to the legislature for future funding. In the states that have provided a funding mechanism, nearly 82 projects are either in progress or in development across the US (Nonprofit Finance Fund, 2018). There is a strong interest in accessing the funds in the states that passed PFS legislation, as evidenced by the fact that each state that passed legislation is currently developing or implementing a PFS project. 14 The common mechanism across states is to establish a trust or sinking fund with a maximum amount that can be appropriated for PFS contracts. A common appropriation cap across states is $50 million as seen in Texas and Massachusetts, though no state has approached that limit yet. Which body appropriates and oversees funding is, thus far, the main difference in PFS programs between states. In Texas the legislation enables the comptroller to act as the trustee who makes payments without legislative appropriations, but the Legislative Budget Board must certify the contract. Any unpaid funds are returned to the state treasury fund or its initial account. In Massachusetts, the Secretary of Administration and Finance oversees the funds and the legislature is responsible for appropriations each fiscal year which correspond to future savings. In Washington D.C. the Mayor enters into contracts and fills the sinking fund with the amount of savings each fiscal year. Colorado has established a dedicated fund for PFS and has also specified the roles of involved parties, defined where the risk is located, and set a maximum project length of seven years. While the content of the legislation can be flexible, having mechanisms to keep funds separate from the General Fund is vital to any PFS legislation. A detailed summary of PFS legislation in select states can be found in Appendix C. Past & Ongoing Pay for Success Projects PFS is still a relatively young concept. Only a few projects that have ended, many are currently ongoing. We analyzed some of the notable PFS projects that encompass a wide variety of topics, outcomes, and approaches. Table 1 provides a brief comparative analysis of six PFS projects and valuable lessons learned from them. Overall takeaways include: not overpromising on outcomes in the contract phase; using a short-term outcome that research ties to long-term savings; and that PFS financing should serve as complementary funding for the service program, and not as replacement funding. For more detailed information on these projects, see Appendix D. 15 Table 1: Comparative analysis of six PFS projects Project Peterborough Rikers Island South Carolina’s Nurse Family Partnership Utah High Quality Preschool Program Chicago Child Parent Center Cuyahoga County PFS Location Peterborough Prison, UK Rikers Island, New York State of South Carolina Salt Lake County, Utah Chicago, Illinois Cuyahoga County, Ohio Year of Launch 2010 2012 2016 Pilot from 2006-2012, PFS contract in 2013 2014 2014 Cost £5 million $9.6 million $30 million $7 million $17 million $4 million Focus Reduce Recidivism Reduce Youth Recidivism Improve Infant/Early Childhood health outcomes Early Childhood Education and Special Education reduction Early Childhood Education and Special Education reduction Homelessness and Child Welfare Status Ended in 2015 when policy changes rendered the comparison group invalid Ended in 2015 when project failed to reach targets In progress with success payments reinvested in expanding service In progress with success payments to investors In progress with success payments to investors In progress Lessons Learned Intermediaries who nurture the relationships of all other parties involved and monitor all the moving pieces are critical. A trusted flow of information between organizations and access to physical space improved effectiveness. Different payment triggering components might be more appealing to investors. A well thought out evaluation is a critical part of PFS projects. Tax-payers are not free from risk. Private investors are not always willing to take on all of the risk, foundation backstops can encourage greater risk taking. External funders can be philanthropic and might expect no interest on their return should repayments be triggered. There are ways around the wrong pocket problem i.e. Medicaid waiver. A feefor-service and Pay for Success model can be combined. Using a proof-of-concept year can be useful in securing state level investment. Using simplified payment triggers allows multiple sources of funding to be integrated. Identifying short term measurables that are correlated with long term outcomes is key. PFS model is feasible for programs such as early childhood education where the returns are usually observed much later in life. Private lenders are willing to enter PFS contracts with periods of repayment as long as 17 years. Integrated Data systems are key to project success. Senior and Subor dinate investors can have different interest rates. Deferred fe es can increase financial interest in project success. Indexing paymen ts to level of project success reduces risk for all parties. 16 PART III: HOW TO MAKE PAY FOR SUCCESS WORK IN MINNESOTA This part provides a roadmap for how a PFS project can be implemented in Minnesota that identifies the steps involved in initiating and carrying out a project. Additionally, we discuss potential interventions that could be funded through PFS. Foundational Aspects Statutes There are two Minnesota Statutes related to PFS (Minn. Stat. § 16A.94; Minn. Stat. § 16A.96) that were passed for the state's pilot in 2011. The Pay for Performance Program (Minn. Stat. § 16A.94) establishes a pilot to “demonstrate the feasibility and desirability of using state appropriation bonds to pay for certain services based on performance and outcomes for the people served.” This statute establishes an oversight committee, authorizes the commissioners of agencies to enter into pay-forperformance contracts with service providers, requires commissioners to establish evaluation methods for the programs, and requires annual reports to the governor and appropriate legislative committees on the pilot program. The second statute, Minnesota Pay for Performance Program (Minn. Stat. § 16A.96), authorizes the sale and issuance of appropriations bonds, up to $10 million for the purpose of funding pay for performance programs. These statutes require the state to sell bonds to pay for PFS projects, rather than allowing for appropriations to be set aside into a trust or sinking fund, an approach that other states have taken. This makes it more difficult, and potentially costlier to initiate a project as the return on investment must also be large enough to cover the state's bonding cost. 17 Financial Mechanisms for a Pay for Success Model Various financing models have been proposed and implemented for PFS projects which can be broadly categorized into two types. The first, and most common involves the government appropriating the required amount of money into a trust or sinking fund prior to signing the PFS contracts or during each year the project is in progress. The second does not require any funds to be set aside prior to or during the PFS project. The first method attempts to address the challenge that the cost savings may not be realized in a typical two-year cycle of a state budget. Therefore, there is a need for a mechanism that allows funds to be set aside beyond the two-year budget period. This could be achieved through laws that support legal contracts and guarantee a future payment to the investors if the PFS program is successful. States including Texas, Colorado, Massachusetts, as well as the District of Columbia, have all set up dedicated sinking/trust funds for PFS projects where the amount corresponding to future savings is to be appropriated each fiscal year that the contract is in effect. Some of the considerations for establishing a set aside account are: • This money could be appropriated by the state, either from the general fund or from the department that is expected to observe the cost savings. It may be more feasible for the state to appropriate the money from the general fund, if the state has a budget surplus. • The set aside account could be created within the state treasury, like the state of Colorado, or outside the state treasury, like the state of Texas. This could have an effect on the administrative process of appropriation of the money for the state, and on paying the intermediaries in the future, if the project is successful. • In cases where projects fail to meet the targets, funds that have been set aside could be returned back to the general fund or appropriate department, as is done in Texas. The funds could also be kept in the set aside account and used for future projects, as is done in Washington DC. 18 • Whether to set a limit on the amount of money that can be set aside should also be considered. A number of states, including Texas and Massachusetts, have kept this limit at $50 million. This would be a reasonable limit for Minnesota as well since most PFS projects are around $10 million. Another promising mechanism that has been tried in Europe and is now expanding to the U.S. is the outcomes rate card approach. Under this model, the government predetermines selected outcomes, its measurement methodology, and sets a price for each outcome up front. It then issues an RFP inviting private investors and service providers to work on the program. Unlike the abovementioned appropriation models, the bulk of the cost-benefit estimation work in the rate card approach is done prior to issuing the RFP. This approach reduces the cost of carrying out an impact evaluation since the outcomes and relevant metrics are determined prior to the project launch. It also has the potential to solve the “wrong-pocket problem,” as the outcome payment can be divided among different departments and across different levels of government. It also provides the ability to contract with multiple service providers at the same time and is easier to expand. There are, however, certain limitations to the prior appropriation of money for PFS projects. In this model, the government would be expected to appropriate the full amount of funds corresponding to the highest possible levels of outcome achievement, even though these levels may not be fully met. This creates a problem of opportunity cost, as the same pool of money could be allocated to other programs, rather than lying idle in a set aside account as a guarantee for potential success payments. This could translate into a political challenge as well, since legislators may not be willing to appropriate a large sum of money up front. A slightly different method of obtaining funds for success payments was established by Minnesota in 2011. The state opted for a financial model that secured funding using social impact bonds. The limitations of this approach, discussed in the previous section, may impede the 19 implementation of PFS in the state. Creating a sinking fund where PFS funds can be appropriated prior to or during the project, as other states have done, rather than the state relying on the sale of bonds to fund the projects, seems to be the most feasible method. However, there may be certain political hurdles to creating a sinking fund where some legislators might demand using this money for other purposes. If the state is unable to create a method for setting aside funds, three potential alternative methods which are outlined below could be considered - One alternative could involve the intermediary or investors entering into a contract directly with the unit of government that would save money if the project's targets are met. Success payments would be made directly from the department's budget as the savings produced by the intervention would reduce the funds required by the department. This approach has not been attempted in any PFS project thus far. One barrier that this method faces is that, because success payments are taken directly from the department budget, the financial savings must occur prior to the success payment. This may limit projects to those that produce savings in a short time or require contracts that allow success payment to be made farther into the future. Additionally, investors may be less willing to invest in a project that does not appropriate the success payments up front. A second way state and local governments can initiate PFS projects without appropriating money before/during the project is by tapping into federal funding newly available for PFS. In early 2018, Congress passed the Bipartisan Budget Act that included $92 million for PFS projects and feasibility studies (H.R. 1892). The new Social Impact Partnerships to Pay for Results Act (SIPPRA) authorizes the U.S. Treasury to award grants to state and local governments for PFS projects related to a range of issues. These grants could potentially allow states to fund projects with federal money, eliminating the need to make appropriations. A third possible way to finance PFS projects could be through a new legislation with the support and involvement of philanthropic organizations. Under this mechanism, legislation allowing 20 private organizations to provide upfront cost of service provisioning would have to be passed, allowing the savings accrued through the program to be shared with the investors. The legislation will also have to be clear about who will evaluate the savings and the details of payback to the investors. Finally, while it might initially seem challenging to carve out a sum as large as $10 million from the state budget for a PFS program, for the state of Minnesota, which has a budget of approximately $40 billion, it amounts to only 0.025% of the total state budget. In fact, it is only around 3% of the forecasted surplus of $329 million in the year 2018 (Berkel, 2018). Timeline for a PFS Project Stakeholders in the process, particularly legislative stakeholders, have an interest in a PFS contract being completed within two years. This reflects the two-year budget cycle used by the state and the desire by legislators to have the saving of the preventive spending accrue within a single cycle. This has both - political value for showing immediate savings, and fiscal value by not tying up funds across budget cycles. However, a two-year timeline is not feasible as the contract development, project implementation, and its evaluation can rarely be completed within a two-year timeframe, making at least a three to four-year timeline more realistic. Roadmap for Implementation in Minnesota The following roadmap provides a pathway for implementation of PFS in Minnesota. While the steps are presented sequentially, the implementation need not necessarily occur in the order shown below. This is particularly true with the first four steps as projects can begin in many ways. For example, a project may begin with a conversation among key strategic partners including an investor, selected community leaders, and representatives from the prospective government agency that might serve as the payor depending on the nature of the program. These partners could commission a feasibility study that would then suggest which other relevant stakeholders could be added to the 21 process. The steps listed can be iterative and occur in different orders. This roadmap is meant to highlight the processes needed to implement a project. Step 1 — Convening of Stakeholders Pay for Success initiatives typically start with discussions among various stakeholders including interested private funders, government agency representatives, nonprofit service providers and interested community members. As mentioned above, these conversations could commence with a set of key strategic partners, before a convening of a larger set of stakeholders occurs. The discussions Step 1 Convening of Stakeholders Step 2 Feasibility Study Step 3 Request for Information Step 4 Contracts and Risk Mitigation Step 5 Implementation Step 6 Evaluation Step 7 Assessing the success of the PFS program Fig. 2: Roadmap for PFS 22 could arise due to shared interests in addressing a particular costly social problem or stakeholders could be brought together with a common interest in PFS who will later identify the social problems to be tackled along with the chosen intervention. A common theme that emerged in our analysis is the need for a passionate champion for a PFS project. Liebman (2013) notes that due to the significant coordination and effort required to implement a PFS contract, these projects are only worth pursuing with a leadership team willing to devote the time and energy to make it work. This champion can be an elected leader or appointed agency head who drives the project forward. Alternatively, this champion may be a community leader concerned about a particular issue. Step 2 — Feasibility Study A feasibility study is conducted in order to understand the potential outcomes and resulting cost savings that could occur from the expansion of various social services while also identifying the relevant level of government that might serve as the payor. This step is strongly suggested to understand the context of the program and could be funded by philanthropic groups. Additionally, a feasibility study can serve as a tool to build consensus for the intervention. Step 3 — Request for Information A Request for Information (RFI) is used to identify possible service providers. A well-designed RFI ensures that the service provider selected has the experience, qualifications, and approach to help a PFS contract succeed. Quality bids should be expected to include organizational background, project goals, intended participants, timeline, location, evaluation capability, data tracking, budget of project, and criteria for selection. Ensure that the request emphasizes future needs of the program, uses open communication with bidders, and uses comprehensive decision making in selecting the service provider. We recommend innovative and effective service should take precedence over price. Also, consider state goals of working with targeted organizations that are majority women, minority, and 23 veteran-owned or located in an economically disadvantaged region. In total, administrative expenses should be 10% or lower. Step 4 — Contracts and Risk Mitigation Contracts To start a PFS project, stakeholders enter into a written agreement. In this step, the involved parties agree upon the structure, payoffs, and timeline of the project. The contract phase is often overseen by a project coordinator, the intermediary, that structures and manages risk. All parties involved in the process must be included in this step. If a party is not involved in the process, it can result in the contract requiring renegotiation which could extend the timeline of the project. Within the process, each stakeholder attempts to protect themselves from risk. Differences in opinion about how risk will be balanced and results will be measured, can lead to delays in implementation and possibly failure of the project. Contract negotiation is part of the transaction cost of successfully implementing a PFS model. We recommend including the following components in a successful contract: • Program Description • Goals of the contract • Clear definitions of terminology • Project timeline and cost • Responsibility for underwriting and securing upfront capital • Eligibility for services • Responsibility for program promotion • Data sharing and access • Governance structure 24 • Benchmarks and evaluation parameters • Payment formula and schedule • Termination clauses • Limitation of liability for advisors to the project • Randomization ratio and sample size • Defining senior investor vs. subordinate investor payments • Well defined flow of funds including initial funding and the flow of excess funds Risk Mitigation Strategy Because most PFS projects span multiple budget cycles of the legislative bodies that are responsible for appropriations, nearly all of the current PFS project have an appropriations risk mitigation strategy. Governmental appropriations are often subject to political and economic forces, which can make obligations promised years into the future difficult to guarantee. Appropriations risk mitigation strategies assure the investors that funds for success payments will be available if and when the project’s outcomes are met. Most current PFS projects have an appropriations risk mitigation strategy that involves setting aside a portion of the total success payment annually or biannually, rather than waiting until the success payments are due before appropriating funds. In most cases, these appropriations are placed into an escrow account or sinking fund. Regularly setting aside funding to be used for success payments is an important part of a PFS project’s design. An appropriations risk mitigation strategy lowers the risk for investors and ensures that funding for success payments is available when needed. 25 Step 5 — Implementation The work completed in steps 1 through 4 culminate in Step 5. This step is when the theoretical framework of the contract reconciles with the actual application. Using the well-defined terms of the contract the oversight committees should be able to provide clarification during implementation. Access and open communication are immensely important during this step. There should be reasonable and supportive access to physical spaces and data between stakeholders. Previous projects note this as an important component to maintain a highly tangled inter-organizational structure. Data sharing allows for smoother collaboration and implementation. The following will depend on the design and contract specifications, but one of the first steps during the implementation phase will be to transfer funds to the service provider. Once the service provider receives their funding, they can begin to expand and provide their services. The success of Step 5 mostly falls on the service providers; but investors, the intermediary, governmental organizations, and evaluators should be active in both supporting and, when needed, participatory roles. Step 6 — Evaluation This step involves using the evaluation plan including the sample size required to measure the effect, randomization of the treatment group, and deciding the parameters for defining and measuring success. Benchmarks or a payment formula can be used to evaluate success and may change year to year. The evaluation process is usually ongoing with the program. Benchmarks chosen should tie to the social good created by the program but are not required to explicitly provide savings. Based on the result of the evaluation, funds could be disbursed to the intermediary and used to repay the investors. Depending on the outcomes agreed in the contract, the payments could vary in size, based on the extent to which the performance outcome targets are achieved, and the duration taken to achieve these targets. However, if the performance outcome targets are not met, the 26 government may have no payment obligations and, the investors could lose some or all of their investment. Step 7 — Assessing the Success of the PFS program Upon completion of the PFS program, the government will need to decide on future funding. If targets are not met, it is unlikely that the program would be funded going forward. However, outside of the financial savings outlined in the PFS project, some other benefits may be realized. These should be taken into account when deciding on the future of the program. If the intervention proves successful, it is likely that the government agency will want to continue funding PFS programs. This can be done through a shift to a more conventional funding approach or through a ‘follow-on’ PFS contract, i.e., extending the contract (Liebman & Sellman, 2013). In order to avoid gaps in service delivery, Liebman and Sellman recommend that decisions about follow-on contracts should be made at least one year before service in the initial contract ends. Recommendations for Interventions and Providers in Minnesota A healthy PFS project will hinge on factors such as how well the project is designed, how much funding is available, and careful selection of a working intervention. An intervention will be a good fit for a PFS project if: it is simple in terms of training and implementation; it has a short time horizon; it yields a high return on investment, and the savings are easily attributable to a specific governmental department. These factors are also highly interconnected – for example, a simple intervention is important because it will save time which satisfies the desire for a short time horizon. More importantly, PFS could unlock the potential of burgeoning service providers. PFS gives service providers a secure, multi-year, and flexible source of funding that allows them to focus on providing high-quality service. This also moves service providers away from a fee-for-service model which allows them to focus on outcomes more than the number of people they serve. 27 Context will play heavily into whether an intervention is a good fit for a PFS project. That is to say, the needs of different populations in different areas will be very different. Understanding this context will help PFS projects target their work which will generate more return on investment. Generally speaking, PFS has focused on workforce development, early childhood education, early childhood development, health, criminal justice, and family welfare interventions because research has shown the cost-saving potential of these preventive interventions. With that in mind, we provide a brief overview of promising interventions. This section examines interventions related to early childhood education, childhood asthma, child dental care, doula care, and homelessness. Early Childhood Education Problem: Research has shown that the benefits of high-quality preschool programs far exceed the costs (Temple and Reynolds 2007). But even within established early childhood educational programs, there are problems that, when addressed, may produce cost savings. There is evidence that lower levels of behavioral and social skills are correlated with poor academic performance and later behavioral problems (Campbell, Shaw, & Gillion, 2000; Gregory, Skiba, & Noguera, 2010). Lower socio-emotional learning among preschoolers could also lead to their suspension or expulsion, with a nationally representative study finding at least 50,000 students were suspended once, and another 17,000 students were estimated to be expelled (Malik, 2017). Intervening effectively in social and emotional learning during the early childhood is critical to forming the foundation for children's learning (Blair 2002; Diamond & Lee, 2011). Costs: Expulsion of children from Pre-K due to lower socio-emotional learning can have an adverse effect on their performance in Kindergarten, requiring them to be offered special education. As per an estimate, Minnesota spends nearly $1.8 billion per school year on special education. While the average annual cost for a general-education student is around $8,500, the average cost for special education of kids is almost 2.5 times higher, at $22,000 (Meitrodt & McGuire, 2013). 28 Intervention: Benefit-Cost analysis has shown that socio-emotional learning programs can produce returns of $11.00 (Belfield et al., 2015). The Minnesota Department of Education (MDE) is currently exploring the feasibility of using PFS to fund the expansion of the use of the 'Pyramid Model' in some of the state’s Voluntary Preschool Programs (VPK). The Pyramid Model is a positive behavior supports framework for educators used to promote young children's social and emotional development, as well as to address challenging classroom behaviors. The goal of using the model in VPK classrooms would be to improve educational outcomes for students and increase the overall quality of VPK classrooms while reducing the number of suspension/ expulsions, and reducing the need of special education at later stages. Recommendation: Upon completion of MDE's feasibility study, expansion of the Pyramid Model in Minnesota's VPK programs using Pay for Success should be considered. Childhood Asthma Problem: More than 24 million Americans have asthma, affecting about 8% of children (Centers for Disease Control and Prevention, 2017), while an estimated 393,000 Minnesota children and adults have asthma (Minnesota Department of Health, 2016a). Every year, asthma accounts for more than 439,000 hospitalizations and 1.6 million emergency visits (Centers for Disease Control and Prevention, 2017). It also affects low-income children and black Americans disproportionately – for example, black Americans are 2-3 times more likely to die from asthma than any other racial or ethnic group (The 6|18 Initiative, 2015, p. 2). Costs: In 2009, the cost of treating asthma across the United States was estimated to be $62.8 billion by private insurance, Medicaid, Medicare, and other sources (The 6|18 Initiative, 2015). Minnesota Department of Health estimated the asthma cost across the state at $669.3 million, including $614.9 million in direct medical expenses and $54.3 million in lost workdays (Minnesota Department of Health, 2016a). 29 Intervention: There are a variety of tactics to mitigate asthma such as, but not limited to medication; monitoring inhaler use; monitoring breathing; identifying triggers such as cold air, air pollution, and pollen; and getting certain vaccinations (The Mayo Clinic, 2018). The following examples show potential interventions that can reduce healthcare costs related to childhood asthma. Recommendation 1: Healthcare provision using the National Asthma Education and Prevention Program (NAEPP) guidelines led to a 41% reduction of asthma-related emergency visits, a return of $3.00-$4.00 (The 6|18 Initiative, 2015, p. 3). Recommendation 2: Primary care providers using NAEPP protocols to diagnose asthma and provide a written asthma plan led to a 20% reduction in symptom days among children and 13% reduction in emergency visits compared to a control group. This creates a return of $3.58 due to savings in Medicaid and State Children's Health Insurance (Ibid). Recommendation 3: Expand home visits by professional and qualified health workers to provide targeted care, education, and identify and reduce in-home asthma triggers. Preliminary research found this creates a return of $5.30-$14.00, with most savings generated with children younger than 6 years old (Ibid, p. 10). Child Dental Care Problem: Dental care is particularly important during early childhood as untreated dental decay has effects on a child’s growth, body weight, and cognitive development (Arrow, Raheb, & Miller, 2013, p. 1). More specifically, in Minnesota, 55% of third graders had experienced dental decay while 18% had untreated cavities (Minnesota Department of Health, 2013). Lack of dental care more heavily impacts the American Indian and low-income populations. For example, 5 out of 10 American Indian children between the ages of 6 to 9 have untreated tooth decay. Meanwhile, only 61% of children living in low-income households visited a dentist compared to 85% for households living well above 30 the federal poverty line. Children under the age of 6 are 2 times less likely to visit a dentist for preventive means compared to children over 6 years old (Minnesota Department of Health, 2016b). Costs: The Minnesota Oral Health Plan 2013-18 estimated the cost of hospital-treated nontraumatic dental emergencies - which could have been treated by a dentist - at nearly $148 million from 2008-2010 (Minnesota Department of Health, 2013). Intervention: Research points to dental visits early in a child's life and education for parents as the leading interventions which can be seen in a comparison between "early starters", children who had their first dental visit before age 4, and "late starters," children who had their first dental visit after age four. When considering fillings, crowns, pulpotomies, and extractions, early starters on average had 7.69 operations compared to 11.27 for late starters (Nowak, Casamassimo, Scott, & Moulton, 2014, p. 491). The average early starter accrued costs of $694.32 over 8 years while late starters accrued $1,051.44 (Ibid). Recommendation: Dental programs that enable parents to provide dental care for their children before age 4 should be considered for PFS projects that attempt to address child dental care issues. Doula Care Problem: Doulas are trained professionals who provide individualized emotional, educational, and physical support during and shortly after the prenatal phase, even though they don't provide direct medical care. Use of doula care continues to rise in the United States as the benefits of doula care become more widely known, but total use remains low at about 6% in 2011 and 2012 (Declercq, Sakala, Corry, Applebaum, & Herrlich, 2013, p. 16). Access is a major issue for women and families seeking doula care, as the costs can vary from $300 to $1,200 with health insurance programs typically not covering doula care (Kozhimannil, Attanasio, Jou, Joarnt, Johnson, & Gjerdingen, 2014, p. e341). Furthermore, there are barriers to individuals seeking doula training, especially for those from disadvantaged groups. 31 Costs: In 2009, the cost of maternity and newborn care across the US was over $27 billion and 45% of that cost was billed to Medicaid programs (Ibid, p. e340). Cesarean births are also about 50% more expensive than vaginal deliveries (Ibid, p. 2). In 2010, nearly 1 in 9 births happened preterm and 35% of all infant deaths were from preterm-related causes. Infants born preterm generate medical costs 10 times higher in the first year compared to full-term infants (Kozhimannil, Hardeman, AlaridEscudero, Vogelsang, Blauer-Peterson, & Howell, 2016, pp. 20-21). Intervention: Research across 12 states in the West North Central and East North Central U.S has shown that births supported by doulas would decrease the number of preterm births and could save $58.4 million among Medicaid beneficiaries (Ibid, p. 25). There is also a growing body of literature that shows having access to doula support can reduce the amount of non-indicated cesarean procedures. Recommendation: A program that provides doula training and expands access to doulas for expecting mothers would be a promising intervention, especially programs that remove barriers to doula training for individuals from disadvantaged groups. Interventions involving doula care have the potential to produce cost savings in a shorter time than many other interventions. Homelessness Problem: Estimates indicate that on any given night in 2014, 578,424 people experienced homelessness and of that group, 401,051 were sheltered while 177,373 were unsheltered (United States Interagency Council on Homelessness, 2015, p. 16). Homelessness impacts African-Americans disproportionately – African-Americans make up 12.6% of the U.S. population but they represent 41.8% of the sheltered homeless population (Ibid). In Minneapolis, more than 3,000 evictions are filed in the 4th District Housing Court each year and most of those cases were concentrated in a handful of zip codes. In two specific zip codes, 55411 and 55412, 45-48% of renters experienced an eviction filing in the last 3 years (Minneapolis 32 Innovation Team, 2016, p. 2). Nearly all of these evictions were due to non-payment of rent, with tenants, on average, behind by $2,000 (Ibid). Costs: In 2013, Hennepin County observed that one-fourth of the families in the emergency shelter had been there before. It is estimated that preventing these families’ return to shelter could have saved $1.5 million in shelter costs per year for the community, in addition to the trauma caused by the homelessness (Kittock, 2016). Intervention: Having a stable home leads to increased educational, economic and health outcomes, decreased reliance on social safety nets, and a reduction in crime, further cutting costs to society. There are currently five PFS projects in the U.S. aimed at reducing homelessness including Assertive Community Treatment (Colorado), Project Welcome Home (California), Home and Healthy for Good (Massachusetts), Just in Reach (California), and the Cuyahoga County Partnering for Family Success program in Ohio (Nonprofit Finance Fund, 2018). Depending on the focus of an intervention, these programs could be good candidates as they will be evaluated over the course of their respective project and can be replicated accordingly in Minnesota. Recommendation: One of the challenges with funding homelessness projects through PFS is that measuring the impact might take more time than ideally desired, making it less appealing to the investors due to the longer time duration of the projects. While this might be a promising issue area for PFS, we would recommend starting with projects that have a shorter duration, to begin with, in Minnesota. 33 CONCLUSION Pay for Success has the potential to drive innovation and save the government money while improving social outcomes. There is interest in implementing PFS in Minnesota, but in order to launch a project, the state must change its approach towards PFS. One recommended step to move away from the bond model currently in the statute would be to adopt the more common model of funding PFS projects by appropriating funds into a trust or sinking fund, which is used in nearly all PFS projects across the country. Legislative action would be required to repeal the current statutes and replace them with a system that allows funds to be set aside. This might appear to be a large undertaking, but many states have already followed this process and their legislation can serve as a model for Minnesota. If prior appropriation of funds by the legislature is not feasible, the state could consider passing legislation to allow philanthropic organizations to fund PFS projects. Finally, the state should consider applying for a PFS grant from the U.S. Treasury (outlined in Part III) to fund a project. Interventions that are smaller in scale and that produce cost savings in a short amount of time should be considered for the first project once the groundwork is set. Such a project could serve as a pilot and allow the state to implement PFS on a smaller scale and in a shorter amount of time. As the number of projects in the state increase and the PFS process becomes more standardized, some aspects of the model (e.g. writing contracts) will likely become less expensive and easier to implement. The roadmap outlined in Part III can serve as a guide for all stakeholders involved in a PFS project. However, it is hard to imagine a PFS project being implemented without a strong champion. PFS projects need a strong leader willing to see the project through, advocating for the project and helping to coordinate all of the stakeholders involved. PFS seems to be an alluring model that benefits all stakeholders involved while giving the opportunity to expand socially beneficial interventions. 34 GLOSSARY Creaming Process where providers only sign up clients with the highest probability for success, rather than serving the clients with the greatest need. Evaluator Person or firm charged with determining if and to what degree a project has had the desired impact. Responsibilities include creating the evaluation plan, gathering information, analyzing data, making a determination and producing a report, the results of which are typically used to determine repayment to investors. Intermediary An organization that coordinated the project, is involved in the selection and payment of the service provider and the evaluator. Pay for Success A public-private partnership in which investors provide upfront capital to scale prevention-focused social interventions. Government re-pays the upfront capital plus a modest return only if the intervention produces measurable social impact. Request for Proposal A type of bidding solicitation in which an organization or institution announces that funding is available for a particular project or program, and companies can place bids for the project’s completion. Senior Investor When there are different classes of investors, senior investor claims to repayment stand in front of those of more “junior” investor classes. (See “Subordinate Investor.”) Service Provider The organization, often a non-profit, that delivers the services to the population stipulated in the contract. Sinking Fund A government pool of money that is set aside for a specific payback purpose typically the gradual repayment of a debt. Social Impact Bond A form of financing in which investors provide upfront financing for the delivery of services in a Pay for Success contract and are repaid only if the services achieve a pre-agreed upon set of outcomes. (Note: this does not work like traditional financial bonds. The misappropriation of the word "bond" has caused confusion.) Subordinate Investor Investor whose claim to repayment stands behind another investor class. Synonym for “junior investor.” (See “senior investor.”) Wrong Pocket Problem Describes a situation where the entity that bears the cost of implementing a practice—including an evidence-based best practice—does not receive a commensurate benefit. In PFS projects, if the benefit or savings from an intervention accrues to an entity other than the likely end payor, project development can be more challenging. Adapted from: Pay for Success Glossary online. Retrieved from http://www.payforsuccess.org/learn/glossary/#C 35 APPENDIX Appendix A – Stakeholders and their role in a PFS Model Investors Through the PFS model, investors can produce a social impact in a targeted community, reap a financial return, diversify their portfolios, and improve their public image (RAND, 2011). In addition to a lead investor(s), there could be one or more subordinate investors who get repaid only after the lead investor is paid back. These subordinate investors are usually philanthropic organizations. Having this kind of a tiered investment structure reduces the risk for the lead investor and is more replicable than the high-guarantee models since philanthropic organizations have a higher tolerance for risk in PFS and have a higher capability to blend financial resources with other risk-averse investors. Intermediaries An intermediary is the stakeholder that plays a major role in managing the relationships and duties of the other actors involved in the PFS. One of the primary roles of intermediaries is to monitor the smooth functioning of each of the parties involved. The intermediaries facilitate all the monetary transactions associated with the PFS. Evaluators The evaluator determines whether the target outcomes were achieved in accordance with the agreement between the parties in the contract. Some commenters suggest that there could be two distinct evaluator roles: 1) an evaluation advisor that helps define performance targets, designs assessment approach, monitors progress during life of PFS project, and analyzes interim assessment results as part of a PFS management team so as to help guide corrective action when/if necessary; and 2) an independent assessor that works on an arm's length basis from all of the other parties, and reports on whether SIB targeted outcomes have been met (Callanan et al., 2012). 36 Government The government passes legislation that allows for the creation of PFS contracts and the appropriation of funds into designated trusts and sinking funds. Additionally, the legislature creates different forms of governance structures who are tasked with supervision and oversight. While there are many potential arrangements for the management and oversight/governance structures of PFS projects, the standard structure for current PFS projects consists of two governance groups: The Operations Committee and the Governance Committee. These committees require all parties involved to dedicate time and energy to project coordination and seeing the effort through from beginning to end. Committees to support PFS Implementation Operational Oversight Structure The Operations Committee is the working group involved in the day-to-day monitoring of project progress. Because of this, these committees often meet more frequently, especially at the beginning of the project. They will meet as often as biweekly at the start of a project, then monthly as the project moves forward. The members typically include representatives from the service provider, the payor, the intermediary, the evaluator, and from the technical assistance provider (if there is such a provider for the project). Some of the responsibilities of the Operations Committee include monitoring operations of the project, identifying and resolving issues while the project is in process, elevating issues to the Executive Committee when necessary, and providing status updates to the Executive Committee. Executive Oversight Structure The Executive Committee is the decision-making body that provides strategic direction and oversight for the entire PFS project. These groups tend to meet less often than the Operations Committee, often quarterly to semi-annually. The members typically include representatives from the 37 service provider, the payor, and the largest investors. Executive Committee members should champion the PFS project within their respective organizations. Some of the responsibilities of this committee include providing leadership to ensure the project stays on track, review status updates from the Operations Committee and resolve issues raised by this committee and ensure compliance with the contract. 38 Appendix B - Risks associated with a PFS Model Intervention Model Risk The chosen social service interventions might not produce the expected outcomes (Burand, 2013). The probability of this occurring increases when the intervention is not administered by a proven social service provider. Inability to expand the model to a larger population (i.e. lacking statistical power to prove the success of the intervention) could make the model fragile. Thus, three ways to minimize intervention risks can be used. Firstly, use proven interventions instead of promising interventions that have no proven deliverable outcomes. Secondly, select measurable and unambiguous outcomes. And thirdly, accurately match the at-risk audience with an intervention that provides the maximum benefit. Execution Risk Execution risks arise from unclear lines of authority, poor communication among multiple participants, lack of resources and capacity, lack of follow-through by one or more partners, or failure to capture timely and reliable data on progress (Burand, 2013). Address risks by conducting rigorous prior research about the quality of service, the capacity, and the sincerity of the service providers. Additionally, strict provisions should be made to ensure the safety and the adequate protection of the population served by the PFS program (Burand, 2011). Warner (2013) expresses concern about organizations serving the most “fragile” clients including disadvantaged children, prisoners, and the homeless. If success targets are not met, the ability to serve these populations might be compromised in the future. Intermediary Risk Intermediaries are involved at every step of the PFS programs. Their involvement may differwhile some intermediaries may be very active in the program, some others may play a passive role in 39 the evolution of the PFS. There is a risk that the intermediaries may fail to perform their duties of coordinating between different agents. It is also possible that the intermediaries may fail to perform their duties for reasons out of the scope of the PFS Program. The scope of the intermediaries' responsibilities should only include those duties which are required for the success of the PFS. Political Risk Political risk is inherent to the PFS model and can be difficult to mitigate. Interference with the measurement outcomes of the PFS, non-cooperation for repayment to the investors, failure to provide support, and forcing re-negotiations of the contract, are all ways politics can impede the success of the PFS. Mutual understanding and agreement among different government elements is necessary to negotiate the outcome-based contractual agreements. Change in the ruling party or a change in the economic condition complicate the risk. A strong political champion of PFS who thoroughly understands its intricacies and is willing to advocate for PFS is needed. Financial Risk The early PFS models involved the private investors bearing all of the financial costs of the program, thus transferring all of the financial risks onto the private investors. However, due to the long term and non-liquid nature of investment, adding to the conditional repayment of the investment, would keep a wider range of potential private investors away from PFS investments. To mitigate these, risk sharing tools such as the provision of external collateral support, reserve funds, and first-loss provisions should be incorporated into the PFS model (Burand, 2013). Reputational Risk The service providers face a reputational risk if the PFS does not succeed. They could be publicized as not having met the desired deliverables for attaining the social outcomes. Burand (2013) recognized another important reputational risk, on the aggregate level there can be contagion risk of 40 highly publicized PFS controversies spilling over to the general marketplace. As PFS programs fail to meet their targets, the general confidence in the PFS reduces, and thus, more government agents and private investors would not be interested in taking these programs forward. Learning from previous PFS programs and avoiding similar mistakes while adopting the successful methods can mitigate this risk. 41 Appendix C – Summary of Selected Pay for Success Legislations Colorado In 2015 the Colorado Pay for Success Act passed. The act creates a PFS contracts fund in the state treasury where the general assembly transfers the funds based on an expectation of reductions in state expenditure resulting from the intervention. The state then pays the lead contractor based on the success of the outcomes based on an independent evaluation. The act identifies a "Lead Contractor" as an organization or local government that either provides interventions directly, or sub-contracts (and oversees the subcontractors) through either its own money or by borrowing it, while the "Provider" provides the interventions. Under the act, the lead contractor or the local government could be a provider if they do not sub-contract the service provision to a third party. In this model, the risk, if the program does not meet the required outcomes, is borne by the lead contractor (if they invest their own money), or the investor (if the lead contractors raise money from private investors). The act does not specify any agreement between the state and the private investors, meaning, the risks and responsibilities for execution of a PFS project lie mainly on the Lead Contractor. In fact, the Act specifically restricts an investor from dictating the manner of delivery of services except for performing due diligence on their investments. The Act identifies a maximum time period of seven years for the intervention unless one or more defined performance targets specified in the contract are met within the first seven years. Texas The state of Texas enacted an Act relating to the administration of PFS contracts. The Act set up a “Success Contracts Payment Trust Fund” outside the state treasury, with the comptroller as its trustee. This enables the comptroller to make success contract payments as per the contract terms without the necessity of an appropriation from the legislature. However, the contract can be executed only after the Legislative Budget Board certifies that the proposed intervention will result in significant 42 cost savings, and the state legislature appropriates the money for credit to the trust fund. If the contract fails to meet the outcomes or is terminated, the unpaid amount is returned back to the state treasury fund or the account from where the amount was initially appropriated. The act restricts the balance of the trust fund to a maximum of $50 million at any time. Washington DC Washington DC passed the Pay-for-Success Contract Authorization Act in 2014. The Act provided for initial funding of services by private investors for social programs performed by nonprofit service providers, with the contract being between the district and a social service intermediary. The Act authorized the Mayor to enter into the PFS contracts and operate a sinking fund where the amount corresponding to the savings would be appropriated in each fiscal year as per the future payment. The DC act ensures that the unrestricted fund balance is not reverted back to the General Fund of the District of Columbia. Massachusetts Massachusetts was one of the first states in the United States to pass an Act regarding the PFS program. The act established a Social Innovation Financing Trust Fund for the purpose of "Pay for Success Contracts." The Secretary of Administration and Finance is responsible for entering into the contracts and overseeing the trust fund. The Act requires setting up of a sinking fund where the amount corresponding to future savings is to be appropriated each fiscal year that the contract is in effect. The act restricts the maximum amount of payment at $50 million. 43 Appendix D - Past & Current Pay for Success Projects PFS is still a relatively young concept and many are still looking for the one-size-fits-all model. It's possible that model does not exist considering how complex PFS projects can become. Only a few PFS projects have fully finished while many are ongoing at the time of this writing. However, as with anything else, history provides valuable lessons that can be used when moving forward. The following section provides brief summaries on five PFS projects and valuable lessons learned from them. Peterborough's Prison – World's First Ever PFS Project The Peterborough PFS program was the world’s first attempt at PFS. The primary mission behind this program was to reduce reoffending by criminals with short sentences. The program was launched in 2010 when Social Finance raised £5 million from a mixture of trusts and foundations. This money then went to One Service who provided the intervention by identifying the needs of reoffenders such as mental health, substance abuse, housing, employment, and access to money. During the program's design phase, the national UK reoffending rate was around 60% (Social Finance, 2017, p. 1). A reoffending reduction rate of 7.5% was set by the Ministry of Justice as the level to trigger payments back to investors. The evaluation determined that this PFS plan reduced reoffending by 9% compared to a national control group and as a result, all 17 investors received their payment which consisted of their initial capital plus a 3% annual return (Social Finance, 2017, p. 1). The Peterborough agreement includes three cohorts of prisoners and early on cohort 1 failed to trigger an outcome payment. Evaluation found an 8.4% reduction in the frequency of reconviction in cohort 1 below the target for early repayment of 10% (Disley, Giacomantinio, Kruithof, & Sim, 2015, p. 3). The evaluators found the presence of a service director whose goal it was to coordinate and facilitate working partnerships throughout multiple phases was vital (Disley et al. 2015, p. 57). The complex inter-organizational design of PFS requires a central stakeholder who focuses on nurturing healthy partnerships. To highlight the importance of inter-organizational cohesion further, evaluators 44 found that amiable information sharing and accommodating access to physical spaces improved the program. Additionally, the evaluators advise future PFS programs to carefully consider the upfront cost of processing and training volunteers and the marginal benefit (Disley et al. 2015, p. 62). All interviewed prisoners expressed positive experiences with the One Service compared to prior experiences. Analysis indicated the primary needs of the participants were: material accommodations, financial aid, and education and training. A potential concern is the focus on the early month needs of prisoners and that longer-term engagement was more difficult (Disley et al. 2015, p. 60). Main Lessons Learned from Peterborough: • Having an intermediary whose primary objective is to nurture the relationships of all other parties involved and to monitor all moving pieces was critical. • Having a trusted flow of information between organizations and access to physical space improved effectiveness. • Having different payment triggering components might be more appealing to investors. Rikers Island - First Attempt at PFS in U.S. In 2012, the Rikers Island PFS project was the first PFS attempt in the United States; inspired by the Peterborough PFS project. The project involved: the Vera Institute of Justice as the evaluator; Bloomberg Philanthropies as the philanthropic backstop; Goldman Sachs as the investor who provided $9.6 million (Olson & Phillips, 2013); and MDRC as the intermediary. The project employed Moral Resonation Therapy and sought to reduce the re-incarceration rate by at least 10% (Porter, 2015). The Rikers Island project failed to trigger payments to Goldman Sachs. In that regard, it is considered a failure, but in many ways, it provided clarity to future PFS projects. Logistical issues contributed to the failure. The Education Department offered and rescinded to put teachers on Rikers Island to assist in the intervention. The Osborne Association, responsible for carrying out the therapeutic program, was 45 eliminated from the project when the teen population fell below the contractual threshold (Porter, 2015). Critics of the Rikers Island project point out that the tax-payer is not risk-free in this scheme. According to MDRC, “the arrangement required considerable in-kind support from city government leaders and staff,” and critics believe that these costs should be accounted for (Cohen & Zelnick, 2015). Critics also stress that all stakeholders involved need to agree on the evaluation metrics and that ample time is needed for honest evaluation. Main Lessons Learned from Rikers Island: • Evaluation will be a critical part of PFS projects and should be thoroughly thought out. • Tax-payers are not free from risk. • Private investors are not always willing to take on all of the risks. South Carolina — First State Wide PFS Project In 2016, South Carolina’s PFS project became the first in the United States to be implemented statewide (Social Finance, 2016, p. 1). The program focuses on health outcomes for both mothers and children living in poverty due to a 2015 report produced by the Annie E. Casey Foundation found that 27%, or 280,000, children in South Carolina lived in poverty. In line with early childhood development research the primary goals of this PFS project were: to create long-lasting results by helping first-time mothers become responsible parents and with their pregnancies; to support early childhood development in both urban and rural areas across South Carolina; to better understand the NurseFamily Partnership (NFP) model; and to create a more accountable government. South Carolina’s PFS project was led by the South Carolina Department of Health and Human Services (SCDHHS) in coalition with NFP as the service provider; J-Pal North America as the evaluator; Social Finance as the intermediary; and BlueCross BlueShield of South Carolina Foundation, The Duke Endowment, The Boeing Company, Greenville County — SC First Step, and 46 the Laura & John Arnold Foundation providing $17 million (Nonprofit Finance Fund, 2017). The funding was supplemented by $13 million coming from a 1915(b) Medicaid Waiver given to SCDHHS for a total of $30 million was available for the project (Social Finance, 2016). With the help of this PFS project, NFP aimed to serve 3,200 additional families over 4 years to the 1,200 they already serve (Nonprofit Finance Fund, 2017). The NFP model, which has been proven through 3 randomized control trials, pairs at-risk first-time parents with specially trained nurses who conduct home visits during pregnancy through the child’s second birthday. The evaluation period is 5 years and the outcomes that would trigger success payments include: a reduction in preterm births by 13.5%; a reduction in childhood hospitalization and emergency department visits by 23.4%; an increase in health spacing between births by 18%; and increased number of first-time mothers served who lived in high poverty areas (Nonprofit Finance Fund, 2017). If outcomes are achieved success payments will be reinvested in expanding NFP service delivery in South Carolina instead of returning to investors. Main Lessons Learned from South Carolina: • Please note that this project is ongoing at the time of this writing. • All external funders involved are philanthropic. • External funders expect no interest on their return should repayments be triggered. • This project addressed the wrong pocket problem (for definition, see Glossary) with a Medicaid waiver. • This project combined both a fee-for-service and PFS model. Utah — A Promising PFS Attempt Nearing Completion The Utah High-Quality Preschool program is considered one of the more promising PFS initiatives (Nonprofit Finance Fund, 2017). The program expanded the availability of preschool with success payments tied to special education avoidance. Currently in progress, the project has a five-year 47 delivery term and uses a longitudinal study to evaluate effectiveness over 12 years. A pilot from 2006 to 2012 showed promising results in reducing achievement gaps in language arts and math. In 2013 Goldman Sachs and the Pritzker Foundation added an investment of $7 million to a two phase PFS initiative. The senior investor, Goldman Sachs, is repaid first then Pritzker Foundation at a rate up to 7.26%. The intermediary is the United Way of Salt Lake. Phase one is the proof of concept which is funded by the United Way of Salt Lake and Salt Lake County and enabled 595 lowincome children to attend high-quality preschool. Phase two is supported by the State of Utah with the passage of H.B. 96 in 2014 that created the school readiness board and allocated funding (Utah H.B. 96 2014). The funding from Goldman Sachs and Pritzker allowed up to 3,050 children to attend the program. Repayment is triggered using results from a pre and post assessment. The project pays for outputs and outcomes. The threshold for repayment is 90% avoidance of special education for at-risk students. Of the 737 low-income students who attended Granite School District preschool only 1 of the 110 at-risk students used special education services by third grade (Goldman Sachs 2015). Therefore, a success payment was made. Main Lessons Learned from Utah: • Using a proof-of-concept year can be useful in securing state level investment. • Using simplified payment triggers allows multiple sources of funding to be integrated. • Identifying short-term measurables that are correlated with long-term outcomes is key. Chicago - A Promising PFS Attempt with a Long Time Horizon In 2014, the City of Chicago passed an ordinance authorizing a PFS contract to implement high-quality pre-kindergarten programs in certain schools. The major goals were to promote kindergarten readiness, increase proficiency in early school achievement, improve socio-emotional learning, increase parent involvement, and enhance educational attainment, career opportunities, and 48 the personal development of parents and family members. Funded by a $17 million social impact bond, the investment into the PFS program was made by Goldman Sachs Social Investment Fund (GSSIF), Northern Trust, and JB Pritzker Foundation (as a subordinate investor), in the proportion of 44%, 32%, and 24% respectively, with the Finnegan Family Foundation underwriting the program evaluation for the first two years. IFF is the intermediary, while the Chicago Board of Education is the outcome payor who has committed to a maximum repayment of $34 million over a repayment period of 17 years. The program has a four-year service delivery window with 374 children served in the first year, and 782 children each year for the subsequent years of operation. The city created a PFS Escrow account to fund the program contingent on its success. Payments triggers to investors were based on the evaluation of kindergarten readiness at the end of the second year of the project, third-grade literacy at the end of the fifth year, and special education utilization to estimate the savings for the City. The first two years of the PFS Program has triggered payments of $500,000 towards Kindergarten Readiness Success Payment, and $17,600 towards Special Education Payment for the Cohort 1, and approximately $900,000 towards the Kindergarten Readiness of Cohort 2. The ordinance included a comprehensive evaluation plan with the details of the construction of the treatment and comparison groups, estimation methods, and the required data and data sharing arrangements in order to evaluate the success of the program. The contract also included additional evaluation components related to the Child-Parent Center program that were not related to the PFS calculations but help improve the performance of the program going forward. Main Lessons Learned from Chicago: • PFS model is feasible for programs such as early childhood education where the returns are usually observed much later in life. • Private lenders are willing to enter PFS contracts with periods of repayment as long as 17 years. 49 REFERENCES 6|18 Initiative, The. (2015). "Control Asthma." Page. 2 & 10. Arrow, P., Raheb, J. & Miller, M. (2013). Brief Oral Health Promotion Intervention Among Parents of Young Children to Reduce Early Childhood Dental Decay. BMC Public Health, vol. 13, pp. 1-9. Belfield, C., Bowden, B., Klapp, A., Levin, H., Shand, R. & Zander, S. (2015). The Economic Value of Social and Emotional Learning. New York, NY: Center for Benefit-Cost Studies in Education, Teachers College Columbia University. Blair, C. (2002). Early intervention for low birth weight, preterm infants: The role of negative emotionality in the specification of effects. Development and Psychopathology, vol. 14, pp. 311- 332. Burand, D. (2013). Globalizing Social Finance: How Social Impact Bonds and Social Impact Performance Guarantees Can Scale Development. NYU Journal of Law and Business, vol. 9, pp. 447. Callanan, L., Law, J. & Mendonca, L. (2012) From potential to action: Bringing social impact bonds to the U.S. New York: McKinsey & Company. Campbell, S. B., Shaw, D. S. & Gilliom, M. (2000). Early externalizing behavior problems: Toddlers and preschoolers at risk for later maladjustment. Development and Psychopathology, vol. 12, pp. 467-488. Centers for Disease Control and Prevention. (2017). "Most recent asthma data." Retrieved March 28, 2018 from: https://www.cdc.gov/asthma/most_recent_data.htm Cohen, D. & Zlenick, J. (2015). "What we learned from the failure of the Rikers Island social impact bond." Retrieved March 8, 2018 from https://nonprofitquarterly.org/2015/08/07/what-we-learnedfrom-the-failure-of-the-rikers-island-social-impact-bond/ Declercq, E., Sakala, C., Corry M., Applebaum, S. & Herrlich, A. (2013). Listening to Mothers III: Pregnancy and Birth. New York: Childbirth Connection. Diamond, A. & Lee, K. (2011). Interventions shown to aid executive function development in children 4 to 12 years old. Science, vol. 33, pp. 959-964. doi:10.1126/science.1204529 Disley, E., Giacomantinio, C., Kruithof, K., & Sim, M. (2015). The payment by results social impact bond pilot at HMP Peterborough: Final Process Evaluation Report. Ministry of Justice. Disley, E., Rubin J.,Scraggs, E., Burrowes, N. & D. Culley (2011) Lessons learned from the planning and early implementation of the Social Impact Bond at HMP Peterborough. Santa Monica, CA: RAND Corporation. https://www.rand.org/pubs/technical_reports/TR1166.html. 50 Duwe, G. (2015). The benefits of keeping idle hands busy: An outcome evaluation of a prisoner reentry employment program. Crime and Delinquency, vol. 61, pp. 559-586. Gransberg, D. D., and Barton, R. F. (2007). Analysis of federal design-build request for proposal evaluation criteria. Journal of Management in Engineering, vol. 23, pp. 105-111. Gregory, A., Skiba, R. J., & Noguera, P. A. (2010). The achievement gap and the discipline gap: Two sides of the same coin? Educational Researcher, vol. 39, pp. 59-68. H.R. Res. 1892, 115th Cong. (2018). (enacted) Retrieved from https://www.congress.gov/115/bills/hr1892/BILLS-115hr1892enr.xml#tocHB9000910D99B40AEA6792496589C3F1F Kozhimannil, K., Attanasio, L., Jou, J., Joarnt, L., Johnson, P. & Gjerdingen, K. (2014). Potential benefits of increased access to doula support during childbirth. American Journal of Managed Care, vol. 20, pp. e340-352. Kozhimannil K., Hardeman R., Alarid-Escudero, F., Cogelsang C., Blauer-Peterson, C. & Howell, E. (2016). Modeling the cost-effectiveness of doula care associated with reductions in preterm birth and cesarean delivery. Birth, vol. 43, pp. 20-27. Kozhimannil K., Hardeman R., Attanasio L., Blauer-Peterson C. & Brien M. (2013) Doula care, birth outcomes, and costs among Medicaid beneficiaries. American Journal of Public Health, vol. 103, pp. 113–121. Liebman, J. (2011), Social Impact Bonds: a promising new financing model to accelerate social. Innovation and improve government performance. http://www.americanprogress.org/wp-content/ uploads/issues/2011/02/pdf/social-impacthbonds.pdf. Liebman, J. & Sellman, A. (2013). Social impact bonds: A guide for state and local governments. Harvard Kennedy School, Social Impact Bond Technical Assistance Lab. Mayo Clinic, The. (2018). "Asthma." Retrieved on March 28, 2018 from Mayo Clinic: https://www.mayoclinic.org/diseases-conditions/asthma/symptoms-causes/syc-20369653 Minneapolis Innovation Team. (2016). "Evictions in Minneapolis." Page. 2 Minnesota Department of Health. Oral Health in Minnesota. Retrieved on March 29, 2018 from Minnesota Department of Health: http://www.health.state.mn.us/divs/healthimprovement/data/quick-facts/oralhealth.html Minnesota Pay-For-Performance Program, Minn. Stat. § 16A.96 Nonprofit Finance Fund. (2017). South Carolina Nurse-Family Partnership. Retrieved on March 7, 2018 from Pay for Success: http://www.payforsuccess.org/project/south-carolina-nurse-familypartnership 51 Nonprofit Finance Fund. (2018). Homelessness. Retrieved on April 24, 2018 from Pay for Success: http://www.payforsuccess.org/learn/issues Nonprofit Finance Fund. (2018). Tables for Projects In Progress. Retrieved on March 25, 2018 from http://www.payforsuccess.org/projects/comparison Nowak, A., Casamassimo, P., Scott, J., Moulton, R. (2014). Do early dental visits reduce treatment and treatment costs for children? Pediatric Dentistry, vol. 36, pp. 489-493. Olson, J. & Phillips, A. (2013). Rikers Island: The First Social Impact Bond, Community Development Investment Review, pp. 97-101. Pay-For-Performance Program, Minn. Stat. § 16A.94 Porter, E. (2015). "Wall St. Money Meet Social Policy at Rikers Island." Retrieved March 8, 2018, from The New York Times: https://www.nytimes.com/2015/07/29/business/economy/wall-stmoney-meets-social-policy-at-rikers-island.html Rothschild, S. (2013) Human capital performance bonds, Community Development Investment Review, pp. 103-108. Social Finance. (2016). "Fact Sheet: South Carolina Nurse-Family Partnership Pay for Success Project." Page 1 Social Finance. (2017). "World's 1st Social Impact Bond Shown to Cut Reoffending and to Make Impact Investors a Return." Page.1. Shah, S. & Costa, K. (2013). "Understanding innovation funds, impact bonds, and impact investing." Retrieved February 22, 2018 from https://www.americanprogress.org/issues/economy/reports/2013/11/05/78792/social-finance-aprimer/ Temple, J., Punay, M., Barr, K. & Wagstrom, A. (2015) Pay-for-performance financing to expand cost-effective social services: Lessons learned in Minnesota. Nonprofits Assistance Fund, Minneapolis MN. Temple, J. & Reynolds, A. (2007). The benefits and costs of investments in preschool. Economics of Education Review, vol. 26, pp. 126–144. Temple, J. & Reynolds, A. (2015). Using cost-benefit analysis to scale up early childhood programs through Pay for Success financing, Journal of Benefit-Cost Analysis, vol. 6, pp. 628-653. United States Department of Education/Health. (2016, December 22). Awards -- Preschool Development Grants - Pay for Success. Retrieved from https://www2.ed.gov/programs/pfs/awards.html 52 United States Interagency Council on Homelessness. (2015). "Opening doors: Federal strategic plan to prevent and end homelessness." Page. 16 Warner, M. (2013). Private finance for public goods: Social impact bonds. Journal of Economic Policy Reform, vol. 16, pp. 303-319.