Evaluation of HUD-DOJ Pay for Success Permanent Supportive Housing Project

 DISCLAIMER The contents of this report are the views of the contractor and do not necessarily reflect the views or policies of the U.S. Department of Housing and Urban Development or the U.S. Government. Evaluation of the HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration Baseline Report Akiva Liberman Mary Cunningham Sarah Gillespie Brian Bieretz Matt Eldridge Evelyn McCoy Alexandra Ricks Samantha Batko Nicole DuBois Contents Acknowledgments 3 Executive Summary 4 Methods 6 Grantees’ Early Progress, Challenges, and Metrics 7 Benefits and Costs of PFS 9 Next Steps for the Evaluation 10 I. Introduction 12 Overview 12 Background 14 The HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration 23 II. Evaluation Questions and Methods 31 Studying Progress and Processes 32 Studying Costs 32 Studying Benefits 33 Providing Consultation on Data and Local Evaluation 33 Year 1 Data Collection 34 III. Grantees’ Early Progress, Challenges, and Metrics 36 Establishing PFS Partnerships for PSH 37 Local Data Use in the HUD-DOJ PFS Demonstration 43 Conducting Feasibility Analyses 52 Establishing Success Metrics and Evaluation 56 Other Issues in Implementing PFS in the Demonstration Sites 62 IV. Benefits and Costs of Pay for Success 67 Benefits of the PFS Process 67 Reported Costs of the PFS Process 69 V. Next Steps for the Evaluation 71 Appendix A. Site Profiles 73 Appendix B. NOFA-Delineated Activities by PFS Phase 108 References 113 EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 1  Acknowledgments This report was funded by the U.S. Department of Housing and Urban Development and the U.S. Department of Justice. We are grateful to them and to all our funders, who make it possible for Urban to advance its mission. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. Funders do not determine research findings or the insights and recommendations of Urban experts. Further information on the Urban Institute’s funding principles is available at urban.org/fundingprinciples. Thanks to Gordon Berlin, Richard Cho, Dennis Culhane, Barbara DiPietro, Joe Finn, Gloria Gong, Eileen Neely, Jim Parsons, Caterina Roman, and Lul Tesfai, who participated in our expert panel and provided insightful feedback on the research design and the findings included in this report. Andre Bethea and Ruby Qazilbash of the U.S. Department of Justice provided comments on this report, which strengthened the overall framing and sharpened the criminal justice focus. We thank them for their leadership on this issue. Calvin Johnson, Marge Martin, Marina Myhre, Deanna Reed, Todd Richardson, Galen Savidge-Wilkins, Mark Shroder, Carol Star, Norman Suchar, and Yennie Tse from the U.S. Department of Housing and Urban Development participated in the expert panel, asked hard questions, and provided important feedback on our findings and report drafts. Finally, we wish to thank the grantees for making time to participate in the evaluation. Any errors included in the report are the responsibility of the authors. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 3  Executive Summary In 2016, the U.S. Department of Housing and Urban Development (HUD) and the U.S. Department of Justice (DOJ) entered an interagency agreement and launched the Pay for Success Permanent Supportive Housing Demonstration (HUD-DOJ Demonstration). The purpose of the HUD-DOJ Demonstration is to strengthen communities’ ability to reduce recidivism and homelessness among the reentry population by increasing Permanent Supportive Housing (PSH) through Pay for Success (PFS), a promising financing mechanism that unlocks private and philanthropic investment in social programs. Permanent supportive housing combines a housing subsidy with intensive supportive services; it helps a subset of the reentry population that experiences long-term homelessness and that cycles in and out of crisis services, including jails, homeless shelters, hospital emergency departments, and psychiatric and detoxification centers. This cycling is detrimental to their health and well-being and comes at a high cost to the public. Despite strong evidence to support its expansion, permanent supportive housing is not yet available at the scale needed in many communities. The HUD-DOJ Demonstration aims to help scale PSH by launching partnerships between government agencies and private investors through PFS. Under PFS, investors pay the up-front costs of PSH and are repaid by the government if the program is successful, as measured by performance measures; these include reductions in jail stays or the use of shelters and are agreed upon in advance. The PFS structure involves multiple actors within a jurisdiction, including governments, funders, financial intermediaries, knowledge intermediaries, service providers, and independent evaluators, who measure the success of the program’s performance. PFS projects proceed through three phases: feasibility analysis, transaction structuring, and contract implementation (including evaluation of performance outcomes and success payments). As part of the interagency agreement between HUD and DOJ, DOJ made Second Chance Act (SCA) funds available for the Demonstration and designated HUD with responsibility for issuing a Notice of Funding Availability (NOFA) and providing oversight for the Demonstration, in partnership with DOJ. Congress passed the SCA (Public Law 110-199) with bipartisan support in 2008. The goal of the act is increasing reentry programming and improving recidivism outcomes for individuals released from state prisons and local jails. SCA grants support efforts related to education and employment assistance, substance use and mental health disorder treatment, housing, family programming, mentoring, victim support, and other reentry efforts. Grants must be collaborative across criminal justice and social service systems and collect data to measure performance. Because evidence shows a subset of the reentry population experiences homelessness and that PSH is a cross-sector solution that results in both housing stability and reductions in recidivism, it is a good fit for achieving the goals of the SCA. In June 2016, through a competitive process, HUD-DOJ awarded $8.7 million in funding to the seven grantees, who are listed in Exhibit 1 below. In each site, the grantee organization is the intermediary organization that is primarily responsible for overseeing the PFS process, including responsibility for assessing the feasibility of PFS, overseeing the process to structure a transaction, and then overseeing the implementation of the PSH project. Since different sites had different degrees of experience with both PFS and with PSH, sites were funded for different phases of PFS, as shown in the exhibit. The grantees also had varying degrees of experience collaborating across sectors, using data, serving a reentry population, implementing supportive housing, and evaluating outcomes. EXHIBIT 1 Pay for Success Permanent Supportive Housing Demonstration Sites Site Anchorage and Matanuska-Susitna Borough, AK Pima County, AZ Los Angeles County, CA Montgomery County and Prince George’s County, MD Lane County, OR Rhode Island Austin/Travis County, TX Intermediary (Grantees) United Way of Anchorage The Sorenson Impact Center at the University of Utah Corporation for Supportive Housing (CSH) American Institutes for Research (AIR) Third Sector Capital Partners The Rhode Island Coalition for the Homeless The Ending Community Homelessness Coalition (ECHO) PFS Phase at Demonstration Launch Feasibility Feasibility Transaction structuring Feasibility Feasibility Feasibility Transaction structuring The NOFA required each to launch a PFS project to fund at least 100 units of PSH for individuals in the target population who are chronically homeless, involved in the criminal justice system, and have high needs. For this Demonstration, participants either needed to meet HUD’s definition of chronic homelessness or have been homeless for the past 12 months cumulatively during the past 3 years or have been homeless for at least one night during each of the last 3 years.1 The criterion for criminal justice 1 “Chronically homeless’’ is defined in section 401(2) of the McKinney-Vento Homeless Assistance Act, 42 U.S.C. 11360 (McKinney-Vento Act or Act), as an individual or family that is homeless and resides in a place not meant for human habitation, a safe haven, or in an emergency shelter, and has been homeless and residing in such a place for at least 1 year or on at least four separate occasions in the last 3 years. The statutory definition also requires that the individual or family has a head of household with a diagnosable substance use disorder, serious mental illness, developmental disability, posttraumatic stress disorder, cognitive impairments resulting from a brain injury, or chronic physical illness or disability, see Chronically Homeless Final Rule. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 5  involvement was specified as having had multiple jail or prison stays within a 3-year period, with the most recent being in the past 12 months. The criteria for “high needs” were specified as having a history of high- cost utilization of services or significant health or behavioral health challenges that require high-cost support. Each site is required to contract with an evaluator to determine the success of the PSH intervention and monitor the success measures that trigger payments. HUD-DOJ contracted with the Urban Institute to conduct a national formative evaluation of the Demonstration. The goals of the national evaluation are to examine how PFS is implemented in the different Demonstration sites, capture the lessons learned across sites, and examine the feasibility of using PFS to fund PSH for a high-need, high-cost, homeless, reentry population. Methods Through the national evaluation, HUD and DOJ seek to assess whether PFS is a viable financing model for increasing the scale of supportive housing for a reentry population. The evaluation design is using a multidisciplinary, multimethod approach to “learn as we do” and meet four evaluation objectives: ■ Document progress and processes for the first 5 years of the HUD-DOJ Demonstration NOFA grant, including the identification of the successes and challenges that each grantee faced and how they overcame these challenges. Data collection activities in the first year involved site visits, interviews with key stakeholders, and monthly updates from sites to understand site progress and challenges. ■ Document the costs incurred by intermediaries, social investors, local governments, service providers, and end payors as they assess whether a PFS PSH model is feasible; negotiate and structure a transaction to implement an evidence-based intervention and facilitate outcome evaluation and validation before success payments; and implement PFS agreements with a focus on service delivery, quality control and oversight, and performance monitoring and outcome validation. ■ Document the benefits of a PFS PSH model for individuals, populations, and systems (that is, criminal justice, homelessness, and health/behavioral health) to inform subsequent cost-benefit, effectiveness, or impact analyses of this model. ■ Deliver consultation to HUD-DOJ Demonstration grantees and their independent third-party evaluation team on the development and collection of baseline outcome data. The evaluation seeks to document the reality on the ground as each community began its PFS project, describe the projects that develop in each site, and, in hindsight, understand how each community’s unique context may have affected the PFS project as it moved through each phase. Grantees’ Early Progress, Challenges, and Metrics This report covers the initial grant period from October 2016, when grant agreements with HUD-DOJ were completed, to December 2017, when baseline data collection was completed. The data collection and summary of grantees’ early progress, challenges, and metrics focus on the process and implementation of each project in their specific PFS phases, as well as how local data are being used across the PFS phases in each site. As noted earlier, sites started at different phases of the PFS life cycle. From October 2016, when grant agreements were completed, to December 2017, the end of baseline data collection, sites worked toward moving to the next PFS phase. As of December 2017, two sites were in the final stages of feasibility analysis: Lane County, Oregon, and Rhode Island. Two other sites had requested extensions to the feasibility analysis phase because tasks within the phase were ongoing: the Municipality of Anchorage and Matanuska-Susitna Borough, Alaska, and Montgomery County and Prince George’s County, Maryland. Pima County, Arizona, completed the feasibility analysis phase but, in January 2018, opted not to pursue transaction structuring. Austin/Travis County, Texas, continued in the transaction structuring phase. Los Angeles County, California, which started in the transaction structuring phase, had progressed to contract implementation in October 2017. The challenges confronted and milestones achieved by the sites during this period are discussed in the following. Establishing PFS Partnerships for PSH Many of the HUD-DOJ sites spent the initial grant period engaging important partners for their PFS projects. Commonly, six primary actors are central to the development and implementation of a PFS project: governments, knowledge intermediaries, financial intermediaries, private investors, service providers, and independent evaluators. Sites faced several challenges engaging partners, including educating stakeholders about PFS, parsing out site leadership roles, understanding the role of risk in PFS, aligning motives for PFS engagement and perspectives on financial feasibility, engaging end payors (particularly viable end payors for health savings), and working with government budgeting rules and processes for future success payments. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 7  Conducting Feasibility Analyses Considerable data are needed to successfully complete each PFS phase, including criminal justice data, homelessness data, and health data. In the feasibility analysis phase, data are needed to explore potential target populations and their size, to identify effective services and providers, to estimate anticipated outcomes, and to model costs and benefits for different government agencies that may make payments if outcomes are achieved. Feasibility analysis is the primary responsibility of the knowledge intermediaries at each site. Data Challenges Demonstration sites faced challenges identifying necessary data, accessing these data, and using data in each phase of their PFS project. Specifically, sites reported challenges obtaining access to data—especially when data owners were not in a PFS working group, challenges using data because data systems were not automated, or individual identifiers were not accessible, and challenges navigating data privacy issues. In three sites (Austin/Travis County, Pima County, and Rhode Island), the intermediary organization had completed feasibility reports by the end of 2017. In each of these sites, the intermediary organization deemed a PFS project for PSH to be feasible for the reentry population. However, in one of those sites (Pima County), the site’s steering committee, including potential end payors, decided to pursue PSH for a reentry population but without the PFS structure. Establishing Success Metrics and Evaluation The feasibility analysis also involves proposing success metrics and an associated approach to measuring them. These activities include analyzing baseline outcomes, identifying outcome metrics, and setting success payments. Engaging a local evaluator and designing an evaluation to measure impact and validate outcomes are issues under discussion during feasibility analysis. However, most sites that had not yet completed a transaction had also not finalized their evaluation designs or formally engaged a local evaluator. The most important evaluation design issue for the sites is how to create a comparison group that is equivalent to the PSH group. Although some sites were planning to use random assignment (that is, experimental design) to create a control group, other sites were deliberating the advantages and disadvantages of different designs. Urban has provided technical assistance to several sites, on request, in thinking through design issues. Other Issues in Implementing PFS in the Demonstration Sites The evaluation found other factors in sites that served to either facilitate or serve as barriers to progress. These included grant startup issues and challenges with program design that included identifying housing capacity and service providers for the project. In addition, projects were being implemented in the context of other ongoing reforms in the criminal justice and homelessness systems, such as efforts to improve coordinated entry- and system-level performance measures in the homeless system and efforts to reduce the use of expensive criminal justice resources for low-level offenders and individuals. These efforts include the Justice Reinvestment Initiative, the Safety and Justice Challenge Network, and the Data- Driven Justice Initiative. Some of these other efforts may help support the work of the project, but some may also add complexities to implementation.2 Benefits and Costs of PFS Proponents of the PFS model frequently reference benefits of the PFS process that go beyond financing a project as a reason to consider PFS, which may offset the upfront time commitment and costs associated with establishing a PFS project. Sites reported increased attention to the target population and the efficacy of PSH, new and strengthened partnerships, and improved capacity in the jurisdiction that could extend beyond the PFS project and target population. Some sites saw investment in data sharing and systems as an ongoing benefit to the community, as well as increased collaboration to provide programming for the target population or a similar population, as a direct result from the feasibility analysis phase of the PFS work. One significant cost of using PFS is the time it takes to engage partners and structure a PFS contract. Some key informants indicated that significant time was spent by the intermediary and intermediary advisers, and at least one site indicated that the funding provided by the HUD-DOJ Demonstration was insufficient for the work necessary to complete the PFS phases. In several instances, partners across sites identified tasks outside of the specific PFS phase tasks as particularly consuming, especially outreach, engagement, and education to potential partners that occurs outside formal PFS meetings. We will be 2 The Justice Reinvestment Initiative, funded by the Bureau of Justice Assistance of the U.S. Department of Justice, uses data analysis to identify costs and drivers of criminal justice populations and to develop evidence-based and cost- efficient policy options to reduce them. The Safety and Justice Challenge Network, funded by the John D. and Catherine T. MacArthur Foundation, aims to reduce the use of jail and prison through sentencing reform and alternative sanctions. The Data-Driven Justice Initiative, launched by the White House in 2016 and led by the National Association of Counties, integrates data to drive strategies that reduce the reliance on criminal justice and incarceration for low-level offenders with mental illness. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 9  collecting more data on both the benefits and costs of PFS during the next year of the evaluation. We describe these next steps below. Next Steps for the Evaluation In the first year of the evaluation, data collection focused on understanding the PFS projects under development at each site, the processes sites navigated as they moved through the PFS phases, and the data they used or planned to use in each phase. This information provides a snapshot of where each community was at the start of the Demonstration, although some began their PFS efforts well before the federal funding was available. Beginning in the second year, the evaluation will collect more data on the costs incurred during each PFS phase and the benefits achieved at the system, program, and individual levels. Specifically, the research team will conduct the following activities: ■ Document the costs of PFS. The research team will document the costs the PFS partners incurred as they move through the PFS phases. Most of the costs associated with developing a PFS PSH project stem from partner participation. Although substantial grant funding is available from HUD- DOJ, the staff time from involved partners often exceeds that supported by the Demonstration grants. Therefore, the research team will collect data on the time that key partners spend on the project and will also identify sources of funding and leverage that are used to support the project in addition to the HUD-DOJ grant. ■ Document the benefits of PFS. The research team will begin to document the overall benefits of a PFS PSH model at the system level, program level, and individual level. By studying benefits, we will learn about both the indirect benefits produced by the PFS model as a mechanism to force systems change, as well as the direct benefits of the program through positive cross-sector outcomes for enrolled individuals. The national evaluation is not set up as a cost-benefit analysis, and we will not try to quantify the benefits we measure through qualitative data collection. Studying the system, program, and individual benefits of each project, however, will help us describe the value added by the PFS model. This report contributes to research on how communities are working to launch PFS projects to fund PSH for the reentry population experiencing chronic homelessness. It describes the implementation of PFS across seven sites during the first 14 months of the Demonstration. As noted above, the data collection activities during the next period will continue to focus on the implementation of PFS and highlight some of the costs and benefits of using this financing mechanism to expand PSH. The research team will report these results in the evaluation’s second report. In addition to annual reports, the research team is reporting site milestones on a biannual basis to provide more up-to-date information on site progress. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 11  I. Introduction Overview In 2016, the U.S. Department of Housing and Urban Development (HUD) and the U.S. Department of Justice (DOJ) entered an interagency agreement and launched the Pay for Success Permanent Supportive Housing Demonstration (HUD-DOJ Demonstration). The purpose of the Demonstration is to strengthen communities’ abilities to reduce recidivism and homelessness among the reentry population by increasing Permanent Supportive Housing (PSH) through Pay for Success (PFS), a promising financing mechanism that aims to unlock private and philanthropic investment in social programs. PSH is an evidence-based intervention that combines a housing subsidy and intensive supportive services to help individuals who experience long-term homelessness and cycle in and out of jail and prison and crisis services, like hospital emergency departments and psychiatric and detoxification centers. This cycling is detrimental to their health and well-being and comes at a high cost to the public. Studies show that PSH results in long-term housing stability, improved physical and behavioral health outcomes, and reduced use of crisis services, including jails (Aidala et al., 2014; Culhane, 2018). PSH is most successful when combined with Housing First, a philosophy in which permanent housing is provided without preconditions and barriers to entry, such as treatment or service participation requirements or sobriety (HUD, 2015). Despite strong evidence showing the intervention works, PSH is not yet available at the scale necessary to address the need in many communities. Thousands of people who cycle in and out of jail or leave prison without stable housing experience homelessness each year (USICH, 2016). The Demonstration aims to help scale PSH by tapping private investors to pay the upfront costs of PSH for a reentry population experiencing chronic homelessness. If the intervention is successful, an end payor, usually the government, “pays for success” measured by reductions in homelessness and recidivism and improved health outcomes. This structure involves multiple actors within a jurisdiction, including governments, funders, financial intermediaries, knowledge intermediaries, service providers, and independent evaluators. PFS projects are broken down into three phases: feasibility analysis, transaction structuring, and implementation. As part of the interagency agreement between HUD and DOJ, DOJ made Second Chance Act (SCA) funds available for the Demonstration and designated HUD with responsibility for issuing a Notice of Funding Availability (NOFA) and providing oversight for the Demonstration, in partnership with DOJ. Congress passed the SCA (Public Law 110-199) with bipartisan support in 2008. The goal of the act is increasing reentry programming and improving outcomes for individuals released from state prisons and local jails. DOJ’s Office of Justice Programs (OJP) administers the grants with the Bureau of Justice Assistance (BJA) and Office of Juvenile Justice and Delinquency Prevention (OJJDP) awarding grants to communities. SCA grants support efforts related to employment assistance, substance use disorder treatment, housing, family programming, mentoring, victim support, and any other reentry efforts. Grants must be collaborative across criminal justice and social service systems and collect data to measure performance. BJA and OJJDP have funded dozens of projects over the years to improve reentry outcomes in communities across the United States. Because evidence shows that a subset of the reentry population experiences homelessness and that PSH is a cross-sector solution that results in both housing stability and reductions in recidivism, PSH is a good fit for achieving the goals of the SCA. HUD issued the NOFA in October 2015, inviting applicants from nonprofit organizations and institutions of higher education to apply for grant funds. HUD-DOJ evaluated the applications on a competitive basis and in June 2016 awarded $8.7 million in funding to seven grantees in communities across the country, including urban and rural areas. Each site could receive a maximum of $1.3 million. Grantees were funded for up to three phases of PFS: feasibility analysis (up to $250,000), transaction structuring (up to $600,000), and project implementation, including evaluation (up to $250,000) and success payments (up to $1.3 million). The following sites were selected: ■ Anchorage and Matanuska-Susitna Borough, Alaska ■ Pima County, Arizona ■ Los Angeles County, California ■ Montgomery County and Prince George’s County, Maryland ■ Lane County, Oregon ■ Rhode Island ■ Austin/Travis County, Texas The NOFA identified the following objectives of the HUD-DOJ Demonstration. ■ Pilot test the PFS model to learn whether the financing approach is feasible to fund PSH. ■ Create an opportunity for communities to increase the availability of PSH for the target population. ■ Determine whether providing PSH for the target population will reduce recidivism to jail and prison and reduce chronic homelessness. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 13  ■ Determine whether a PFS-financed PSH intervention can significantly improve taxpayer savings or cost-effectiveness by decreasing enough government spending on corrections systems, homeless services, Medicaid and other health costs, or crisis services to justify the use of a PFS framework and to ultimately cover the cost of expanding PSH to other jurisdictions (in addition to a return to investors if PFS financing funds future expansion). ■ Identify lessons learned and best practices from other jurisdictions on scaling up and replicating effective PSH and other service delivery interventions, with or without federal funding and technical support. HUD-DOJ contracted with the Urban Institute to conduct a formative national evaluation of the Demonstration. The goals of the evaluation are to examine how PFS is implemented in the different Demonstration sites, capture the lessons learned across sites, and examine the feasibility of using PFS to fund PSH for a reentry population. Background Considerable need exists for permanent support housing. In 2017, HUD estimated about 554,000 people in the United States experience homelessness on any given night, including almost 87,000 individuals who are chronically homeless, defined as having experienced homelessness for long or repeated periods of time and having a disability (Henry et al., 2017). Research shows a strong link between homelessness and incarceration. Individuals sleeping on the street are frequently cited for public-nuisance offenses such as public intoxication, panhandling, and trespassing, and as they are arrested and cycle in and out of jail, homelessness becomes a law enforcement issue. With no other place to turn, many also spend time in detoxification and sobering centers and emergency rooms. Because follow-up services after release are often lacking, individuals return to the same risks and experience a recurring cycle of negative outcomes. This cycle results in high costs across agencies—shelters, corrections, and health— and community service providers. In December 2016, approximately 6.6 million people in the United States were under the supervision—held in prisons and jails or on probation and parole—of adult correctional systems (BJS, 2018). About 1.9 million were incarcerated in local jails (about 600,000) or prison (about 1.3 million) (BJS, 2018). The cost of running public corrections agencies (jail, prison, parole, and probation) is estimated to be $81 billion per year (Wagner and Rabuy, 2017). Recidivism is a challenge; researchers estimate that 68 percent of people released from state prison were re-arrested within 3 years, 79 percent within 6 years, and 83 percent within 9 years (Alper, Durose, and Markman, 2018). Homelessness and housing instability elevate the risk of recidivism and readmission (Lutze, Rosky, and Hamilton, 2014). Criminal Justice Outcomes and Costs When people are released from jails or prison, many face challenges with housing instability and homelessness (Gouvis Roman and Travis, 2004). They may lack family, friends, and other support systems to return home to; they are followed by their criminal records, which can make it more difficult to find employment and housing; and they often have severe trauma, mental illness, and substance use disorders, placing them at high risk for homelessness (Hunt, 2018). Prior research documents a range of findings on rates of incarceration among homeless population samples. Metraux and Culhane (2006) found that 23 percent of people staying in public shelters in New York City had been incarcerated within the preceding 2 years. Burt and coauthors (1999), using data from the National Survey of Homeless Assistance Providers and Clients, found that 49 percent of clients of homeless programs surveyed had spent 5 or more days in a city or county jail. The National Alliance to End Homelessness estimates that one in five people leaving prison become homeless upon reentry into the community (NAEH, 2016). National data show that almost 50,000 people leave correctional facilities and enter shelters immediately (USICH, 2016). A growing number of studies have measured the impact of PSH on housing stability and criminal justice involvement (usually focused on jail entry and exit). A quasi-experimental New York City FUSE II evaluation studied the impact of supportive housing on a population that had experienced at least four stays in both shelter and jail. The evaluation found 91 percent of participants in supportive housing retained housing after 1 year, and 86 percent retained housing after 2 years. Additionally, participants in supportive housing experienced 19.2 fewer days in jail than the comparison group, a 40-percent reduction (Aidala et al., 2014). A 2002 study tracked the public service use of homeless, mentally ill New York City residents, both those who remained homeless and those who became supportive housing tenants. The study found an average reduction of 82.9 shelter days over 2 years (a 61-percent reduction from pre- intervention) and found an average reduction of 8 prison days and 4 local jail days over 2 years, as compared with a control group (Culhane, Metraux, and Hadley, 2002). A pre-post study conducted in Seattle (Larimer et al., 2009) found a 90-percent reduction in shelter nights and a 32-percent reduction in jail days, whereas another pre-post study in Portland, Maine (Mondello et al., 2007), found a 98-percent reduction in shelter nights and a 62-percent reduction in jails days. Health Outcomes and Costs The literature also documents the link between homelessness and poor health. Almost one-third of people experiencing chronic homelessness have a serious mental illness, and around two-thirds have a primary substance use disorder or other chronic health condition (ONDCP, n.d.). The Affordable Care Act’s state EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 15  option to expand Medicaid to adults at or below 138 percent of the federal poverty level has provided an opportunity for many people experiencing homelessness to enroll in Medicaid. This development has brought more attention to the uneven distribution of spending within Medicaid—just 5 percent of Medicaid beneficiaries account for one-half of total Medicaid expenditures (Mann, 2011). Increasingly, insurers are realizing the value proposition of a more integrated solution for a population that tends to have intensive needs but faces social barriers to care, which results in a cycle of expensive acute care usage. Interventions to interrupt this cycle of super-utilization and serve high-need, high-cost populations are growing. In 2015, the Center for Health Care Strategies identified programs to serve these types of populations in 26 states. This number will grow as health care insurers across the country seek ways to identify their super-utilizer populations and work with other sectors such as housing providers to achieve better outcomes. The literature suggests that supportive housing shows promise in improving health and health care outcomes, but the evidence is mixed. Under the theory of change, individuals stabilize in housing, and supportive services such as case management are provided to better manage chronic physical health conditions such as HIV/AIDS, diabetes, and asthma. This management leads to more appropriate care and better health. In particular, avoidable emergency room visits, hospitalizations, and readmissions are reduced as chronic physical health conditions are better handled. More appropriate care for substance abuse and persistent mental illness, such as more outpatient mental health and substance abuse services, also results in fewer detoxification and psychiatric hospitalizations as behavioral health needs are better treated and managed. Increased health may also reduce nonavoidable emergency room visits and hospitalizations. However, existing evidence is mixed on the impact of supportive housing on health outcomes and has been limited by studies with short followup periods and use of matched comparison groups, not randomly assigned controls. Several studies found the use of emergency rooms, both avoidable and unavoidable visits, decreased with the provision of supportive housing (Martinez and Burt, 2006; Mondello et al., 2007; Sadowski et al., 2009; Levanon Seligson et al., 2013). The literature is not consistent, however. In their pre-post and retrospective cohort studies, Aidala and coauthors (2014) and Kessel and coauthors (2006) found no reductions in emergency department visits for individuals in supportive housing. The literature does find reductions in hospitalizations for psychiatric reasons. Aidala and coauthors (2014) found that supportive housing participants spent one-half as many days hospitalized for psychiatric reasons, compared with a comparison group. Similarly, matched comparison and pre-post studies by Seligson and coauthors (2013), Culhane and coauthors (2002), and Mondello and coauthors (2007) all found reductions in psychiatric hospitalizations for individuals who moved into supportive housing. Culhane and coauthors (2002) found an average 32-percent reduction of inpatient Medicaid claims along with an increase in outpatient Medicaid claims. Cost savings were driven by decreased utilization of the most expensive healthcare services, particularly reductions in hospital visits and inpatient psychiatric services. The literature overall does not suggest reductions in detox visits. Aidala and coauthors (2014) found no effect on detoxification facility days, as did Larimer and coauthors’ (2009) quasi-experimental study, whereas a small pre-post study in Denver (Perlman and Parvensky, 2006) found an 82-percent reduction in detoxification services for participants in supportive housing. Another pre-post study in Portland, Maine (Mondello et al., 2007), found a 22-percent increase in substance use treatment. Expanding PSH using PFS Developing PSH can be challenging; it requires many stakeholders at the table, including housing, corrections, and health agencies, as well as supportive service providers. Further, PSH is like other preventive social programs in that financing can be difficult to marshal because the upfront costs are usually borne by one system although the program’s benefits and cost savings accrue to multiple systems that did not share in the cost of implementation. This makes the program’s costs appear to outweigh the benefits to the implementing system. This is often referred to as a “wrong pockets” problem. In the case of the reentry population, the cost of PSH is carried by the Continuum of Care (CoC), the local planning body responsible for coordinating funding for homelessness programs, but outcomes beyond housing stability such as reductions in recidivism and improvements in health—which are the big-ticket cost offsets—are accrued by corrections and health agencies. This problem is not unique to supportive housing. Although governments fund social programs, public financing does not always allocate funds to the most effective programs. Sometimes this is because of, as noted above, the wrong pockets problem. At other times, barriers exist because programs are expensive to launch, with savings accruing in the future; because benefits are cost-effective only at scale, requiring more upfront capital; or because the benefits are spread, making them hard to measure (Roman, 2015). All these barriers have the potential to erode support for financing even effective programs (Roman, 2015). Government risk aversion and lean budgets are other primary barriers that often limit the implementation of preventive programs. Under traditional public financing, the government assumes all financial risk, paying to implement interventions even when they fail. When budgets are lean, this risk may prevent any funding at all (Galloway, 2015). Further, tightened budgets leave limited opportunities to experiment with innovative prevention strategies, limiting or eliminating opportunities for downstream savings to the government (Sager, 2015). EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 17  PFS is a promising financing mechanism that aims to shift the risks of implementing an evidence-based social program from a government to a private investor and provides an opportunity to increase the scale of evidence-based programs like PSH. By transferring much of the upfront costs and risk of preventive programs to private investors, and by requiring rigorous evaluation to direct how and when the government will pay for outcomes, PFS can give governments more latitude to implement promising prevention strategies and programs (Sager, 2015). Similarly, PFS can circumvent the wrong pockets problem by including private funders that are motivated by program-level outcomes instead of agency- level budgets (Roman, 2015). The use of PFS as a financing mechanism for PSH is still in its infancy, and the field itself is still in the early stages of development. At the time of this report, the Nonprofit Finance Fund which tracks PFS projects, identified 20 PFS active contracts, 1 completed contract, and many other projects at various stages of development across the United States.3 Existing PFS projects encompass a great diversity of stakeholders. Federal, state, county, and city governments have all launched PFS projects. Private and philanthropic investors in PFS projects range from large multinational banks to foundations and individual investors brokered through a firm. Service providers for PFS projects include national nonprofits, local school districts, housing authorities, and smaller local nonprofits. Independent evaluators range from nonprofit research firms to large public universities (Gustafsson-Wright et al., 2015). The 21 existing PFS projects fund a range of programs and target populations. Four of the existing PFS projects fund programs for homeless populations, six projects are focused on early childhood outcomes, and eight projects target populations involved in the criminal justice system; the remaining three focus on improving education, environmental, and health outcomes. The time periods for existing projects range from 3 to 7 years, and investments range from $4 to $30 million. Three current PFS projects are funding PSH interventions in the Commonwealth of Massachusetts; the County of Santa Clara, California; and the City and County of Denver, Colorado. The Massachusetts and Santa Clara projects are serving a chronically homeless population and measuring housing retention as the outcome for success payments. Denver is serving a homeless population with frequent criminal justice involvement and is measuring housing retention for interim success payments and using a randomized experiment to measure the impact on jail stays as the final outcome for success payments. In Denver, the Urban Institute was involved in the feasibility phase, helping to develop the logic model, define the target population, and provide evidence for the assumptions included in the financial model, as well as working with the PFS team to develop the research design. Now that Denver is in the implementation phase, Urban is serving as the independent evaluator (Cunningham et al., 2016). 3 For updates on current PFS projects, see www.payforsuccess.org.  Pay for Success Phases Stewarding a PFS project from development through implementation is a long, resource-intensive, and complex process. Launching a project takes at least a year (Azemati et al., 2013), with many taking multiple years (Commonwealth of Massachusetts Executive Department, 2014). This process has many moving parts, which can be conceived of in three general phases: feasibility analysis, transaction structuring, and outcome evaluation and success payments (City and County of Denver, 2014; HUD, 2015). These phases can overlap during the project’s development, with some tasks in one phase still in development while pieces of the next phase are put into place. FEASIBILITY ANALYSIS During this phase, both institutional and programmatic feasibility are examined through informal and formal conversations and data analysis. Tasks: The first task recognized in most PFS projects is to gauge alignment with the interests and needs of the broad network of stakeholders, including government officials, investors, and providers, that will form the core of the project (Centre for Social Impact Bonds, n.d.). On an institutional level, PFS transactions require government partners to consider their own enthusiasm and commitment—ideally with the support of government budget offices, agency heads, and executive leadership—as well as interest from investors (Liebman and Sellman, 2013). Ideally, PFS priorities will be clearly aligned with existing priorities of executive leaders and based on an identified cost driver at an institutional or population level (Roman et al., 2014). Because these institutional factors vary from place to place and partner to partner, a PFS project that is suitable in one community or for one partner might not be right for another. Next, projects must define feasibility on a program level. Various PFS publications have put forward criteria for choosing an appropriate program, which commonly include considerations of outcomes and evaluation feasibility, financial and cost implications, support of key stakeholders, and prior evidence of effectiveness. Once an intervention is selected, data analysis follows to look at how the target population and expected outcomes will work within the structure of the PFS agreement. This work might transition into structuring, but it is key to lay the groundwork early on; access to data should be confirmed and negotiated during the feasibility phase, and sustainable relationships built to maintain the matched data that are key to identifying the target population and the potential financial value of serving the target population (Third Sector Capital Partners, 2016). Government stakeholders should think through the range of benefits the intervention is forecast to accrue, including monetized and budgetary savings, nonmonetized social benefits, and the feasibility of structuring a schedule of payments to align with them (Liebman and Sellman, 2013). An evaluation methodology may also be considered in this phase. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 19  Partners: All key PFS partners may participate during the feasibility phase. It is especially crucial that the government functions as the initiator, identifying which areas to target and what outcomes to pay for (Roman, Eldridge, and Hawkins, 2015). Investor interest is also key, and projects usually seek to engage both local and national funders, exploring philanthropic, commercial, and other types of funders such as community development financial institutions. Knowledge and financial intermediaries can provide additional expertise to the government on program selection and transaction development during the feasibility phase (Third Sector Capital Partners, 2016). In some projects, service providers and evaluators are also engaged during the feasibility phase to understand service capacity and historical performance, as well as support the program feasibility analysis. Challenges: Attracting private investment sometimes presents a challenge because of the risk involved and experimental nature of PFS projects, but philanthropic capital has been used successfully in some projects to bring private investors to the table (Azemati et al., 2013). Participation of strong service providers with a proven record of success can alleviate these worries, but stakeholders have voiced concerns during the feasibility phase that too few nonprofit service providers can demonstrate rigorous evaluation of their interventions. Service provider capacity can also present a challenge in terms of collecting and reporting data for the PFS evaluation (Sager, 2015). These requirements can be resource- intensive for nonprofits newer to performance measurement and evaluation, and the costs may have to be absorbed by the organization (Czerwinski, 2010), making entry into the PFS field prohibitively costly. Similarly, the high standards mean that few organizations in a given policy area are ready to deliver services for any given PFS project, and those that are ready may not align with the geography or program area of the project (Azemati et al., 2013). Feasibility analysis may also identify cost-benefit challenges. For example, people who frequently interact with police, jail, detox, and emergency care systems are disproportionate drivers of government spending, leading to millions of dollars of spending (City and County of Denver, 2016). The magnitude of spending is not a guarantee, however, that reductions in spending on crisis services will generate enough savings for the programs to pay for themselves; governments may need to navigate political will to use PFS to shift from remedial to preventive services to secure desired outcomes (Azemati et al., 2013). TRANSACTION STRUCTURING The transaction structuring phase brings all parties together to cement the PFS project in a legal contract, often accompanied by several other agreements outlining implementation and governance of the initiative and the parameters for success payments. Tasks: Tasks in the structuring phase primarily include financial modeling, obtaining legislative authority, developing partners, designing the program and evaluation, and legal negotiations. Financial modeling includes developing the schedule of outcomes on which investors will be paid, the expected cash flow of the project, and the timing of investor payments to finance that service provision. The specific financial model and payment schedule is often influenced by the legislative authority underpinning the PFS agreement. As of February 2016, 11 jurisdictions have enacted PFS legislation (Third Sector Capital Partners, 2016). Most of this legislation includes payment schedules to specify how payments will be linked to outcomes (Teicher, Grossman, and Chong, 2016). Establishing a PFS project may require multiple rounds of legislation to establish the government’s ability to enter into a project at all. In Denver, for example, this meant legislation to approve the creation of a funding mechanism, the legislative authority to pay the evaluator (Resolution No. CR16-0016, 2016), and legislation to approve the contract and funding account (Resolution No. CR16-0015, 2016).4 Procurement is another common piece of partner development during the structuring phase. Many PFS projects have issued a formal request for proposals (RFP) processes to select both the service provider partners and evaluators. Once these partners are selected, program and evaluation design can be finalized, often hand in hand. Data analysis may be required that was not completed during the feasibility phase, or new questions may arise during the structuring phase that require additional data analysis. The final program and evaluation design decisions are finalized in legal negotiations, along with the payment schedule. These negotiations also establish governance and management procedures, which are usually operationalized through a committee that includes those who will be involved in the day-to-day implementation of PSH and a governance body that includes leadership from each PFS stakeholder. Negotiations result in the PFS contract between the government and intermediary. Often, the intermediary also holds legal loan agreements with the investors and service contracts with the service providers. The evaluation is also finalized through a contract with either the intermediary or government partner. Some models have also emerged such as a direct partnership between investors and service providers and a full or partial guarantee for private investors. With a relatively small number of projects closed to date, there is not yet evidence of the most effective PFS structures (Blum et al., 2015). Partners: All PFS partners are at the table during the structuring phase. The intermediary often serves as project manager; playing this key role early on can aid in raising and finalizing investor capital and the procurement and negotiation of contracts with other partners (Third Sector Capital Partners, 2016). Evaluation partners can also play an important role during the structuring phase, helping ground decisions in evidence and raising important research design questions, such as how to operationalize targeting criteria to enroll participants and ensure sufficient sample size to prove the effects of the program, which is key to its success (Gillespie et al., forthcoming). 4 Council Bill No. CB14-1017, 2014; City and County of Denver Resolution No. CR16-0016, 2016; and Resolution No. CR16-0015, 2016. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 21  Challenges: This phase is resource- and time-intensive, requiring extensive legal and technical resources to negotiate the complex contracting process between all parties (Sager, 2015; Third Sector Capital Partners, 2016). Other challenges that arise during the structuring phase likely relate to each stakeholder’s motivations and risk tolerance. These characteristics will have a large influence on the final payment schedule, for example, how often investors will require payments and whether those payments will be based on the final impact or other interim metrics of success. CONTRACT IMPLEMENTATION Once a PFS contract closes, the work of implementation and evaluation begins, which includes outcome evaluation and success payments. Tasks: A ramp-up or pilot period can be critical to the overall success of implementation and evaluation (Third Sector Capital Partners, 2016). This period allows stakeholders to test and tweak program assumptions and processes before performance is tied to payments and the pressure increases. It is critical that pilots are implemented to reflect the plan for the project; otherwise, it is difficult to glean lessons learned to inform the launch of the project. Other important tasks on the programmatic side include training on new clinical practices or evidence-based services and relationship building among providers and other stakeholders who will collaborate during implementation (Third Sector Capital Partners, 2016). Government, intermediary, and service provider partners may each have PFS project managers to support referrals, track performance, troubleshoot challenges, and participate regularly through operating and governance bodies (Liebman and Sellman, 2013). The outcome evaluation will begin during implementation. Evaluators have some unique practical tasks for PFS projects (Gillespie et al., 2016). For example, evaluators work during the early stages of implementation to closely monitor program implementation and to raise and navigate any operating concerns that should be solved before they have a negative impact on the evaluation. Based on the payment schedule negotiated in the PFS contract, the evaluator will measure the outcomes on which the end payor(s) will calculate success payments to investors. In some projects, an independent validator will also verify the evaluator’s outcome analyses. Finally, in some projects, the evaluator will also evaluate other outcomes of interest in the PFS evaluation besides those outcomes required for success payments. In some projects, evaluators also conduct an implementation study to capture the processes behind the outcomes. Partners: All stakeholders are at the table during implementation and evaluation. As mentioned earlier, government, intermediary, and service provider partners have important project management roles which should be clearly defined and assigned to dedicated staff positions. During this phase, the evaluator is tasked with implementing the evaluation, both through ongoing project monitoring and analysis of outcomes. Investors’ participation in the implementation phase is largely through the governance structure established in the PFS contract. In some projects, funders with specific issue-area expertise also inform program implementation, particularly by funding a project ramp-up period (Third Sector Capital Partners, 2016). Challenges: One challenge that existing PFS projects have faced during the implementation, evaluation, and success payments phase concerns defining and framing success. PFS success is often framed in terms of a “double bottom line” typical of an impact investment: a positive social impact and a positive financial return yield a successful investment (Global Impact Investing Network, n.d.). However, experience with PFS projects has shown that interpreting these terms of success can be ambiguous, regardless of whether the provisions of the PFS contract were met. Challenges to evaluation methods and findings during this phase, however, can cast doubt on what government is actually paying for in a PFS transaction. When an intervention fails to meet outcome targets and yield financial returns for investors, the PFS transaction may achieve its goal of ensuring that the government pays only for outcomes. The project may also help build the evidence base on what works. Whether to deem such a PFS project a “success” remains an important question to answer (Liebman and Sellman, 2013; Popper, 2015). The HUD-DOJ Pay for Success Permanent Supportive Housing Demonstration HUD-DOJ awarded seven grants to navigate the pay-for-success phases and provide funding for feasibility analyses, transaction structuring, and contract implementation. Grant agreements were completed in October 2016. The intermediaries listed in Exhibit 2 are designated as the grantee and are responsible for managing grant funds and implementing the project. EXHIBIT 2 Pay for Success Permanent Supportive Housing Demonstration Sites Site Intermediary (Grantee) PFS Lifecycle Phase at Demonstration Launch Anchorage and Matanuska-Susitna    United Way of Anchorage   Feasibility Borough, AK Austin/Travis County, TX Lane County, OR Los Angeles County, CA Montgomery County and Prince George’s County, MD Pima County, AZ Rhode Island The Ending Community Homelessness Coalition (ECHO) Third Sector Capital Partners Corporation for Supportive Housing (CSH) American Institutes for Research (AIR) The Sorenson Impact Center at the University of Utah The Rhode Island Coalition for the Homeless Transaction Structuring Feasibility Transaction Structuring Feasibility Feasibility Feasibility EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 23  As noted in Exhibit 2, sites started at varying PFS phases. From October 2016, when grant agreements were completed, to December 2017, sites worked toward moving to the next PFS phase. As of December 2017, two sites—Lane County and Rhode Island—were in the final stages of feasibility analysis. Two sites— the Municipality of Anchorage and Matanuska-Susitna Borough, and Montgomery County and Prince George’s County—requested extensions to the feasibility analysis phase because tasks within the phase were ongoing. Pima County completed the feasibility analysis phase but in January 2018 opted not to pursue transaction structuring. Instead, they are moving ahead with self-funding a PSH project for the reentry population and are considering performance-based contracting. Austin/Travis County continued in the transaction structuring phase. Los Angeles County, which started in the transaction structuring phase, progressed to contract implementation in October 2017. Demonstration Theory of Change A theory of change is an articulation of how a program or intervention is expected to bring out specific outcomes or results. Exhibit 3 describes the theory of change behind using PFS to fund PSH and highlights the expected results. The target population for the Demonstration interacts with several systems, including housing, criminal justice, and physical and behavioral health. These interactions can be tremendously costly to individual sectors and on aggregate, but because the challenge is cross-sectoral, coordination and collaboration among systems can be difficult. At the center of its theory of change is that PFS acts as a mechanism to bring cross-sector partners together to work collaboratively for better outcomes for a vulnerable population and to work strategically across silos. Bringing these sectors together to investigate the feasibility of PFS can increase the use of evidence to inform program development, the sharing of data to understand the target population, and the identification and removal of system barriers to providing housing for homeless reentry populations, and, ultimately, increase funding, and thus supportive housing units. PSH, the intervention, integrates rental assistance with case management and the services of multiple systems to deliver coordinated care. Using a Housing First approach engages individuals and helps get them into housing as quickly as possible, supplying permanent housing without preconditions or barriers and then engages them in services. The provision of these services will produce several expected outcomes, including increases in housing stability, decreases in criminal justice involvement and recidivism, and increases in appropriate health care services, which will lead to improved health outcomes. All these outcomes will lead to reductions in costly stays in shelters, jails or prisons, and the use of crisis health and mental health services. EXHIBIT 3 Theory of Change: Using Pay for Success to Fund Permanent Supportive Housing PFS    PFS outcomes   Intervention Individual-level outcomes Increase housing stability ■ Reduce Cost avoidance, offsets, or savings accrued Avoid costly shelter, jail/prison stays, and use of crisis health Bring together cross-sector partners, including private investors, to investigate the feasibility of PFS, structure the transaction, and develop a PFS contract. Data sharing Housing First Housing subsidy ■ to define the target population Use evidence base ■ to define outcome-based payments Identify and remove system-level barriers ■ increase access to effective services ■ to improve outcomes Increase funding and PSH ■ for the target population Case management services ■ Develop a case plan ■ Facilitate access to benefits ■ Provide referrals ■ Coordinate care and system involvement Coordinate with Community corrections healthy, stable housing unit ■ Decrease shelter use Decrease criminal justice involvement and reentry to jail and prison ■ Decrease arrests ■ Decrease jail and prison days ■ Decrease jail and prison reentry Increase appropriate health care services and improve health ■ Decrease detox visits ■ Decrease avoidable ER and hospital visits ■ Connect to mental and physical health care and substance abuse treatment ■ Decrease the severity of illness ■ Improve mental health ■ Improve physical health ■ Provide rent assistance in a housing unit that is safe, sustainable, functional, and conducive to tenant stability homelessness services ■ Provide a safe, Defining the Target Population, Permanent Supportive Housing, and Pay for Success Each grantee must launch a PFS project to fund at least 100 units of PSH for individuals in the target population. The Demonstration helps grantees meet the needs of their local communities by allowing them some flexibility to define the target population and the model of supportive housing. The NOFA did, however, set grant requirements, which are described below. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 25  HUD-DOJ REQUIREMENTS FOR THE TARGET POPULATION The HUD-DOJ Demonstration focuses on high-need individuals who are experiencing chronic homelessness and who have been, or are currently, involved in the criminal justice system. The Demonstration requires that individuals have the following characteristics to be considered part of the target population. ■ Criminal justice involvement, defined as multiple jail or prison stays within a 3-year period, including at least one in the last year. ■ Homeless history, defined as meeting the definition of chronic homelessness: being homeless for 12 months over the past 3 years or for one or more nights during each year in the past 3 years. ■ “High needs,” defined as having a history of high-cost use of services or significant physical or behavioral health challenges that require high-cost support. HUD-DOJ REQUIREMENTS FOR PERMANENT SUPPORTIVE HOUSING HUD-DOJ laid out specific requirements in the NOFA for PSH. Specifically, the model “must be an evidence-based PSH Intervention(s) that addresses the needs of the target population—permanent affordable housing paired with voluntary supportive services that help individuals remain in housing, live with maximum independence, connect to needed clinical and mainstream services, and facilitate the attainment of their goals and aspirations.” The PSH model should include the following subcomponents. ■ Outreach ■ Quality permanent and affordable housing ■ Accessible transportation and employment opportunities ■ Housing First approach ■ Housing stability services ■ Informed property or landlord management ■ Care management and service coordination ■ Coordination with the criminal justice system (for example, courts, community corrections) HUD-DOJ REQUIREMENTS FOR PFS The NOFA defined the PFS component of the Demonstration with distinctions between “PFS strategies,” “PFS contracting,” and “PFS financing.” Although “strategies” is an umbrella term to define the structure and activities commonly understood as PFS, the NOFA subdivides PFS into two components: (1) the performance-based contracting model, which rewards verified achievement of agreed-upon outcomes by success payments, and (2) the PFS financing structure, which bridges the up-front costs of service delivery with conditional future outcome payments by securing third-party investment. In other words, the NOFA emphasized the fundamental aspect of PFS—governments (or other outcome end payors) pay only if results are achieved and verified through an evaluation—but it acknowledged the distinct practical financial innovation that shifts the up-front capital responsibility from the provider (or government) to a third-party funder or investor (Box 1). In addition, the NOFA provided guidance on the eligible activities the grant could fund, broken into the three phases of a PFS project: feasibility analysis, transaction structuring, and contract implementation. These activities (detailed in Appendix B) help frame the evaluation’s understanding of the anticipated activity scope for each phase. ■ Feasibility analysis includes technical assistance designed to understand and identify the target population (which requires linking and examining data across systems), designing the program model and financing needed through the PFS transaction, and understanding the costs of and outcomes of business as usual. These activities also include efforts to support “the infrastructure required to implement PFS contracts, such as improving data collection and analysis capacity” (HUD, 2015: 13). ■ Transaction structuring includes activities with the following objectives: to provide transaction coordination and support, to raise capital and develop the PFS capital structure, to mediate and facilitate agreement between parties to the transaction, to prepare for contract implementation, and to ensure the contract accounts for modifications informed by the ramp-up phase. Reflecting the overlap between this phase and actual program implementation, eligible activities also include “early contract implementation activities up to the delivery of housing and services to ensure the PFS contract is moving towards successful implementation” (HUD, 2015: 15–19). ■ Contract implementation (outcomes evaluation and success payments) includes the implementation of the terms of the contract and the launching of the PSH program. An independent third-party evaluator monitors outcomes and documents if the metrics are achieved to trigger success payments. During this phase, the terms of the PFS contract are carried out. Homeless assistance providers administer the PSH intervention to the reentry population, the intermediary provides continued oversight and support throughout service delivery, and third- party evaluators track outcomes and success payments. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 27  BOX 1 HUD NOFA Definition of Pay for Success “PFS strategies are typically public-private arrangements that enable a government to test or expand innovative programs while paying only for those that achieve agreed-upon target outcomes. PFS strategies are typically associated with preventative social interventions and are put in action through PFS contracts between a government (or other payor for social outcomes) and a nonprofit social service provider. PFS contracting models involve end payors and service providers who agree that all or some portion of payment will not be paid until an agreed-upon set of outcomes or level of impact has been verified. Such payments for outcomes are known as success payments. Instead of being applied to the direct costs of housing or services, success payments are made based on the degree to which specific milestones agreed upon in advance are achieved, using payment rates also determined in advance. Achievement of outcomes is typically verified by an independent evaluator agreed upon by all parties to a transaction. “Given that verification of outcomes that trigger success payments may take several years, service providers often will not have the resources to self-finance costs of implementing a preventative intervention during a contract period. For this reason, PFS contracting may require PFS financing, sometimes referred to as ‘social impact bonds’ or ‘outcomes financing,’ through which third-party investors provide financial resources necessary to carry out the intervention. Such third-party investment is typically at-risk and return of capital (and any potential return on investment) via the payor is dependent, in whole or in part, on the achievement of outcomes identified in the PFS contract.” Source: U.S. Department of Housing and Urban Development (HUD). 2015. “Pay for Success Permanent Supportive Housing Demonstration NOFA.” https://www.hud.gov/sites/documents/2015-PFSPSHDEMO-NOFA.PDF. PFS TASKS As outlined in the NOFA, HUD and DOJ expected that feasibility analysis would require up to 15 months to complete, transaction structuring would require up to 18 months, and project implementation would require at least several years before an outcome evaluation might trigger success payments. Because each site is unique and has different community contexts, HUD-DOJ intentionally left the NOFA open in several areas (including the method of doing feasibility analysis, transaction structuring, agreed-upon outcome measures) to give the sites maximum flexibility. The tasks outlined in Exhibit 4 are commonly associated with each PFS phase.  EXHIBIT 4 Common Tasks by Pay for Success Phase PFS phase Feasibility analysis Transaction structuring Contract implementation outcomes evaluation success payments PARTNERS Common tasks ■ Educate stakeholders and raise awareness about PFS ■ Attract and sustain political will and navigate political transitions ■ Navigate legal and regulatory questions ■ Identify a shared target population ■ Design a program model and participant referrals pathways ■ Lead data integration and analysis ■ Interpret data for decisionmaking ■ Identify staff capacity to lead project work and authority for decisionmaking ■ Finalize end payors and investors ■ Finalize outcomes and payment benchmarks ■ Update data analysis for new partners or outcomes ■ Identify housing resources and finalize a service model and provider(s) ■ Design an evaluation ■ Build models for budgeting and systems for tracking data ■ Draft and negotiate the contract ■ Fund and implement a pilot phase ■ Refer participants to the program ■ Provide participants with services ■ Develop a program monitoring dashboard ■ Track program performance and make program adjustments as needed ■ Collect data on payment outcomes and other evaluation outcomes ■ Verify outcomes for success payments ■ Facilitate payments to investors as dictated by the PFS contract ■ Communicate project outcomes Exhibit 5 provides an overview of the key actors typically involved in PFS projects. Under the initial PFS framework, governments often function as project initiators, identifying which outcomes to pay for if established metrics are met. Private and philanthropic funders provide the up-front source of capital, assuming both the financial risk and the potential of financial return if the project meets specified outcome metrics. Intermediaries have two common roles, which can be played by the same or separate partners. The financial intermediary provides deal-making expertise for the project, such as facilitating negotiations and assembling requisite funds. The knowledge intermediary provides support for the content of the intervention, such as identifying high-performing programs, managing performance, and conducting ongoing research. Service providers implement the chosen program model for the target population to achieve the agreed-upon outcomes. Independent evaluators assess whether the intervention has met the agreed-upon outcome metrics. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 29  EXHIBIT 5 Pay for Success Actors Source: Urban Institute (2016). Existing PFS projects encompass a great diversity of stakeholders: end payors, including city, county, and state governments; private and philanthropic funders, from foundations and individual investors brokered through a firm to large multinational banks; service providers, from small local nonprofits to local school districts and public housing authorities to national nonprofits; and independent evaluators, from nonprofit research firms to large public universities.  II. Evaluation Questions and Methods Through the national evaluation, HUD and DOJ seek to assess whether PFS is a viable financing model for increasing the scale of PSH to improve outcomes for a high-need, high-cost, homeless, reentry population. The main goal of the evaluation is to learn how the PFS model is implemented in diverse settings with different structures, populations, and community contexts. The evaluation design provides a multidisciplinary, multi-methods approach to “learn as we do.” The evaluation has four objectives. ■ Document progress and processes among the intermediaries, social investors, local governments, services providers, and end payors for the first 5 years of the Demonstration NOFA grant; this includes identifying the successes and challenges each grantee experienced and how they overcame challenges. ■ Document the costs incurred by intermediaries, social investors, local governments, service providers, and end payors as they assess whether a PFS PSH model is feasible; negotiate and structure a transaction to implement an evidence-based intervention and facilitate outcome evaluation and validation before success payments; and implement PFS agreements with a focus on service delivery, quality control and oversight, and performance monitoring and outcome validation. ■ Document the benefits of a PFS PSH model for individuals, populations, and systems to inform subsequent cost-benefit, effectiveness, and impact analyses. ■ Deliver consultation to Demonstration grantees and their independent third-party evaluation teams on the development and collection of baseline outcome data. The overall evaluation is complicated in that each of the seven projects started in a different place— not just in different PFS phases but also in their community’s experience with collaborating across sectors, using data, serving a reentry population, implementing supportive housing, and evaluating outcomes. The evaluation design seeks to document the reality on the ground as each community began its PFS project, describe the project that developed in each site, and, in hindsight, understand how each community’s unique context may have affected the PFS project as it moved through each phase. The national evaluation is not intended to replace the local evaluations, which measure PSH projects’ individual-level impact on participating clients to inform whether private investors should receive success payments from the government end payor. We identified research questions for each of the four research objectives; these are provided below. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 31  Studying Progress and Processes By studying process, the evaluation documents how the PFS model is implemented in diverse settings with different structures, populations, and community contexts. Specifically, the evaluation aims to answer the following questions. ■ What are the primary tasks accomplished in each PFS phase, and how are they accomplished? ■ Who are the primary partners or stakeholders in each PFS phase, and what are their roles and motivations? ■ What are the primary challenges faced in each PFS phase, and how are they mitigated? ■ What milestones or performance metrics in each PFS phase indicate progress and, in hindsight, are viewed as critical to the success of the PFS project? The evaluation uses the following data collection activities and data sources for this objective— ■ In-person site visits and key respondent interviews ■ Ongoing project status calls ■ Project documents, including feasibility reports, local evaluation designs, and program manuals and guidance Studying Costs By studying costs, the evaluation tracks the staff time required of each key partner, as well as other budget expenditures. Specifically, the evaluation aims to answer the following questions. ■ How much time do partners spend developing the PFS PSH project in each phase? How does this vary by site? ■ Outside the HUD-DOJ grant, what other funding is secured or leveraged to implement the PFS PSH project? ■ During each PFS phase, which partners or stakeholders are spending the most time on project development? What level of staff is working on the project? How does this change over time? ■ What are the aggregate costs by PFS partner type? By PFS phase? By site? The evaluation uses the following data collection activities and data sources for this objective. ■ Analysis of HUD-DOJ financial data submitted by grantees ■ Grant work plans and implementation documentation ■ Budgets from project and partner agencies ■ Key respondent interviews Studying Benefits By studying benefits, the evaluation documents the advantages of the PFS model at the system, program, and individual levels. Specifically, it aims to answer the following questions. ■ Throughout the PFS phases, how do PFS partners’ perceptions and interactions evolve beyond “business as usual” in ways that change community-level systems and benefit the target population? ■ In implementation, how do PFS projects produce program-level benefits for the target population, particularly in terms of new supportive housing units and program performance? ■ In implementation, what are the individual-level outcomes for those participating in supportive housing funded by a PFS project, particularly housing, criminal justice, and health? The evaluation uses the following data collection activities and data sources for this objective— ■ Annual partnership survey ■ Key respondent interviews ■ Cross-site project performance dashboard ■ Grantee performance reports and project outcome reports Providing Consultation on Data and Local Evaluation By providing consultation to the grantees, the evaluation team seeks to increase the quality and completeness of grantee data and build a stronger evidence base on the impact of supportive housing for the target population. Specifically, the evaluation aims to answer the following questions. ■ What are the baseline outcome data for each Demonstration project, particularly for homelessness, criminal justice, and health? ■ Are the proposed local data collection efforts viable, and how could they be improved? EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 33  ■ Are the local outcome evaluation plans appropriate for the model being studied and the PFS outcomes being measured, and how could they be improved? For this objective, the evaluation team is doing the following: ■ Providing guidance on recommended baseline outcome data ■ Convening grantees and local evaluation partners to discuss the role of PFS evaluation partners and practical considerations for PFS evaluations ■ Developing a consultation plan for local data collection and evaluation efforts ■ Reviewing local evaluation plans to provide feedback and recommendations Year 1 Data Collection In the first year of the evaluation, data collection focused on understanding each community’s progress and their plans for advancing the project. After the launch of the evaluation in January 2017, the evaluation team identified site liaisons for each of the seven sites. Data collection included monthly calls with site partners from June to December 2017, key informant interviews conducted on two-day site visits in September and October 2017, a data self-assessment completed in September 2017, and documents received from each site in 2017. This report includes data collected during the 7-month period, from June 2017 to December 2017. The sites were awarded grants in June 2016, and HUD-DOJ completed grant agreements in October 2016. At the time of our data collection, sites had been working on their projects for 14 months. Data collection activities are described in more detail in the following. ■ Monthly communication. The evaluator’s site liaison teams communicated with their assigned grantees each month through a call scheduled for this purpose or by joining the grantee team’s meetings. During these calls and meetings, site teams collected data on grantees’ progress and movement through the PFS phases. The teams documented observations and knowledge in a monthly call log. ■ Key informant interviews. Each site team conducted a 2-day site visit to meet with key local actors. Interview respondents included staff from the intermediary, government end payors (potential and secured), investors, and evaluators. The interviews were customized based on each respondent and focused on the following topics: (1) the primary tasks in each PFS phase, (2) milestones achieved, (3) the challenges faced and how they were mitigated, and (4) the costs and benefits of the PFS project. ■ Data self-assessment. Each grantee conducted a self-assessment of their data use. The assessment included questions about how the PFS project was planning to use data (for example, feasibility analysis, outcome measures, implementation), who owned the data, what types of data they were accessing (for example, homelessness, criminal justice, health), and challenges with accessing data and data linking. ■ Documents. We reviewed feasibility reports and site evaluation reports submitted to HUD and other project materials that described the program and PFS development at the sites. Based on this ongoing qualitative data collection, this report examines the early experiences and challenges among the seven communities engaged in the Demonstration. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 35  III. Grantees’ Early Progress, Challenges, and Metrics In the initial evaluation period (June 2016 to December 2017), data collection and the summary of grantees’ early progress, challenges, and metrics focused on the process and implementation of each project in its specific PFS phase and on understanding local data use across the PFS phases in each site. All but two of the sites (Los Angeles County and Austin/Travis County) spent the initial grant period working on the feasibility analysis phase of their projects. For sites that had progressed beyond this phase, we also collected information on their earlier experiences in feasibility analysis. The summary of grantees’ early progress, challenges, and metrics below is organized by the three basic tasks of the feasibility analysis phase: ■ Establishing PFS partnerships for PSH ■ Conducting feasibility analyses, including identifying the target population and its costs ■ Establishing success metrics and evaluation Within this section, we discuss the basic tasks and accomplishments of the sites and the challenges they encountered. We then discuss other implementation issues sites raised as barriers to or facilitators of progress on their PFS projects. Section IV includes a discussion of the benefits and costs of using the PFS process to implement PSH. A profile of each site—including information on local criminal justice, homelessness, and health reforms efforts; project partners; project milestones; target population; anticipated benefits; challenges; and next steps—is provided in Appendix A. As most sites were still finishing their feasibility analyses and formally and informally negotiating with partners at the end of 2017, this report generally does not name sites or identify stakeholders when discussing issues that might be sensitive to those parties. Establishing PFS Partnerships for PSH Many of the HUD-DOJ sites spent the initial grant period solidifying partners for their PFS projects. Commonly, six primary actors are central to the development and implementation of a PFS project: knowledge intermediaries, financial intermediaries, local and state governments, private investors, service providers, and independent evaluators, as shown earlier in Exhibit 5. In the following, we discuss key partnership themes from the Demonstration so far. Intermediaries. HUD-DOJ designed the Demonstration so that the organization acting as the PFS intermediary would be the direct grantee for the HUD-DOJ Demonstration, rather than service providers or the government entities who are typically the end payors. Feasibility analyses are the responsibility of the intermediary, as is working with all partners to structure the transaction. The intermediaries’ feasibility recommendations must often be approved or accepted by government partners. This approval is particularly important among government partners that will eventually be end payors, meaning the unit of government that will make success payments based on project outcomes. Developing and reviewing the feasibility report and recommendations is generally done through a steering or executive committee composed of interested parties across the criminal justice, housing, and health sectors of government, as well as representatives from service organizations that work in these domains. This structure requires close collaboration between the intermediary and the relevant government agencies. HUD’s funding announcement stipulated that the intermediary is responsible for assessing the feasibility of a PFS project and/or structuring a transaction and/or overseeing implementation of a PFS project. These activities may include partnership building, capital-raising activities, program design for the target population at the Demonstration Site, managing contracts with service providers, making Success Payments on behalf of the government entity or other end payors, and managing third-party evaluations (Section I.A.1.iv.(a)). Some sites’ intermediaries have managed previous PFS projects. Examples include Third Sector Capital Partners and the Corporation for Supportive Housing (CSH). Other partners in these sites reported that such intermediaries’ experience and deep knowledge of PFS were critical for project management and organization. Partners in Lane County said it was important to have an intermediary that understood PFS feasibility analyses as well as budgeting and contracting processes. Other sites’ intermediary organizations do not have prior PFS experience. In such cases, the NOFA required grantees to have as a project partner an “intermediary adviser” with PFS experience. During the initial project period, strong intermediary advisers played an important role in helping some sites progress through the feasibility analysis phase. For example, the United Way of Anchorage—the PFS intermediary and grantee in Anchorage and Matanuska-Susitna Borough—received support and guidance from an intermediary adviser with prior experience implementing a PFS supportive housing project, the United Way of Massachusetts. Social Finance, an organization with considerable PFS experience, supported the EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 37  Rhode Island Coalition for the Homeless (RICH) grantee as an intermediary adviser by helping navigate the feasibility analysis phase despite leadership changes in the grantee organization. Government partners. Government stakeholders can play different roles in a PFS project. In early phases, governments’ main roles are to support partner engagement, provide access to data for feasibility analyses, champion the project, and help the intermediary in its role as convener. Government involvement in the Demonstration projects involves different sectors of government, including housing, human services, and criminal justice, and often spans different levels of governments, including city, county, and state governments. One aspect of the theory of change behind the Demonstration is that PFS will help government agencies collaborate in addressing the challenges posed by frequent users of all these systems. This collaboration among the relevant government entities is facilitated by the intermediary organization. Ultimately, the costs of not addressing the needs of the target population fall to government, and, therefore, governments are generally needed as an end payor for success payments if success outcomes are achieved in implementation. Government end payors may or may not be the same branch, agency, or level of government that participated in earlier PFS phases. For example, a representative from the mayor’s office in Anchorage was deeply involved in the feasibility analysis phase, although the municipal government is not expected to be the primary end payor. Participating partners by site and phase. The establishment of a PFS project requires several key partner roles, including an intermediary, investors, end payors (who are generally government partners), service providers, and an evaluator. As a site moves through the PFS phases, the involvement of different partners changes. Exhibit 6 shows which roles were filled in the early period. Because most sites had not yet completed the feasibility analysis phase by the end of 2017, investors, end payors, and service providers had not yet been definitively identified. Most sites without completed transactions had also not definitively identified a local evaluation partner, except for Montgomery County and Prince George’s County, Maryland. In Los Angeles County, the Demonstration grant began in the transaction structuring phase; in 2017, the transaction was complete, project implementation began, and partners were identified for all roles. EXHIBIT 6 Partner Roles Identified by Permanent Supportive Housing Demonstration Site, through 2017 Intermediary Intermediary    Adviser   Investors ✓✓ ✓ End Service Payor Provider Evaluator Sites in Feasibility Analysis Phase Municipality of Anchorage and Matanuska-Susitna Borough, AK Pima County, AZ Montgomery County and Prince Georges County, MD Lane County, OR Rhode Island Sites in Transaction Structuring Phase Austin/Travis County, TX Sites in PFS Contract Implementation Phase Los Angeles County, CA ✓✓✓ ✓✓ ✓ ✓✓✓ ✓✓✓✓✓✓ Note: In Pima County, Arizona, the feasibility analysis was completed, but the county opted to pursue a Permanent Supportive Housing project without Pay for Success financing and therefore not to pursue transaction structuring. Sources: Grant applications, 2017 site visits, and monthly site calls. Partnership Challenges The necessity of including a broad array and a great number of partners in the PFS project can come with challenges. Within the Demonstration, sites faced several challenges, including recruiting, education, and retaining partners and reaching consensus among them. Stakeholder education about PFS. Across the sites, partners reported that much of the work in the initial grant period included educating stakeholders about the PFS model. Throughout the feasibility phase, many participating stakeholders remained unclear about important aspects of PFS. In at least three sites, intermediaries reported that ongoing education about PFS was required to address government partners’ struggle to understand why PFS would be a better funding model than directly paying for services. Partners reported that some sites needed education to reach a shared understanding about how a PFS deal is structured; that is, the way information and funding is intended to flow among government end payors, the intermediary, service providers, and investors. Site leadership roles. A PFS project requires close collaboration between the intermediary and government partners. Absent PFS, governments typically stipulate the services provided to a target EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 39  population and directly fund those services. As needed or available, governments can apply for any relevant state or federal grants. Similarly, in the Demonstration, the government partners at most sites are also expected to serve as end payors and therefore serve as the ultimate decisionmakers on initiation of the PFS project. But they are not the direct grantees of the current project and may formally be subgrantees to the intermediary organization that leads the work in the feasibility analysis and transaction structuring phases. In at least one site, this structure may have led to conflict about roles and about the leadership of the Demonstration. Motives for PFS engagement. Across the Demonstration sites, partners expressed different motives for initiating or joining the PFS effort in their communities. For example, some stakeholders expressed interest in using PFS to change business as usual, or the way government usually procures services, to improve the effectiveness of those services. In several sites, government partners were engaged in the PFS effort out of a motivation to secure more funding for supportive housing, driven by the scarcity of existing financial resources. These partners were often entirely new to the PFS model. Some partners seemed to expect PFS to serve as a source of new funding, without a clear expectation that PFS investors’ funds would be repaid by the government upon successful outcomes. However, other sites were motivated by the HUD-DOJ grants providing up to $1.3 million in funding for success payments, which ultimately pay for the services that are successful. Finally, in other sites, key partners had already been interested in finding a way to bring PFS to their community, and the HUD-DOJ grant opportunity motivated them to explore PSH as a focus. Likewise, some government leaders had previously expressed support for bringing in new public-private partnerships and using innovative financing models like PFS. Some stakeholders also hoped that the involvement of third-party private investors would increase community support for interventions with the PSH target population. The role of risk in PFS. A primary benefit of PFS is often thought to be the ability to shift the risk of paying for a service to third-party investors until it is proven successful. However, few stakeholders expressed this as their primary motivation for engagement in the project. Moreover, in some sites, PFS was discussed as a general approach to improving effectiveness through financing, either through PFS transactions with up-front funding by third-party investors or through other financial arrangements that do not shift risk, such as the addition of outcomes-based bonuses to more traditional service contracts. Stakeholders at some sites did not think that funding supportive housing had much risk. That is, they were convinced that supportive housing for the target population was more effective than the more common uncoordinated response to the frequent user population. As a result, stakeholders were committed to the project idea but skeptical of the benefit of including investors through a PFS structure and paying them a financial return for success. Different perspectives on financial feasibility. Different government stakeholders, especially in the role of end payor, varied in their perspectives on financial feasibility. PFS projects are intended to generate better outcomes and value for public dollars. A key question is whether the project is also expected to generate budgetary savings for government end payors. The Demonstration sites varied in whether their partners believed that budgetary savings to government agencies were necessary for a project to be deemed feasible. For example, in Rhode Island, partners considered societal benefits as well as cost savings but were not intending to tie success payments directly to cost savings in agency budgets. Similarly, in Austin/Travis County, partners saw cost savings as important but were interested in societal benefits more broadly. Their potential county and healthcare end payors were interested in potential cost saving from avoided emergency room visits and reduced jail stays, and they were also interested in using the project to improve health and well-being for the target population beyond any cost savings. In Pima County, government stakeholders acknowledged that the county might not see direct cost savings from the project but were focused on potential long-term cost avoidance. In contrast, in another site, some stakeholders believed that a project would be financially feasible only if the savings would offset the costs of the program. In addition, stakeholders in at least one site raised concerns about the political feasibility of paying a return to investors or paying interest payments above the cost of services provided, regardless of the outcomes achieved. This concern highlights another way the PFS model may differ from the government’s business as usual and why additional engagement may be required to secure government participation and commitment to a PFS project. Some stakeholders were not sure about the benefit of PFS if the government was committed to serving the target population regardless. In Pima County, for example, government stakeholders expressed commitment to funding a housing intervention for the target population with or without the PFS project. Because of this, some stakeholders were concerned about the additional complexities and expense of implementing such a project using PFS and the additional costs of financial returns to investors. End payor engagement. Intermediaries at the sites had different perspectives on when to engage government, specifically as end payors. Some intermediaries wanted to first have a completed feasibility analysis to pitch to potential end payors. Others wanted to involve potential end payors earlier; they believed that engaging end payors later, in transaction structuring, might require revising feasibility analyses based on end payors’ specific interests. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 41  Most sites planned to engage more than one potential end payor. In Austin/Travis County, the project was targeting three potential end payors—not all of whom were government agencies—which required an enormous amount of coordination, more so than other PFS projects with which their partners had previously been involved. Viable end payors for health savings. Many sites expressed interest in engaging the health care system as potential end payors. Given the target population’s frequent use of health services, PSH could reduce the use of expensive emergency health services and increase the use of more appropriate preventive services. Indeed, in Austin/Travis County, a collaborative of nongovernmental healthcare providers considered participating as end payors. However, current health care financing can make PFS a difficult model for potential health care partners. Medicaid savings is a benefit of PSH that most Demonstration sites expected to achieve. But because Medicaid is heavily subsidized at the federal level, especially in Affordable Care Act expansion states, it is often less clear whether PSH would produce health savings at the state and local levels and who the appropriate health care end payors would be. In addition, although decreased use of expensive health services might be a positive outcome from the community perspective, from health providers’ perspectives, such a decrease might be a net loss if they operate using a traditional fee-for-service Medicaid model. Because Texas was engaged in an ongoing conversation about moving to bundled services and paying for performance, partners saw Medicaid reform as a driver for PFS. Government budgeting for future success payments. A challenge for potential government end payors in many sites concerned the mechanics of budgeting and allocating funding for future success payments. In Anchorage and Matanuska-Susitna Borough, the Alaska state constitution prohibits binding future legislation or dedicating future funds. In Montgomery County, Maryland, the charter prevents the county from being indebted for more than a year for operating expenses. Other government stakeholders in Lane County and Travis County had similar concerns about the mechanics of future appropriations. For example, in Rhode Island, the governor’s office pushed legislative changes necessary for the state government to serve as a PFS end payor. By the end of 2017, this was still a work in progress.5 Stability and change in project decisionmakers and champions. PFS requires a long-term commitment from government partners to span multiple PFS phases. When key government decisionmakers are expected to change during those phases, the project is challenged. For example, in Montgomery and Prince George’s Counties, term limits and upcoming elections (primaries in June 2018) meant that key 5 For other examples of legislation to allow PFS financing on other projects, see Hawkins et al. 2017.  government decisionmakers would be transitioning, which created some uncertainty in negotiations with the government. At a somewhat lower level of decisionmaking, staff turnover among key champions can also slow projects down. In Pima County, for example, the initial project champions within both Pima County and Sorenson Impact did not lead the work during feasibility analyses. In other sites, continuity of leadership helped them address challenges. In Rhode Island, although the grantee experienced staffing turnover and capacity issues, partners reported that the governor’s office was a consistent driving force for the feasibility analysis. The governor’s office had also played a leadership role in exploring PFS opportunities more broadly before the HUD-DOJ grant. During the feasibility analysis, the governor’s office was the primary decisionmaker, along with input from other government and nongovernment stakeholders, through a working group and a steering committee. Local Data Use in the HUD-DOJ PFS Demonstration The PFS model creates considerable demand for data in all phases of the project. Critical tasks, such as using data to identify the right target population for the intervention and collecting accurate outcome data that will dictate success payments, often elevate data challenges beyond what communities may experience in other types of projects. Applied to a cross-sector project, as in the HUD-DOJ PFS Demonstration—which requires data concerning criminal justice, housing and homelessness, and health care utilization—those PFS data demands are raised significantly. Considerable data are needed to complete each PFS phase. ■ In the feasibility analysis phase, data are needed to explore potential target populations and their size, to identify effective services and providers, to estimate anticipated outcomes, and to model costs and benefits for government agencies that may make payments if outcomes are achieved. ■ In the transaction structuring phase, feasibility analyses might be refined in response to the needs and requirements of different government agencies or potential investors to design evaluation metrics and determine performance thresholds for success payments. ■ In the project implementation phase, real-time data may be necessary to identify potential clients and monitor their progress. Success payments will depend on rigorous outcome and impact evaluation, which will require individual-level data for both participating clients and comparison populations. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 43  In the feasibility analysis and transaction structuring phases, aggregate or de-identified data may suffice for many tasks. In the implementation phase, however, projects will generally need to share and link individual-level data across sectors. This evaluation explored the following questions about data use: Which types of data were sites attempting to use during feasibility analysis to identify a target population and estimate baseline outcomes? Which types of data did sites use or anticipate using during project implementation and outcome evaluation? What challenges were sites experiencing around data? We draw here on the data self-assessment each site was asked to complete in fall 2017. We asked sites which types of data they expected to use for feasibility analyses, for project implementation and monitoring, and for outcome evaluation (for success payments or otherwise). Responses are shown in Exhibit 7. (Some sites still in the feasibility analysis phase declined to speculate on success payments or outcome metrics.) EXHIBIT 7 Data Use by Pay for Success Phase 01234567 Number of Sites Arrests Jail stays Prison stays Healthcare in jail Court data Shelter stays/Homeless services Housing retention/Stability Vulnerability index Homeless service referrals Emergency room visits Ambulance rides Behavioral health services Hospitalization/Inpatient days General health services Feasibility analyses Project implementation Outcome evaluation Note: Bars represent the number of Demonstration sites reporting an intended use of data among seven sites. Source: Sites’ data self-assessments, fall 2017 Local data use across PFS phases. Different project phases involve different types of data, as can be seen in Exhibit 7. In fall 2017, most sites planned to use criminal justice data for all phases, but these data would come from different sources for different phases. For instance, most sites were planning to use both law enforcement and jail data for feasibility analyses and project implementation, but only one site was planning to use law enforcement data to assess outcomes. Most sites were also planning to use housing data for all phases but different types of data at different phases. Most sites were planning to use data on retention in housing as an outcome. But in the absence of prior robust PSH projects for the target population, they were not using housing retention data in their feasibility analyses. Instead, the focus of feasibility analyses was on homeless history and vulnerability as measured by an assessment tool, usually the VI-SPDAT (a combination of the Vulnerability Index and the Service Prioritization Decision Assistance Tool). EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 45 Housing and Health Data Homelessness Data Criminal Justice Data For the health domain of their feasibility analyses, most sites were attempting to obtain and use data on high-cost services such as emergency room and inpatient behavioral health services. Most sites were not necessarily planning to use those data for outcomes, although a few were (and three sites indicated that health outcomes were yet to be determined). Feasibility analyses. For feasibility analyses, most sites were using or were planning to use data from each of the three domains of homelessness, criminal justice, and health. Sites mentioned various data sources managed by different local or state partners. Most data were primarily held by government agencies, including local or state criminal justice agencies and local or state health agencies. Many sites began data analysis during the feasibility analysis phase with a focus on identifying a target population with a history of homelessness and other indicators of vulnerability. Homelessness data are held by the agency operating the jurisdiction’s Homeless Management Information System (HMIS), which can be a government agency or a local nonprofit. When obtaining HMIS data is difficult, homeless data can also be sought from a shelter or another homeless assistance program. Identifying vulnerable populations experiencing homelessness was a natural starting point for communities developing HUD’s required “coordinated entry” to homeless services, which refers to prioritizing people with the greatest needs for any type of assistance, including PSH. Some sites in the Demonstration have experience creating a “by- name list,” a registry that documents people experiencing homelessness and their needs. Sites usually measured vulnerability through an assessment tool such as the VI-SPDAT. Most sites used or were planning to use both law enforcement and incarceration data to understand the anticipated criminal justice involvement and recidivism among the target population, to understand the benefits and cost saving from reducing recidivism. For feasibility analyses, sites most often used data on arrests from local police departments and data concerning jail stays. Most sites were focused on jail reentry and return; Lane County, Oregon, was focused on prison stays, although some individuals serve part of their felony convictions in jails, thus that jail data are also important. Incarceration data often involve using data from multiple sources, including local jails, local sheriffs, or state prison data, usually from state departments of correction. In Anchorage and Matanuska-Susitna Borough, both jail and prison data can be collected from the Alaska Department of Corrections because jails are run by the state in Alaska’s unified correctional system. The municipality of Anchorage also has additional important local data because it runs the local Anchorage Safety Center to temporarily house individuals incapacitated by alcohol or drugs in a public place. Austin/Travis County was also considering using court data in all phases. More generally, criminal justice data are organized differently in different jurisdictions, which depends in part on the local structure of criminal justice. Other sources of data included court data and data on parole and probation. Sites also worked to obtain and use data on high-cost health services, such as emergency room visits or inpatient behavioral health services, for the target population. As discussed in the following, this proved challenging in many communities. In Austin/Travis County, however, health data integration was a strength of the feasibility and structuring phases. The community nonprofit Integrated Care Collaboration, which consists of several hospitals and other member health care providers (CommUnityCare Health Centers, n.d.), had already created a health information exchange that linked data from multiple health care sites across multiple systems in accordance with established laws and policies. The PFS project was able to link data from the Homelessness Management Information System and the criminal justice system with health data from the Integrated Care Collaboration. Outcomes. Los Angeles County’s grant started in the transaction structuring phase, and it finalized target outcomes in a transaction during 2017. The success outcomes in the transaction are 6-month and 12-month housing retention and jail avoidance for 2 years following housing placement (Los Angeles County’s evaluation design is discussed in the following). Some other sites, where outcomes have not yet been finalized, also reported intending to use housing stability as a success outcome. For criminal justice, sites listed several possible recidivism-related outcome measures, including reduced arrests, jail entries, prison and jail stays, or days in jail or prison. Some sites have also indicated an interest in health outcomes, including reduced emergency room visits, psychiatric hospitalizations, and inpatient days. Identifying the Target Population A key goal of the feasibility analysis is the identification and definition of a target population. As many sites remained in this phase, they were still identifying the target population in December 2017. However, indications show that several of these elements will vary across the sites (Exhibit 8). Sites have been targeting different criminal justice populations: people reentering from prison, frequent users of jail, and those being diverted from jail through a special court. Additionally, sites have been defining high needs in varying ways. Some sites seek those who score high on an assessment tool. Other sites intend to target frequent or costly users of emergency, health, and other local systems and services. Homeless history represents the one category of eligibility that has been more consistent across the sites. As of 2017, most sites were planning to use the broadest homeless history allowed by the HUD- DOJ Demonstration: homeless for one or more nights during each year in the past 3 years. Most sites report targeting the most frequent users of services in the different systems; some sites specifically report attempting to target the costliest users of those systems. (Whether and how sites differentiate frequency from costliness in targeting their projects remains to be seen in the feasibility reports or final transactions.) EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 47  EXHIBIT 8 Target Population by Permanent Supportive Housing Demonstration Site Municipality of Anchorage and Matanuska-Susitna Borough, AK Pima County, AZ Los Angeles County, CA Montgomery County and Prince Georges County, MD Lane County, OR Rhode Island Austin/Travis County, TX Criminal justice Frequent users Frequent users Reentry and diversion Reentry Reentry Frequent users Costliest users Homelessness At least 1 day in each of the last 3 years Costliest users At least 1 day in each of the last 3 years At least 1 day in each of the last 3 years Currently homeless Frequent users Costliest users “High needs” Frequent users of EMS and sobriety centers Costliest users of health services Upper 3rd or 4th quadrant of VI-SPDAT, disabling condition Costliest users Highest priority on a modified homeless prioritization tool Frequent users Costliest users Note: Reentry populations would be identified as they return from incarceration or after their release, whereas diversion populations would be identified before either conviction or incarceration and diverted to the PSH intervention as an alternative to the need for incarceration. Sources: Grant applications, 2017 site visits, and monthly site calls. Data Challenges Conducting feasibility analyses requires the use of considerable data across multiple sectors. Sites encountered challenges in accessing and using data around three major issues: data access and automation, data quality and timeliness, and privacy. Exhibit 9 shows the number of sites that reported these challenges with data from the criminal justice, housing and homelessness, and health systems. EXHIBIT 9 Data Challenges by Domain Source: Sites’ data self-assessments, fall 2017. Data access and automation. Most sites’ working groups (or steering or governance committees) included stakeholders representing agencies that own data in each required domain (criminal justice, housing and homelessness, and health). This representation was a factor in each project’s ability to access data, and active partnerships helped ease data access across systems. Partners helped negotiate data- sharing agreements or the execution of Memoranda of Understanding and helped allocate or prioritize staff time for data requests. For example, in Los Angeles County, a long-standing relationship between the intermediary and the jails, through their pilot project, facilitated access to individual-level jail data. In Lane County, the three primary partners were funded as subgrantees (Sponsors, a service provider; Lane County Parole and Probation; and the Lane County housing authority) to support their time and work on the PFS project. In Rhode Island, project partners established a biweekly working group meeting of stakeholders, including organizations with necessary data on the target population and current housing and service inventory. The support of these organizational representatives—including from the HMIS system, Department of Corrections, and Executive Office of Health and Human Services—was critical in helping match data for the feasibility analysis. For sites lacking active stakeholders from each system, access to individual-level data seemed to be a larger challenge. For example, in Anchorage and Matanuska-Susitna Borough, although accessing individual-level data from Medicaid and Corrections was helped by the PFS partnership, the site struggled with homeless data because the coalitions with access to those data were not as engaged. Several projects EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 49  also required additional data from agencies beyond those involved with the project and having partners from each sector helped provide introductions to or leverage relationships with those other agencies. Data quality and timeliness. For some data, the quality concern focused on the lack of appropriate identifiers to link individuals across systems. Identifiers used in one system, such as names or Social Security numbers, might not be available in other systems or might be recorded incorrectly, introducing challenges for data matching. Data quality was also commonly reported as a challenge among sites using data from their local HMIS, which is a community data system used to collect client-level data and data on the provision of housing and services to individuals and families at risk of and experiencing homelessness. Gaps in coverage, different definitions and standards, and the difficulty of capturing individuals experiencing all forms of homelessness contributed to HMIS data quality challenges. HMIS coverage rates, or the share of homeless service providers reporting data to HMIS, vary from community to community and among Demonstration sites. In sites where some shelters do not report to HMIS, projects had to negotiate separate data-sharing agreements to build a full picture of individuals experiencing homelessness in the community. Other sites struggled to align the definitions and data standards for reporting on individuals experiencing homelessness across different systems. For example, in at least one site, the local police department used a different definition of homelessness and captured a different catchment area than the local HMIS system, making data harder to match and compare. Finally, sites reported that local HMIS systems did not necessarily include individuals experiencing all the types of homelessness their projects would be interested in serving. HMIS data can miss individuals who are experiencing homelessness but avoid shelters or services, instead opting for camps and other unsheltered situations. For example, in Anchorage and Matanuska-Susitna Borough, a summer point-in- time count captured over 100 unsheltered persons who were a match for the other systems, but they were not captured in HMIS. In addition, individuals just returning from long periods of incarceration might be underrepresented in HMIS data, given their limited time in the community so far, but they might still be an appropriate target population. For example, in Montgomery County and Prince George’s County, Maryland, the corrections departments identified some individuals as homeless, but they were not found in HMIS because of definitional and eligibility issues. One site was working to identify this population by modifying an existing homeless assessment tool to include relevant criminal justice information. Timeliness of data was a concern at many sites and delayed some projects. Sites faced delays while navigating multiple layers of approvals for data requests or while waiting for staff to have time to work on data requests. Several sites also noted the long lag in Medicaid claim data. Although such delays could be managed during the feasibility analysis phase for analyses based on historical data, sites recognized that challenges with more frequent or regular data pulls could be more problematic during project implementation and outcome evaluation, when service delivery and success payments will depend on real- time data access. Data privacy. In one site, negotiating access to homelessness data held by a nongovernmental entity raised concerns about whether client releases of information covered this use of data. Some sites also reported challenges in negotiating confidentiality issues with criminal justice partners. The most common privacy issues by far, however, were related to access to health data and concerns around the Health Insurance Portability and Accountability Act (HIPAA) compliance. These privacy concerns made it difficult to negotiate access to individual-level data on healthcare utilization and to link these data with other systems to explore how individuals’ needs overlap across sectors. For example, identifying information such as an individual’s name is often protected health information, and sites commonly reported challenges with finding appropriate identifiers to link health data with other systems. For project implementation and outcome evaluation, some sites planned to obtain consent for access to health data from clients who participate in a supportive housing program. Sites recognized, however, that such consent would be hard to obtain for potential comparison groups and, retroactively, for the feasibility analysis tasks at hand. The Demonstration sites tried different strategies to address privacy issues associated with access to health data. One approach was to link data from the criminal justice and homelessness systems first and send the identified merged dataset to the data partner in the local health system. The health system then searched for the individuals of interest in their own data, attached the relevant health data, and deidentified the final dataset to be analyzed by the PFS project. In Rhode Island, partners used a similar matching process but then provided aggregate rather than individual-level data to the PFS project for analysis. While this worked for feasibility analyses, the site recognized that agreements would need to be negotiated to use individual-level data in the outcome evaluation. Other sites worked to connect with locally established data hubs that could receive identified data from multiple sources, merge the data, and provide de-identified data sets to authorized users, such as evaluators or intermediary organizations. For example, in 2017, Prince George’s County, Maryland, was working to leverage its participation in the Data-Driven Justice Initiative, led by the National Association of Counties, which aims to build local integrated data platforms. As mentioned earlier, Austin/Travis County leveraged the community’s existing health information exchange to match data on health services with other local systems. Finally, one site reported exploring a HIPAA waiver for the data and analysis associated with its PFS project. Institutional review boards, authorized by the U.S. Department of Health and Human Services, are tasked with reviewing the data privacy and confidentiality concerns (among other issues) of covered EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 51  research projects. These boards have the authority to review and waive the informed consent requirement for the release of HIPAA-protected data for research purposes under certain circumstances.6 Conducting Feasibility Analyses Feasibility Reports In three sites (Pima County, Rhode Island, and Austin/Travis County), the intermediary organization had completed feasibility reports by the end of 2017. These reports culminate the feasibility analysis phase of the Demonstration and conclude with recommendations about whether a PSH PFS project is feasible. These reports were submitted to the local operating, steering, and executive committees and to HUD. Analyzing project feasibility is understood to be a precursor to engaging in the difficult work of structuring a PFS transaction, but no general template exists for feasibility reports. What different sites’ intermediaries include in their feasibility report varies and can reflect differences in the available data, in stakeholders’ needs and concerns, and other considerations. What was discussed in these feasibility reports is shown in Exhibit 10. Snapshots of these three feasibility reports are included in the site profiles in Appendix A. 6 The IRB must determine that the following three criteria are satisfied: (1) The use or disclosure of protected health information involves no more than a minimal risk to the privacy of individuals. (“Minimal risk” is a technical term meaning approximately everyday risk.) (2) The research could not practicably be conducted without the waiver. (3) The research could not practicably be conducted without access to and use of the protected health information. For more detail concerning these requirements, see “Research,” HHS, last modified June 13, 2018, https://www.hhs.gov/hipaa/for-professionals/special-topics/research/index.html. EXHIBIT 10 Overview of Pay for Success Feasibility Analyses Components Identify and define the problem Describe the current service environment Identify gaps in the availability of PSH Define the target population Identify high cost users Assess service use of the target population Monetize (assign dollar values) to public systems of service utilization Assess financial feasibility Quantify other PFS transaction costs X Conduct a cost-benefit analysis X X X Provide an economic model X X Defining the PSH intervention and assessing its feasibility Describe core components of PSH X X X Conduct a literature review of PSH X X X Describe core components of a care coordination model X X Assess the current service provider landscape and capacity X Identify housing resources X X X Determine program referral pathways Define program outcomes Determine specific success metrics X X Determine other outcomes Determine how success will be measured (evaluation design) Provide recommendations and next steps Describe work to be completed at future phases (transaction structuring) X X X Identify organizations that should be part of a working group X X Identify Investors Identify and define the problem being addressed. Each feasibility report began with an overview of the specific problem being addressed. This could include information on the scarcity of PSH in the community, Austin/Travis Pima Rhode County County Island X X X X X X X X X X X X X XXX Quantify the costs of the intervention X X X Quantify benefit to different public systems EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 53  past efforts to address the problem, or the current service environment. Each report noted a need for PSH but varied in the details provided. Define the target population and its costs. Each site sought to identify high-cost or frequent users but varied in the data used to identify them. Austin/Travis County used jail and health data. Pima County used jail data that included homelessness status. Because of difficulty linking data, Rhode Island identified three high-cost groups based on jail and shelter use, health and shelter use, and jail, health, and shelter use. Because different sites estimated costs on somewhat different bases, they are not comparable. For example, Pima estimated costs per person per year to be $32,500, while Austin/Travis County estimated $101,200 (although that estimate did not seem to be based on a standardized observation window per person) (Exhibit 11). Assess financial feasibility. Financial considerations were a focus of the feasibility reports. The steps for assessing financial feasibility included quantifying the costs and benefits of the intervention to determine if the benefits outweighed the costs. Each site’s report concluded that benefits exceeded costs and deemed the project financially feasible. The estimated per person cost savings to the public ranged from $27,000 to $78,000, but again, these savings estimates may not be comparable. Projects estimated similar jail and shelter cost reductions but varied greatly on health care estimates. Exhibit 11 summarizes the intermediaries’ findings regarding the target populations; the costs from that population to the criminal justice, homelessness, and health systems; and the resulting financial feasibility of a PSH project. EXHIBIT 11 Permanent Supportive Housing Demonstration Sites’ Target Population and Financial Feasibility Define the target population Identify high-cost users Monetize the costs of service utilization to public systems Assess financial feasibility Quantify benefit to different public systems Quantify the costs of the intervention Quantify other PFS transaction costs Conduct a cost-benefit analysis Austin/Travis County, TX 500 high-cost users based on health care and jail costs Average cost of $101,218 per person per year for shelter, jail, emergency room, hospital, and court Expected cost avoidance is $78,158 per person across all systems $28,557 per person per year for housing, services, and other program costs Other costs estimated at $1.6 million over 5 years Total expected savings after repaying investment is $42.8 million over 5 years. Pima County, AZ 560 individuals with two or more bookings and homeless Average cost of $32,538 per person per year across jails, shelters, emergency rooms, hospitals, and behavioral health Expected cost avoidance is $27,378 per person across all systems $17,425 per person per year for housing and services Costs expected to range between $350,000 and $2 million over 5 years Total expected benefit of $20.7M over5 years. Total savings not calculated Rhode Island Limited data linking; three subgroups identified: jail and HMIS; health and HMIS; and health, jail and HMIS Costs not linked; $13,239 in HMIS and DOC costs and $42,710 in Medicaid costs Expected cost avoidance $5,954 in HMIS/DOC savings; $10,724 in Medicaid savings Cost estimated to range from $15,000 to $20,000 per person per year N/A Full benefits/savings not quantified Define the PSH intervention and assess its feasibility. Each report included information about PSH and its evidence base. Each site also provided information on the housing resources needed for the project, although information varied in the level of detail. Only one site (Rhode Island) included information from service providers in its feasibility report. Two sites (Austin/Travis County and Pima County) identified Assertive Community Treatment (ACT) or a variation of ACT as the preferred care coordination model to accompany housing support, with the third site not providing details on a specific program. No site included detailed information on referral pathways or other mechanics of the intervention. The feasibility reports did not establish eligibility criteria for the future PFS target population or describe how the project should identify target individuals for recruitment into PSH during implementation. Define program outcomes. Outcomes are a key consideration for a PFS project because they form the basis for success payments. Austin/Travis County and Rhode Island identified specific outcomes in their EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 55  feasibility reports, including housing stability, jail days, and some health care utilization metrics. Pima County did not include outcomes in their report. No site’s report identified additional outcomes to be tracked, and no site recommended an evaluation design. Provide recommendations and next steps. The feasibility report at each of these three sites deemed a PFS-supported PSH project to be feasible and recommended developing the intervention further and proceeding to transaction structuring. None of these feasibility reports identified investors, although Pima County and Rhode Island included considerations about engaging investors. In Austin/Travis County and Rhode Island, the intermediaries recommended developing the intervention further and proceeding to transaction structuring. In Pima County, the feasibility report suggested that there was a ready target population and a feasible PSH project and concluded that Pima County was “well positioned to structure and implement a PFS pilot project” for PSH with the potential to fully scale the intervention if the pilot is successful. A pilot project would involve identifying specific housing resources, contracting with specific service providers, and assembling individual-level data. The county and steering committee agreed that a PSH project was feasible but elected to continue pursuing such a project by funding it outright, without the PFS funding structure. Establishing Success Metrics and Evaluation The feasibility phase also includes the initial identification of success metrics and evaluation. These activities—which include analyzing baseline outcomes and identifying outcome metrics, engaging a local evaluator, setting success payments, and designing an evaluation to measure impact and validate outcomes—are described in the following. Outcome Measures and Baseline Outcomes Sites planned to measure outcomes across all three domains of criminal justice, homelessness, and health. For success payments, however, most sites were focused on housing and criminal justice outcomes. For housing-related payment outcomes, sites considered reductions in shelter stays and stable retention in permanent housing. Those sites planning to use criminal justice measures as a payment outcome for reducing recidivism were focused on either jail or prison stays. When debating payment outcomes, some sites were grappling with the more limited evidence base for the impact of PSH on criminal justice outcomes. In Los Angeles County, for example, investors felt confident in the housing stability metric because high-quality research exists on the efficacy of supportive housing, and the pilot project had high-quality data that could be used as a benchmark. The criminal justice metric was more difficult to determine. Ultimately, because of uncertainty on the criminal justice metric, 60 percent of Los Angeles County’s success payments were based on the housing metric (housing retention) and 40 percent on the criminal justice metric (arrests). In part because of the difficulty of data access and the limited evidence base, health outcomes were the least commonly anticipated payment measures. Austin/Travis County is an example of one site seriously exploring health-related success measures, given the project’s plans to include the local health district as a potential end payor. Exhibit 7 illustrates the numbers of sites that reported using or planning to use different types of data for feasibility analyses, project implementation, and outcome evaluation. Engagement of Local Evaluators Pay for Success aims to move government procurement and financing toward paying for successful outcomes and impacts achieved, rather than just for services intended to produce those outcomes. As such, a critical element of any PFS project is a rigorous evaluation of those outcomes. The evaluation design and the structure of payments must be in place before a PFS contract can be finalized; input from a local evaluator is therefore advisable before those evaluation terms are built into the contract. Only Los Angeles County, which began its grant in the transaction structuring phase and has moved to project implementation, had formally engaged local evaluators at the end of 2017. Although some sites had provisionally identified likely evaluation partners in their applications for the HUD-DOJ Demonstration, most sites intended to contract with a local evaluator through a competitive request for proposals after the project progressed past the feasibility phase. If engaged early, an independent local evaluator can strengthen a transaction and sometimes push sites toward stronger evaluation designs. In one site, for example, a potential local evaluator was willing to participate only if the site would use a random assignment design. Strong evaluation partners can also help think through evaluation methods and overcome common obstacles or propose innovative solutions. In Los Angeles County, RAND Corporation was brought on as the evaluation partner early in the planning phases, before the HUD-DOJ PFS grant. The evaluation team received funding from the James Irvine Foundation to draft an evaluation plan to share with potential investors. At other sites, however, funding an evaluator early on was a challenge. Most grantees did not plan funding for an evaluation partner until the outcome evaluation was in the implementation phase, although EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 57  partners across sites understood that earlier tasks would benefit from the involvement of an evaluation partner. For example, Lane County wanted to complete a power analysis to understand evaluation feasibility, and Austin/Travis County hoped to design and use the same referral procedures in their pilot project as they would in the PFS project. Both sites were navigating how to do these tasks before contracting with an evaluator. Success Payments and Impact Evaluation HUD and DOJ require participating sites to conduct rigorous impact evaluations. Impact evaluation requires a comparison group—either a control group created through random assignment of potential participants or other quasi-experimental comparison groups. It is therefore important to distinguish paying for participant outcomes relative to some predetermined benchmarks (for example, 6-month stability in housing, 6-month avoidance of emergency room use) as distinct from paying for the program’s impact on participants. Outcome-based payments use other information (for example, from historical samples or other sites) to set outcome benchmarks for the program participants, without reference to an evaluation with comparison groups. But evaluating impact requires a quasi-experimental or experimental design to demonstrate improved participant outcomes in contrast to a strong comparison or control population. The structure of the transaction signed July 2017 in Los Angeles County, where program implementation began in fall 2017, was to base success payments on participant outcomes without reference to a comparison group but then to conduct a supplemental quasi-experimental impact evaluation, discussed in the following. As indicated earlier, success payments are based on two participant outcomes: (1) housing stability at 6 and 12 months, as indicated by continuous lease agreements, and (2) jail avoidance rates over 2 years. Success benchmarks were based on historical Los Angeles County data and New York City data from prior work by the grantee, CSH. Los Angeles County’s evaluation plan. Los Angeles County completed structuring its transaction, finalized its contract, and begun project implementation, including agreeing to a final evaluation design. The Urban Institute (Urban) is tasked with reviewing sites’ proposed evaluation designs and providing HUD an assessment. In Los Angeles County’s PFS program, Just in Reach, eligible participants are identified for recruitment while in jail and then referred to Just in Reach. Just in Reach recruits participants and then places them in temporary bridge housing for up to 3 months, until an individual is leased into a PSH unit. Once placed in PSH, individuals are enrolled in the program for evaluation. CSH’s experience has been that most eligible individuals decline to participate, and so enrolled individuals are expected to be a minority (40 percent, as a rough estimate) of all individuals referred. In Los Angeles County’s evaluation design, the evaluator will construct a comparison group from eligible individuals who were in jail during the program period but who were not enrolled. Propensity score methods (matching or weighting) will be used to control for any differences between the treatment and comparison groups on characteristics available in administrative data, which include demographics, VI- SPDAT, prior use of homeless services, prior arrests, and prior use of mental health, substance use, and general relief services. Outcomes to be evaluated include housing stability and service utilization. Services include those to address physical health, mental health, and alcohol and drug issues; general relief; the Supplemental Nutrition Assistance Program; and bundled services through jail or probation. Urban’s assessment of Los Angeles County’s quasi-experimental design noted selection bias as a potential issue because the treatment group will consist only of individuals who successfully enter PSH, whereas the comparison population will presumably include individuals who were eligible for the program but not identified during the time period. (Los Angeles County is also enrolling individuals in a separate PSH initiative from the same target population, who will presumably be excluded from the analyses.) Whether such group differences can be adequately controlled through statistical methods, such as the use of propensity scores, remains a question difficult to answer definitively. How important such design issues are when concluding whether a program is successful can depend heavily on local circumstance and on a substantive understanding of the population and the enrollment process. For PFS, whether a comparison group provides adequate grounds for determining program success is an issue for negotiation among partners, especially the government end payors and third-party investors who will have a financial stake in the outcomes. In the case of Los Angeles County’s PFS project, the grantee noted that this design was used in a prior Los Angeles County evaluation, and partners agreed to this evaluation design, which was approved by HUD. Planned Evaluations Typically, sites do not finalize their evaluation designs before yet completing a transaction. The most important evaluation design issue is how to create a comparison group equivalent to the PSH group. Although some sites, such as Lane County, Oregon, plan to use random assignment (that is, an experimental design) to create control groups, other sites have been deliberating the advantages and disadvantages of different quasi-experimental designs. In Austin/Travis County, in 2017, both random assignment and quasi-experimental designs were being considered and discussed with possible evaluators. Random assignment of eligible participants to either the “treatment” or “experimental” condition—here, to EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 59  receiving PSH—or to a control condition that does not receive PSH is typically considered the “gold standard” in program evaluation. In simple random assignment designs, everyone has an equivalent chance of obtaining the PSH intervention. (In more complex designs, randomization is conducted within subgroups or strata so that each person in the same stratum has the same chance of being assigned to the intervention, although the different strata are given different priority. For example, a program might decide to prioritize individuals returning directly from prison and randomly enroll 60 percent of them to PSH, whereas they would randomly enroll 40 percent of individuals returning from jail to PSH. Because of randomization, no other characteristics—such as differences in the level of motivation or level of risk or need—should differ systematically between the treatment and control groups. This means that we can have a high degree of confidence that differences in outcomes are because of the intervention per se. Other designs attempt to prevent any confounding of the intervention with other factors through a combination of design elements and statistical procedures but very rarely can then generate the degree of confidence that is possible with randomization. The most obvious nonexperimental design is to compare the outcomes of participants in the intervention with the outcomes of otherwise eligible individuals who did not participate. However, this comparison generally confounds the causal effect of the intervention with any other characteristics that differentiate the groups. For example, if more motivated (versus less motivated) individuals are more likely to have better outcomes and are more likely to participate in the intervention, it is difficult to know whether to attribute differences in outcomes to the differences in motivation or to the intervention. Measurement of the important confounding characteristics is then central to efforts to isolate the causal effects of the intervention through statistical procedures, whether through regression controls, matching designs, or propensity score methods. Other potential designs include the use of comparison groups from similar jurisdictions or from earlier cohorts of individuals before the intervention was available. Each such design has its own challenges or “validity threats,” although their discussion is beyond the scope of the current report. Common Design Challenges Urban has been providing technical assistance to several sites, on request, in thinking through design issues. Here, we summarize some major considerations sites have discussed with us. Many of these issues surfaced in discussions about the feasibility and challenges of random assignment in particular, but these tend to be issues that arise with other designs as well. Is the target population large enough to accommodate both a PSH group and a control or comparison group? Generally, this will double the needed target population. A site’s feasibility analysis may identify an eligible population large enough to mount the PSH housing program (that is, to enroll at least 100 participants, per the NOFA) but not large enough to accommodate a similarly sized control group. This issue arises in all designs involving contemporaneous comparison groups drawn from an eligible population, whether or not the comparison group is created through randomization. Would randomization force a site to provide services to low-priority participants to have a large enough group? Feasibility analyses may suggest that the costliest users have a considerably larger potential impact than less costly users. For example, Rhode Island’s feasibility analysis suggested that baseline per-person Medicaid costs were $68,000, $57,000, and $43,000, respectively, for the top 75, 100, and 150 users in a prior period. Does this suggest limiting the PSH project as narrowly as possible, say to 100 people, rather than expanding to 200 and randomly assigning one-half to PSH? Possibly, but only if the future costliest users can be reliably differentiated from the others in advance, before program enrollment, using data that will be available in real time during program implementation. Supplemental analyses may be necessary to explore this question. This issue arises with randomized designs as well as with any design that intends to include a comparison group of equivalent individuals. Is it ethical to randomly assign some people who would benefit from PSH into the control group? This issue has arisen in many Demonstration sites. Generally, when there are more potential participants than available slots, randomization is both a fair and ethical way to allocate these scarce resources while allowing for rigorous evaluation. But for this to be true requires a clear advance understanding of who is expected to benefit from PSH. More generally, can program eligibility criteria be delineated clearly enough to facilitate randomization? Clear eligibility criteria are needed to identify the pool to be randomized, and these criteria must not be casually overridden. Randomization removes operational staff’s discretion to prioritize on the fly. Thus, clarifying program eligibility criteria may require supplemental feasibility analyses to explore the implications of using alternative eligibility criteria. Sites may also benefit from a pilot study. What enrollment rates are expected for people targeted for PSH? The Demonstration targets a population with considerable behavioral health needs, and many may be reluctant to enter PSH. Client engagement can be a lengthy process, and enrollment rates may be low. Understanding enrollment rates is important for identifying a large-enough population for the program and figures into program feasibility. Low enrollment among people who are recruited (that is, high attrition from recruitment to enrollment) can lead to underestimating the effect of PSH. Standard analyses will treat everyone who is randomized as if they receive PSH, even if they never enter housing; these are called intent-to-treat analyses. Because people who do not participate will accrue no program benefit, including them in analyses leads to considerable underestimates of program effectiveness. But statistical methods now exist to correct for this EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 61  problem and to produce unbiased estimates of the effect of PSH on those who actually participate (that is, the “local average treatment effect,” sometimes referred to as an instrumental variable estimate). These estimates are generally considerably larger than intent-to-treat estimates. When in the recruitment process will random assignment be done? It is critical to identify exactly when randomization should be conducted to maximize system actors’ and eligible participants’ amenability to abide by random assignment. Especially if engaging the target population is difficult and lengthy, it is important to be clear about when randomization should occur. Once people are randomized to PSH, in standard analyses, they are considered part of the intervention group, even if they never enter PSH. The equivalent issues for nonrandomized designs occur exactly when in the process an individual is identified and considered as “participating” in PSH and when in the process a comparison individual is identified. If signed consent to participate is required, when should it be requested? The advantage of obtaining consent before random assignment is that people who refuse are never included in either PSH or the control group, which leads to better estimates of effectiveness. Providers are often reluctant to engage potential participants who might then not be offered PSH. However, obtaining consent after randomization may generate low enrollment rates among people randomized to PSH. Such considerations need to be balanced carefully. Other Issues in Implementing PFS in the Demonstration Sites The evaluation also raised additional issues that were either barriers to or facilitators of progress. These issues, including grant startup issues, challenges with program design, and how the PFS project is implemented in the context of ongoing reforms in the criminal justice and homelessness systems, are discussed in the following. Grant Startup Issues Sites experienced challenges during the Demonstration startup period. After HUD made the initial awards, grantees had to meet several requirements to begin drawing down funding for eligible activities. These requirements included tasks common to federal funding, such as establishing a work plan. However, sites commonly reported technical difficulties during this start-up phase that delayed funding for intermediaries and other partners and altered project timelines for tasks in later phases; these included engaging potential end payors who operated with specific budget cycles and enrolling participants as new housing opened. Partners reported these challenges decreased after the initial grant startup period. Program Design Challenges Program design was a challenge for sites where Housing First was not business as usual and where existing PSH programs were scarce. For example, partners in Pima County and Lane County had experience with transitional housing from their work with prison reentry populations, which focused on helping participants reach self-sufficiency in a short time. In contrast, the PFS projects are intended to target a more vulnerable chronically homeless population for which PSH may be a long-term solution. Sites worked to find appropriate program models, outcomes, and performance benchmarks as they shifted to a Housing First model. Challenges also surfaced in building communities’ housing capacity to expand the implementation of PSH for the PFS project. Whether paving the way to develop new housing or negotiating the dedicated vouchers needed for the project, several communities reported that securing housing subsidies was a challenging task on the horizon. Most sites indicated that they will likely rely on existing supportive housing capacity. Sites varied regarding their plans to use scattered-site or project-based supportive housing. Los Angeles County’s project is using a scattered-site PSH model that leverages bridge or interim housing during the search for permanent units. The services sites most commonly intended to provide were criminal justice coordination and case planning, intensive case management, and referrals or connections to other services. Los Angeles County’s model includes jail discharge planning and coordination and then case management and service referrals. Other Ongoing Reforms Sites were often working within the context of ongoing reforms in the criminal justice, homelessness assistance, and health care fields, which provided both challenges and opportunities for PFS. Homelessness reform efforts. In 2009, Congress passed major reforms of the McKinney-Vento Homeless Assistance programs with the passage of the Homeless Emergency Assistance and Rapid Transition to Housing (HEARTH) Act. The HEARTH Act required jurisdictions to make several key changes to remain competitive for federal resources. First, the HEARTH Act placed an emphasis on Housing First programming, providing an incentive to local jurisdictions to continue to increase investment in PSH. Second, the HEARTH Act changed the focus of performance evaluation from program-level performance to systemwide performance measurement, focusing on several key metrics. The systemwide performance measures were intended to provide incentives for efforts to decrease the average amount of time spent homeless and increase the number of people exiting to permanent housing and remaining housed. Lastly, the HEARTH Act required jurisdictions to develop coordinated processes for prioritization for programming, which became known as coordinated entry. Several sites also have experience creating a by- EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 63  name list, or a registry that documents people experiencing homelessness and their needs. The systemwide performance measures and the requirements around coordinated entry encouraged jurisdictions to prioritize those who have been homeless the longest or are perceived to be the least likely to independently exit and remain housed. HUD’s requirements around coordinated entry facilitated PFS conversations in several sites. Coordinated entry requires communities to develop systemwide standards for the prioritization of people for programming types. This requirement for standards and the movement toward Housing First and systemwide performance measures, as well as HUD guidance on the prioritization of chronically homeless persons, incentivized jurisdictions to ensure that people with the greatest needs receive priority for homelessness assistance, particularly for long-term, service-intense interventions such as PSH. For sites farther along on implementation of coordinated entry, partners reported that data analysis during feasibility naturally began with identifying a target population that had a history of homelessness and other indicators of vulnerability. As noted previously, sites often measured vulnerability through an assessment tool such as the VI-SPDAT. Criminal justice reform efforts. Most sites are also participating in various criminal justice reform efforts, to change the way communities engage with the target population of individuals who frequently interact with the police and jails. In general, reforms are ongoing to reduce the use of expensive criminal justice resources for low-level offenders and individuals primarily driven by behavioral health needs or homelessness. Notably, five sites are participating in the Justice Reinvestment Initiative funded by the Bureau of Justice Assistance (BJA), which is intended to help jurisdictions use data analysis to identify costs and drivers of criminal justice populations and then develop evidence-based and cost-efficient policy options to reduce them. Two sites are participating in the Safety and Justice Challenge Network, funded by the John D. and Catherine T. MacArthur Foundation, which aims to reduce the use of jail and prison through sentencing reform and alternative sanctions.7 Three sites are participating in the Data-Driven Justice Initiative, led by the National Association of Counties, which facilitates data integration to help reduce the reliance on criminal justice and incarceration. Sites participating in the initiative are addressing the data integration challenges often important for PFS projects. 7 The Justice Reinvestment Initiative, funded by the Bureau of Justice Assistance of the U.S. Department of Justice, uses data analysis to identify costs and drivers of criminal justice populations and to develop evidence-based and cost- efficient policy options to reduce them. The Safety and Justice Challenge Network, funded by the John D. and Catherine T. MacArthur Foundation, aims to reduce the use of jail and prison through sentencing reform and alternative sanctions. The Data-Driven Justice Initiative, launched by the White House in 2016 and led by the National Association of Counties, integrates data to drive strategies that reduce the reliance on criminal justice and incarceration for low-level offenders with mental illness.  Exhibit 12 summarizes which Demonstration sites are participating in these ongoing criminal justice reform efforts, and ongoing efforts are discussed in each site profile in Appendix A. EXHIBIT 12 Permanent Supportive Housing Demonstration Sites’ Participation in Other Criminal Justice Reform Efforts Los Angeles, CA Pima County, AZ State of Alaska State of Rhode Island Lane County, OR Prince George’s County and Montgomery County, MD* Travis County, TX a Montgomery County only. Safety and Justice Challenge X X Data-Driven Justice Initiative X X X* Justice Reinvestment Initiative X X X X X The effects of these reform efforts on the Demonstration can be complex. All these efforts complement the drive to provide more supportive housing for the program’s target population. At the same time, such changes in criminal justice policy and practice will change the program’s financial analyses and feasibility considerations. In Alaska, for example, Senate Bill 91 changed sentencing, bail, pretrial services, and parole and probation rules to reduce the number of people in prison and cut correctional costs (Appendix A). By keeping more of the target population in the community, these reforms make the problems posed by the target population more visible and motivate the community to develop a programmatic intervention. Healthcare Reform Efforts. Healthcare reform is another context to understand. As discussed earlier, the federal government has funded Medicaid expansion to improve access to health care for more of the target population. At the same time, expansion creates an additional challenge for end payor discussions in considering which level of government might benefit from the potential positive health outcomes of PSH. The shift to pay for health outcomes rather than health services has facilitated PFS conversations in some sites. As Medicaid pushes communities toward a more performance-based financing model, PFS can be a natural next step. Finally, Medicaid 1115 waivers have been another facilitator for some sites. Under this authority, the federal government may waive certain provisions of the Medicaid law to give states additional flexibility to design and improve their programs. The PFS projects in Anchorage, Los Angeles County, and Austin/Travis County, for example, all explored how to use their states’ 1115 waivers to help fund supportive housing services. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 65  PFS Pilot Studies Pilot studies can improve the chances for successful implementation. During the initial grant period, four sites used or planned a pilot to pave the way for other tasks in program implementation. Anchorage and Matanuska-Susitna Borough and Austin/Travis County reported that they were planning pilot programs to establish referral procedures and service models that would pave the way for PFS implementation. In Los Angeles County, the PFS program was an expansion of a previous pilot project that had been evaluated and adapted to improve program performance. The pilot provided a common understanding that the target population existed in sufficient numbers, identified existing partnerships to help the project succeed, and established a baseline from which to build success measures. Much of the feasibility work was embedded in the initial pilot, including a statement of work, robust policies and procedures for subsidies and delivery of services, and procurement of service providers. Similarly, in Lane County, the service provider began housing people in units designated for the PFS project while partners worked through the feasibility analysis phase. This provided valuable performance data for the feasibility analyses, including information on the target population, service outcomes, and lengths of stay. It also pushed the service provider to build the data infrastructure necessary for tracking program performance when the PFS project moved into implementation. Philanthropic partners helped some sites move through early PFS phases and conduct pilot studies. In Los Angeles County, the Irvine Foundation paid for the feasibility analysis (before the HUD-DOJ grant), while the Conrad N. Hilton Foundation funded a pilot for PFS implementation and is now a primary investor. In Austin/Travis County, the St. David’s Foundation funded a pilot of the PFS program and is a potential PFS investor, along with the Episcopal Health Foundation. In addition, these philanthropies helped bring other partners to the table to move the project forward. For example, the health-focused philanthropies in Austin/Travis County helped the intermediary negotiate with Central Health, a potential end payor for the project. IV. Benefits and Costs of Pay for Success Data collection during the first year of the evaluation focused on the implementation of PFS.8 To understand the Demonstration sites and their progress and challenges, the evaluation relied on monthly updates from the sites and site visits that included interviews with key stakeholders. Although benefits and costs were not the focus of the site visits, the evaluation team did begin to explore this topic during interviews with key respondents. Several interview questions focused on the benefits and costs of financing PSH through PFS. These exploratory questions are discussed further below. Benefits of the PFS Process Proponents of the PFS model frequently reference benefits of the PFS process that go beyond financing a project. As noted earlier, some argue that these extended benefits may offset the up-front time commitment and costs associated with establishing a PFS project. At the time of this writing, most Demonstration sites are still in the feasibility phase, and it is still too early to document these potential benefits. We did, however, ask key respondents if benefits to the PFS model were emerging. Sites consistently reported benefits, including increased attention to the target population, the efficacy of PSH, new and strengthened partnerships, and improved jurisdictional capacity that could extend beyond the PFS project and its target population. Increased Awareness of the Population and Solutions Key informants from several sites, including Anchorage and Matanuska-Susitna Borough, Montgomery County and Prince George’s County, and Rhode Island, reported that the feasibility analysis phase provided broader awareness of the target population, particularly to new stakeholders for whom the target population was not previously a priority, and solidified stakeholder relationships that could serve as building blocks for collaboration. 8 In the coming years, the evaluation team will continue to collect qualitative data, as well as deploy additional study tools, to collect more information on the benefits and costs of PFS. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 67  In Anchorage and Matanuska-Susitna Borough, which includes a large, rural area, informants recognized that jurisdictions lack the social service infrastructure to provide adequate services to the target population and that, currently, the target population is using overcrowded psychiatric and sobering centers as well as jails instead of appropriate services. Additionally, informants acknowledged that PSH could better serve the population, creating cost offsets and decreasing burdens on service systems operating over capacity. In Rhode Island, serving the target population was considered an expensive challenge and, despite agreement on the issue’s importance, development of PSH has been limited. This project facilitated collaboration across government agencies, and the leadership of the governor’s office offered an opportunity to overcome inertia. In Austin/Travis County, the Demonstration created awareness, especially among key stakeholders in local health care organizations, about the importance of PSH in providing housing and services for vulnerable populations, as well as potential cost offsets or savings for other sectors. New and Strengthened Partnerships Respondents in sites across the Demonstration reported that new partners, as well as strengthened and formalized partner relationships, were key benefits of the PFS project. Partners in many sites shared that, although previous attempts at coordination and collaboration had been made, the PFS process solidified relationships. In Montgomery County and Prince George’s County, small-scale and informal coordination to expand PSH was happening before the PFS project, but participation in the feasibility analysis solidified and formalized those processes. Key stakeholders in Los Angeles County reported a similar benefit: an earlier pilot involved many of the same stakeholders, but the PFS project strengthened and formalized those partnerships, particularly between the sheriff’s department and the behavioral health services agencies. Several sites indicated that PFS brought new partners to the table. For example, in Lane County, both the housing authority and the probation and parole agency became new partners with important insights and resources for the project. In sites that moved beyond feasibility analysis, the involvement of investors appeared to be a clear pathway to new partners. In Los Angeles County and Austin/Travis County, the investment of the respective health agencies, United Healthcare and Central Health, in supportive housing was viewed as a victory and a foundation for future investment beyond PFS. Capacity Building and Data Sharing Sites across the Demonstration identified ways it has helped them build capacity on data analysis and sharing. Key shifts included increased attention to performance, such as in Lane County and Rhode Island. In Lane County, the structure of PFS pushed partners to be more analytical with data and to use it for performance management instead of solely for compliance and reporting. In Rhode Island, partners considered how the project could complement broader ongoing efforts to improve the state’s public management, including an outcomes-oriented approach to procurement. Sites also identified the investment in data sharing and systems as an ongoing benefit to the community. Pima County used the Demonstration grant, along with a MacArthur Safety and Justice Challenge grant, as an impetus to build a new Data and Grants Office. Partner agencies also used data accessed during their feasibility analysis to think more strategically about the target population and how to serve them in the future. Partners in Lane County identified building or reinventing databases such as grant management systems as a benefit that will extend beyond the PFS project. Tools and templates created for PFS data analysis and budgeting were also thought to be helpful beyond the life of the project. Finally, the last, and perhaps most important, benefit is increased capacity to provide the target population with services. As a direct result of feasibility analyses and collaborations, many sites, including Anchorage and Matanuska-Susitna Borough, Lane County, and Austin/Travis County, were beginning or were planning to begin programming for the target population or a similar population, even if that work did not directly relate to or lead to the PFS project. And Los Angeles County, which started in the transaction and structuring phase and moved into the PFS contract implementation phase, has increased capacity to serve the target population by 300 supportive housing slots. Reported Costs of the PFS Process A frequently expressed concern about PFS is that it requires a significant time commitment among partners and staff from advisory organizations. As planned, during the first year of the evaluation, the research team did not collect extensive data on costs. In key informant interviews, however, the team did ask about the time stakeholders were dedicating to PFS. Most key informants could not explicitly quantify time commitments, but those who did reported that intermediaries and their advisers spent significant time on the project. At least one site indicated that HUD-DOJ Demonstration funding was insufficient for the work necessary to complete the PFS phases. Several partners across sites identified activities outside the phase-specific PFS tasks as particularly time-consuming. These activities included outreach to, engagement with, and education for potential EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 69  partners outside formal PFS meetings. Educating government staff and local political leaders was also time-consuming but often noted as necessary for moving projects forward. Another factor sites seemed to identify as a “heavy lift” was the potential number of investors or, in some cases, end payors. A few respondents in three sites indicated that PFS overall was a large time investment and that they would recommend those considering PFS look at all other performance-based (and other funding) options before deciding whether to proceed with the PFS financing mechanism. V. Next Steps for the Evaluation In the first year of the evaluation, data collection focused on understanding the PFS projects developing in each site, the processes they navigated as they moved through the PFS phases, and the data they used or planned to use in each phase. This information provides a snapshot of where each community was at the start of the Demonstration, although many began their PFS efforts well before federal funding was available. This information will provide context for the data to be collected by the national evaluation as projects continue to develop. In the second year, the evaluation will collect more data on the costs incurred during each PFS phase and the benefits achieved at the system, program, and individual levels. ■ Documenting the costs of PFS. The research team will begin to document the costs incurred by partners as they move through the PFS phases using financial reports submitted to HUD. Most costs associated with developing a PFS PSH project stem from partner participation. Although all seven sites are receiving substantial funding from HUD-DOJ, the costs of staff from each partner agency go beyond budgeted costs. Thus, simply analyzing expenditures will not provide accurate costs of the PFS project. Therefore, the research team will focus on quantifying costs by collecting data on the time each key partner spends weekly on the project and on other sources of funding and leverage used to support the project outside the HUD-DOJ grant. ■ Documenting the benefits of PFS. Through a survey of the partners, key informant interviews, and summarizing site data, the research team will begin to document the benefits of a PFS PSH model at the system, program, and individual levels. We will learn about both indirect (as a mechanism for forcing systems change) and direct (through positive cross-sector outcomes for enrolled individuals) benefits. The national evaluation is not set up as a cost-benefit analysis, and we will not try to quantify the benefits we measure through qualitative data collection. Instead, we will be able to provide the perspectives of key stakeholders and the benefits they identify. Studying the system, program, and individual benefits of each project, however, will help us describe the value added by the PFS model. This report contributes to research on how communities are launching PFS projects to fund PSH for homeless frequent users of the criminal justice system, describing PFS implementation across seven sites during the Demonstration’s first year. As noted earlier, data collection activities during the next year will focus on the implementation of PFS, highlighting the costs and benefits of using this financing mechanism to expand PSH. The research team will report these results in the evaluation’s year two report. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 71  Appendix A. Site Profiles EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 73  Municipality of Anchorage and Matanuska-Susitna Borough, Alaska United Way of Anchorage Site Summary as of December 2017 As of December 2017, the Anchorage and Matanuska-Susitna Borough, Alaska, site was in the feasibility analysis phase, which was extended to March 2018. Work is headed by a small team from the United Way of Anchorage, the Anchorage Mayor’s Office, and Agnew Beck. The site retrieved data from the state’s Departments of Corrections and Health Services. Despite working closely with the local homeless coalitions, they struggled to obtain data contained in HMIS because of privacy concerns related to what is allowable under the “release of information” form signed by clients. The site team is thinking about planning for future phases: deciding on an evaluator and considering investors and service providers. The Criminal Justice Reform Context in Alaska Alaska is one of six states with a state unified correctional system,9 meaning that the state Department of Corrections (DOC) has jurisdiction over all correctional facilities within the state as well as other correctional agencies, such as probation, parole, pretrial services, and community corrections. This is distinct from how most correctional systems operate. Usually, jails are run locally by the county corrections agency or sheriff’s office. A state unified correctional system means that people who are detained in state facilities can fall under different status categories, including post-arraignment and pretrial, sentenced with jail, and sentenced with prison. Thirteen correctional centers are located across the state. Between 2005 and 2014, Alaska’s incarcerated population grew 27 percent, at a rate almost three times faster than the general population. Including the number of people on community supervision, the 9The six unified correctional systems in the United States include the Alaska Department of Corrections, Connecticut Department of Corrections, Delaware Department of Corrections, Hawaii Department of Corrections, Rhode Island Department of Corrections, and Vermont Department of Corrections.  corrections population in Alaska grew 45 percent. Specifically, the state’s pretrial population grew 81 percent within this time frame. Alaska has high recidivism rates, with nearly two in three people that are released from prison returning to incarceration within 3 years. In terms of national efforts, Alaska joined the Justice Reinvestment Initiative in 2015 and founded the Alaska Criminal Justice Commission to join stakeholders from state and local branches of government to develop strategies to reduce the state’s correctional population. The Pew Charitable Trusts and the Crime and Justice Institute at Community Resources for Justice jointly identified several trends in the State of Alaska’s criminal justice system, including significant growth in the pretrial population, longer prison stays for felony offenses, extended community supervision, and a majority of post-conviction admissions that are low-level, nonviolent offenses. In July 2016, Governor Bill Walker signed into law SB 91 to address the following issues: to reduce the number of people incarcerated by 13 percent by 2024 and save $380 million, with reinvestments of $98.8 million into behavioral health (including substance use disorder and mental health treatment in both the community and correctional facilities), pretrial assessments and supervision, and reentry supports. This law also created the Pretrial Enforcement Division of the Alaska Department of Corrections.10 Most notably, effective on January 1, 2018, this law ended the cash bail system in Alaska and instead requires the use of an assessment to determine the likelihood to appear in court and reoffend to provide a pretrial release (Brooks, 2018). Homelessness Reform Efforts The two jurisdictions in the Demonstration site in Alaska, the city of Anchorage and the bordering Mat Su Territory, are in different CoCs. Anchorage is its own CoC, led by the Anchorage Coalition to End Homelessness, and the Mat Su Territory is part of the Balance of State CoC, led by the Alaska Council on the Homeless. Both CoCs have recently drafted plans to end homelessness with emphasis on housing as the solution to homelessness, improved data infrastructure, and increased systemwide coordination of resources and service provision. The plan from Anchorage’s mayor also outlined a strategy to address long-term homelessness with stabilization metrics that align with the goals of the PFS project. Anchorage and the Balance of State CoCs are already heavily invested in PSH, with nearly 1,000 beds of PSH between the two CoCs. This PSH capacity is concentrated in the city of Anchorage despite the fact that point-in-time counts show that chronic homelessness is more prevalent in the Balance of State CoC. 10See Alaska Criminal Justice Commission (2015). EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 75  Both CoCs have seen increased investment in PSH over time. PSH growth, however, has been concentrated in the Balance of State CoC with a 225-percent increase in PSH capacity from 2010 to 2017. Key Project Partners ■ United Way of Anchorage: Intermediary, grantee ■ United Way of Massachusetts: Adviser to the intermediary ■ Anchorage Mayor’s Office: Adviser to the intermediary ■ Agnew Beck: Feasibility analysis consultant ■ Alaska Department of Corrections: Government partner, data source, potential end payor ■ Alaska Department of Health Services: Government partner, data source, potential end payor ■ Catholic Social Services: Service provider ■ Alaska Housing Finance Corporation ■ RurAL CAP: Service provider ■ Day Break: Service provider Project Milestones ■ June 2017—Hired consultant to lead the feasibility analysis ■ November 2017—Received August 2017 point-in-time data for matching ■ December 2017—Met with a potential investor Target Population Criminal Justice Homelessness   “High Needs” 2+ stays in the past 3 years and    Identified in HMIS for past 3 years at least 1 stay in the last 12 months or flagged by police in encampments Frequent users of health and EMS services and sobering centers Anticipated Benefits Opportunity for improved services for target population. The jurisdictions, particularly Matanuska-Susitna, lack the infrastructure to provide adequate services to the target population. Currently, the target population is using overcrowded psychiatric and sobering centers as well as jails. The project would provide better services for the population while simultaneously decreasing the burden on the service systems currently operating over capacity. Opportunity for offsetting costs. Currently, the city of Anchorage is paying for a large sobering center. If the project can decrease the population using that center, it could be moved to a smaller facility that would cost less to maintain and secure. Challenges Access to HMIS data. The local homeless coalitions operate HMIS in the jurisdictions. Some were concerned about whether the releases of information individuals sign when entering programs and agreeing to have their information put into HMIS allows for the sharing of that information with the project. To mitigate this, the project team used information from a summer point-in-time count to identify the target population. The project team continues to work on gaining access to HMIS data. Staff turnover. The intermediary, United Way of Anchorage, experienced several instances of turnover that stalled progress on the project. During periods in which no leading staff was at the intermediary organization, the mayor’s office took the lead in advancing the project. The project team estimates that these staffing changes delayed them about 1 year in the project. Partner capacity. The state health agency had little capacity to produce reports for the project. Instead of running reports for the PFS team, the project team took advantage of reports run for the agency’s efforts on an 1115 Medicaid waiver. Legal roadblocks. The Alaska constitution prohibits binding future legislatures or earmarking future funds for specific programming. Additionally, concurrent to the United Way of Alaska’s application for this opportunity, a review of PFS opportunities locally by a legislator determined that PFS was not feasible in the state. Community resistance to PSH. Finding housing will be difficult for the project because of zoning in Anchorage and community resistance (not in my backyard, or NIMBY). Loss of support services resources. The Alaska Legislature significantly rolled back many provisions of SB 91 and will not be providing resources as planned for substance disorders use and mental health services. Next Steps ■ Complete feasibility analysis phase: March 2018 ■ Finalize an evaluator: The project team has been in talks with the University of Alaska Institute for Circumpolar Health EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 77  Pima County, Arizona The Sorenson Impact Center at the University of Utah Site Summary as of December 2017 In December 2017, the Project Steering Committee11 reviewed Sorenson Impact Center’s (SI) feasibility study and recommendation to launch a PFS pilot project and decided instead to pursue a self-funded project rather than moving forward with the PFS transaction structuring phase. This decision followed a presentation of the draft feasibility analysis by SI to the Pima County executive management team in October. Despite long-standing interest within Pima County to explore PFS,12 the county and other steering committee stakeholders were not ultimately convinced that a PFS transaction was the optimal financial vehicle for launching a PSH project for this population. Instead, Pima County intends to pursue a self-funded, 2-year pilot project; the target population and services were not yet fully defined at the end of 2017. The City of Tucson’s Housing Department, which acts as the county’s housing authority, committed during the feasibility process to provide preferential placement to program participants and reaffirmed that commitment for the self-funded pilot project. The Criminal Justice Reform Context in Pima County Pima County is the second-most-populous county in Arizona, with more than a million residents. The sheriff’s department runs two jail facilities that serve law enforcement agencies in the metropolitan Tucson area and the rest of the county.13 The county’s jail capacity is 2,377 individuals with an average daily jail population of around 2,100 (Safety and Justice Challenge, 2017). For people who are criminal justice system-involved and homeless, the Tucson City Court and the Pima County Consolidated Justice Court offer a homelessness court. Such specialized courts—for example, mental health courts, domestic violence courts, prostitution courts, and sex offender courts—focus on specific offenses or characteristics of people 11 The Steering Committee included participation from the County Administrator, Chief Deputy County Administrator, City of Tucson City Manager, Tucson Housing Director, Tucson Police Chief, and leadership from the Community Foundation of Southern Arizona. 12 An interest which stakeholders identify remains. 13 The two jail facilities are the Mission/Main Adult Detention Complex, which houses more than 95 percent of the jail detainees, and Ajo, a small misdemeanor jail.  who commit offenses. Tucson’s homelessness court program is for people who are homeless and in residential treatment for at least 45 days. As far back as 2013, Pima County stakeholders began discussing the possibility of addressing the target population through PFS and securing county and philanthropic funding in 2014 to explore its feasibility. SI began its engagement with Pima County in 2015 as part of a grant from The Kresge Foundation. This grant, which ran from November 2015 to March 2016, focused on determining Pima County’s readiness to pursue a PFS transaction to address the PSH target population. The results of this work served as the impetus for the successful HUD-DOJ grant application. In terms of national efforts, Pima County was selected in 2016 to be one of the 20 jurisdictions to participate in the Safety and Justice Challenge Network, funded by the John D. and Catherine T. MacArthur Foundation. This national network was founded to address over-incarceration and reduce the potential misuse and overuse of jails. Specifically, the county is focused on increasing pretrial screening to everyone booked at jail and launched a specialty caseload for high-risk individuals with behavioral health or mental health challenges, broadening weekend and weeknight court options for outstanding warrants, improving data collection, expanding home monitoring for people who are sentenced, and establishing an automated court-date reminder system for people awaiting adjudication. Additionally, Pima County created a 33-member Community Collaborative, with participants from a cross-sector of criminal justice stakeholders, including presiding judges, prosecution, defense, pretrial, law enforcement, service providers, victim advocates, and formerly incarcerated community members. The goal is to reduce the daily jail population 17 percent over 3 years. This is in reaction to several issues within Pima County’s jail, including its nearing capacity. Through a data-driven approach, the county has identified the pretrial jail population as a main driver of overcrowding; it constitutes more than 80 percent of the total jail population. Most people in this population have been jailed on warrants for failure- to-appear charges, misdemeanor charges including shoplifting and DUIs, and low-level felony charges. In addition, although Native Americans constitute 2.4 percent of the county’s population, they constitute 7.7 percent of the pretrial population and 8.0 percent of the population incarcerated on failure-to-appear charges (Safety and Justice Challenge, n.d.1.). Last, Pima County, along with 140 states, cities, and counties, has committed to the Data-Driven Justice Initiative. Participating local governments will use data to develop strategies that divert people with low-level offenses and mental illness from the criminal justice system and that employ alternatives to pretrial incarceration (The White House, 2016). EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 79  Homelessness Reform Efforts The Pima County CoC is led by the Tucson Pima Collaboration to End Homelessness. Pima County released a 10-year plan to end homelessness in 2006 that focused on prevention, maximizing existing resources, transportation, employment, and improved data infrastructure. The plan also called for low- demand housing for people experiencing chronic homelessness, particularly those with mental health disorders. But the plan did not prominently feature PSH as a solution for chronic homelessness. Despite this, investment in PSH was made and PSH capacity in Pima County nearly doubled from 2010 to 2017. Key Project Partners ■ University of Utah, Sorenson Impact Center (SI): Intermediary, grantee ■ Pima County Administrator’s Office: Government partner, subgrantee, end payor ■ Pima County Criminal Justice Reform Unit: Government partner and program implementation partner ■ Pima County Grants and Data Office: Government partner, data source ■ Pima County Sheriff’s Department: Government partner ■ Pima County Attorney’s Office: Government partner ■ Pima County Public Defender: Government partner ■ Pima County Health Department: Government partner ■ Pima County Community Development and Neighborhood Conservation: Government partner ■ Pima County Sullivan Jackson Employment Center: Government partner ■ City of Tucson City Manager: Government partner, possible end payor ■ City of Tucson Housing Department: Government partner, PFS housing voucher assistance ■ City of Tucson Police Department: Government partner ■ Community Foundation for Southern Arizona: Community partner, possible fiscal intermediary and special purpose vehicle manager ■ Federal Reserve Bank of San Francisco: Knowledge partner ■ Tucson/Pima Collaboration to End Homelessness: Community partner Project Milestones ■ September 2016—Grant awarded ■ May 2017—Ratification of collaboration and subaward agreement between Pima County and SI ■ May 2017—Data collection begins for feasibility analysis ■ June 2017—Launch working groups ■ October 2017—Draft feasibility study submitted by SI to Pima County ■ December 2017—Final feasibility study completed by SI ■ December 2017—Decision made to not proceed with PFS transaction structuring with an addendum added to the report Snapshot of the Feasibility Report for Pima County The feasibility report prepared by SI primarily used aggregate-level data provided by Pima County.14 The parties reviewed the figures in September and October 2017, and the results of the economic model were presented in late October. SI’s feasibility report, finalized in December 2017, concluded that Pima County was “well positioned to structure and implement a PFS pilot project” for PSH with the potential to fully scale the intervention if the pilot is successful. The problem being addressed. SI identified 2,033 individuals were as eligible for PSH (scoring above an 8 on VI-SPDAT) using data inputted in HMIS for Pima County between July 1, 2016, and June 30, 2017. This need is offset against a limited supply of PSH units. A 2015 gap analysis report (Tucson Pima Collaboration to End Homelessness, 2015) commissioned by the local CoC estimated that the current housing stock includes only 1,101 units of PSH, only 120 of which—per county officials with whom SI spoke— were currently available to nonveterans. To end homelessness in Pima County, the 2015 report estimated that an additional 1,812 units of PSH would be needed over the next 10 years, with 1,011 required within the first 5 years. The target population. SI used data from the Pima County Sherriff’s Department, Homeless Management Information System (HMIS), Regional Behavioral Health Authority (RBHA), the Pima County Health Department, and the Pima County Behavioral Health Department. Building upon prior analysis conducted as part of the readiness assessment supported by The Kresge Foundation, SI and Pima County identified 560 individuals who had been booked two or more times and whose home address was listed as “General Delivery” or a known homeless shelter. These 560 individuals averaged more than 48 days in jail and 23 nights in a shelter, with more than three emergency room visits. One limitation of the economic analysis conducted during the feasibility assessment was that only aggregate-level health data were available (see Data access, below); because of the data available, SI was unable to link health data to their criminal justice and housing data. SI and Pima County defined the potential target population as individuals who accrued at least two county jail bookings in the last year (all offenses) and were identified as “homeless” by general delivery address or known shelter address at the time of jail booking. The target population was further narrowed according to service utilization history over the year by these metrics: ■ Homelessness history—Costliest users identified by HMIS data ■ High needs—Costliest users of health services in the past 2 years, identified by County Health, Behavioral Health Data, and RBHA data 14 On some measures, proxy data were used, informed by SI’s review of similar interventions. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 81  Financial feasibility. The feasibility report estimated the total status quo costs to public systems averaged $32,538 per person per year. SI further estimated that the project would save $27,378 of these costs per person per year, which are primarily derived from criminal justice costs and some other costs (for example, avoided reimbursements for behavioral health treatment at the Crisis Response Center). Pima County’s share of avoided costs was estimated to be approximately $6,000 per person per year.15 The largest share of the total costs avoided would benefit hospitals and the rest of the health system, which the county does not operate or manage. The total estimated benefit of the project was $20.7 million,16 with Pima County avoiding $4.4 to $4.7 million in potential costs. The estimated cost of the intervention at full scale over five years was $17,425 per person per year for 150 program participants. Feasibility of permanent supportive housing. SI described the typical user of PSH and noted characteristics of the Housing First approach. For the care coordination model, SI provided an overview of both Assertive Community Treatment and Forensic Assertive Community Treatment, as well as describing their evidence base, key components, and differences. This section also included considerations for expanding the intervention, such as determining the number of units needed, where units could be found, and how services would be delivered. Recommendations The primary recommended next step by SI in its feasibility report was for Pima County to conduct a pilot project as a precursor to fully implementing the project. Specifically, the report concluded that “Pima County is well positioned to structure and implement a PFS pilot project to provide Housing First PSH with Assertive Community Treatment or similar wrap-around services for high utilizers of justice, health, and behavioral health systems.” The feasibility report did not recommend a dosage level, service delivery length, or portion of population served by FACT/ACT for the pilot project because of the absence of individual-level data. Sorenson recommended that before launching a pilot, several steps should be taken, including obtaining individual-level data, identifying exact program components and housing resources, engaging investors, and selecting an evaluator. Pima County and the steering committee ultimately decided not to pursue a PFS pilot project and opted instead to continue exploration of a self-funded, 2-year pilot project that will provide housing and a yet- to-be-determined mix of services to the target population. 15 The estimated discounted rate per person, per year was $5,880, with the undiscounted rate at $6,230. The feasibility report initially erroneously reported the estimate as $5,053. 16 Total benefit figures are derived from costs for shelter utilization; criminal justice (legal and jailing); health care (emergency care, paramedic calls, hospital care, and mental health treatment); and environmental remediation.  Benefits Opportunity for collaboration and partnership building. The project benefited from the engagement of stakeholders across the criminal justice, behavioral health, and housing sectors both inside and outside government. This was noted by participants as an important achievement. Capacity building. This PFS grant, along with the earlier Kresge PFS readiness grant and a MacArthur Safety and Justice Challenge grant, gave the county the impetus to build and use a new data and grants office. Partner agencies were able to use the data from the feasibility analysis to think more strategically about the target population and how to serve them in the future. Challenges Partnerships and staff changes. The initial project champions at project conception for both Pima County and SI did not lead the work during feasibility analyses, and the leads who assumed responsibility for the project did not forge as strong a partnership as those initial champions. Some issues related to areas of responsibility, the pace of the PFS process, and data sharing were difficult to resolve. Data access. The feasibility report used aggregate-level data provided by Pima County, but both the intermediary and the county noted that individual-level data would be necessary if the project advanced to transaction structuring. Using aggregate-level data for the report’s estimation of costs and benefits meant that these estimates included wide ranges and did not enable identification of highest-cost users. Sorenson flagged in the report that to launch the recommended PFS pilot project, Pima County would need to finalize efforts to obtain individual-level data for the target population. The county and other steering committee members noted that aggregate-level data were sufficient for the feasibility phase, and the lack of individual-level data had no impact on their decision in relation to the findings of the report. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 83  Los Angeles County, California Corporation for Supportive Housing (CSH) Site Summary as of December 2017 As of December 2017, the Los Angeles County site was 3 months into the implementation of the PFS contract. Twenty-six people had enrolled in the program since it began in October 2017, meaning they had moved into a PFS-financed PSH unit. The site has two monitoring and oversight committees. The operations committee meets monthly to review key performance measures and troubleshoot challenges. The executive committee meets quarterly, and representatives from the investor agencies, the Conrad N. Hilton Foundation, and United Healthcare, are invited to attend. The two success payment metrics, housing retention outcomes and arrests, are being monitored quarterly. Los Angeles County’s Just in Reach Pilot Project Los Angeles County has the highest rate in the United States of unsheltered individuals who experience homelessness, and this rate has continued to increase during recent years (Hunter et al., 2017). Individuals with mental health issues are overrepresented in Los Angeles County jails and in need of alternative services and community-based options. In 2016, 25 percent of the jail population was receiving some level of mental health treatment (Safety and Justice Challenge, n.d.2.). In June 2008, the Los Angeles Sheriff’s Department Community Transition Unit (CTU) was allocated $1.5 million to administer a 24-month jail in-reach Demonstration program. CSH, through the generous support of the Robert Wood Johnson Foundation, provided an additional $250,000 to leverage the public investment. The resulting pilot, Just in Reach (JIR), was designed to focus on the hardest-to-serve population: homeless, repeat offenders (incarcerated three times over 3 years and with three episodes of homelessness in 5 years). In the pilot, which lasted from 2008 to 2010, only 34 percent of participants were rearrested. This rate is remarkable, considering the hard-to-serve population and compared with a 70-percent recidivism rate for the general jail population. Just in Reach 2.0 (JIR 2.0) launched in January 2014 to serve chronically homeless, frequently incarcerated individuals with a permanent housing solution. All clients had a diagnosable substance use disorder, serious mental illness, developmental disability, post-traumatic stress disorder, cognitive impairments resulting from a brain injury or chronic physical illness, or disability, including the co- occurrence of two or more of those conditions. In November 2012, a motion sponsored by Los Angeles County Supervisor Mark Ridley-Thomas to explore expanding JIR 2.0 through a PFS financing structure was approved by the board. In October 2013, the board passed another motion directing the CEO to develop a blueprint for PFS within Los Angeles County. In spring 2015, partners at the Los Angeles Sheriff’s Department submitted a proposal to the board on behalf of JIR 2.0. In July 2015, the board approved the county CEO’s recommendation that JIR 2.0 be prioritized as Los Angeles’ first PFS project. The Criminal Justice Reform Context in Los Angeles County Los Angeles, California, is home to the world’s largest jail system and is critically overcrowded. Managed by the Los Angeles County Sheriff’s Department, its eight facilities have a capacity of 21,811 individuals, but overcrowding remains a serious problem. Los Angeles County was selected in 2015 to be one of 20 jurisdictions to participate in the Safety and Justice Challenge Network, funded by the John D. and Catherine T. MacArthur Foundation. This national network was founded to address over-incarceration and reduce jail misuse and overuse. Specifically, the county is focused on implementing strategies to expand alternatives to jail for low-risk people who are incarcerated and those cycling in and out of medical and mental health facilities, develop and implement a validated risk assessment tool for pretrial defendants, expand Mental Evaluation Teams (METs) to divert people with mental illness to community treatment programs, and increase community-level engagement with the Los Angeles City Attorney’s Neighborhood Justice Program. In addition, the district attorney presented a Mental Health Diversion Plan, which outlined several recommendations and priorities for the county, including a section focused on PSH for people who are justice-involved and have mental health issues. In reaction to this plan, the board of supervisors created an Office of Diversion and Reentry, which was monumental in reform for the county. The county also launched Law Enforcement Assisted Diversion (LEAD) for people with drug offenses, among other efforts (Safety and Justice Challenge, n.d.2). Finally, it is important to note that leadership at the Department of Human Services (DHS) facilitated the creation of housing pathways through Housing for Health in 2012, focused on justice-involved individuals. DHS’s efforts helped promote criminal justice reform within the county. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 85  Two major criminal justice reform efforts in the past decade have been implemented in California, including the Public Safety Realignment in 2011 and Proposition 47 in 2014. The Public Safety Realignment resulted in shifting the responsibility of corrections for people with low-level felonies from the state to local county jails. Within the Public Safety Realignment, people who violated the terms of their release on parole but were not convicted of a new felony are now sent to jail instead of prison. Also, people with low-level offenses serve sentences in county jail or under county probation if they do not have a record of sexual, violent, or serious crimes. These adjustments resulted in an overall decline in the incarcerated population in California for people with low-level offenses but an increase in county jail populations (Lofstrom and Martin, 2015). Proposition 47 alleviated jail overcrowding by reducing penalties associated with certain drug and property offenses, requiring that prosecutors charge them as misdemeanors in most cases. Four preliminary results of this proposition include a decline in new bookings on arrests and warrants for offenses identified in Proposition 47, a decline in the number of convictions for people on these offenses, an increase in people receiving pretrial services, and a decline in the average length of stay for people who are sentenced (Bird et al., 2016). Homelessness Reform Efforts The Los Angeles CoC is led by the Los Angeles Homeless Services Authority (LAHSA). Los Angeles County has had several plans and initiatives to end homelessness, including a targeted chronic homelessness initiative. Despite these efforts, homelessness has continued to increase in Los Angeles. Although homeless-dedicated organizations continue to work on addressing homelessness, including through the development of coordinated entry, other systems of care have created programs to address homelessness. For example, the Flexible Subsidy Pool is administered by the Department of Health and is used to target high system utilizers who also experience homelessness among other specific populations. In late 2016 and early 2017, Los Angeles residents passed two ballot measures, Measure H and Measure HHH, to dedicate significant resources to addressing homelessness in the coming decade. Funds from these ballot measures will be used to increase both emergency and permanent housing capacity in the county. Los Angeles County is heavily invested in PSH, with more PSH bed capacity in the county than temporary (emergency shelter and transitional housing) bed capacity. Despite the chronic homelessness initiative, PSH bed capacity has fluctuated, with some years seeing substantial decreases. From 2010 to 2017, the net growth was only about 1,300 PSH beds. Key Project Partners ■ Corporation for Supportive Housing (CSH): Co-intermediary, grantee, technical assistance provider ■ National Council on Crime and Delinquency: Co-intermediary ■ County Executive Office: End payor ■ Hilton Foundation: Investor ■ UnitedHealthcare: Investor ■ Department of Health Services: Government partner, housing assistance source agency ■ Office of Diversion and Reentry: Government partner, service provider ■ Los Angeles County Sheriff’s Department: Government partner, data source ■ Brilliant Corners: Service provider, housing location and assistance ■ RAND Corporation: Evaluator Project Milestones ■ July 2015—Feasibility analysis completed ■ June 2017—Transaction structuring complete, PFS contract signed ■ October 2017—Enrollment began Target Population Criminal justice    Homelessness   “High needs” Discharge from County facility in 30-120 days, eligible for diversion services through alternative court Homeless at least one night during the year in each of the last 3 years Upper 3rd quadrant on VI-SPDAT, disabling condition Supportive Housing Model Housing. Clients are placed in interim housing while a supportive housing unit is located. Ultimately, clients are placed in scattered-site PSH placements using resources from the Los Angeles Department of Health’s Flexible Housing Subsidy Pool. Services. Jail in-reach and discharge coordination, intensive case management and services, referral to additional clinical services. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 87  Success Metrics Los Angeles County is using two success metrics for payment: housing stability and jail avoidance rates. Most stakeholders felt comfortable with the housing metrics because clear data were available on the success of previous PSH programs with regards to housing. Stakeholders felt less certainty about the criminal justice measures and took time deciding what they should measure and what the success benchmark should be. Because there was more confidence in the housing metric, it accounts for 60 percent of success payments. Anticipated Benefits Expanded capacity to serve the target population. The PFS project increased the capacity to serve the target population by 300 supportive housing slots over the period of the project. New partners. Once investors were identified and confirmed, a key benefit observed was that United Healthcare was a new stakeholder invested in the success of supportive housing in Los Angeles. Strengthened and formalized partnerships. An earlier pilot of the project involved many of the same stakeholders, but the PFS project strengthened and formalized those partnerships. This was identified as being particularly the case in the partnership between the sheriff’s department and the behavioral health services agencies. Challenges Time commitment. Nearly every stakeholder indicated that PFS required a large time investment. Multiple stakeholders indicated that they would recommend that others carefully consider all other performance- based options before deciding to use PFS. Criminal justice data. The sheriff’s department data system is older and requires manual data work to find specific individuals. The difficulties with accessing these data were not restricted to the feasibility phase. The sheriff’s department data will only be reported on quarterly as opposed to the monthly reporting for housing retention, and each time it will require a process within the sheriff’s department. Next Steps ■ Achieving monthly enrollment target: 12 placements ■ Addressing implementation challenges: » Accessing mental health services in the first month following release » Locating viable units in a scattered-site model in an extremely tight rental market ■ Maintaining prioritization of the project amid competing local priorities and initiatives EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 89  Montgomery County and Prince George’s County, Maryland American Institutes for Research (AIR) Site Summary as of December 2017 As of December 2017, the Montgomery County and Prince George’s County, Maryland, site was in the feasibility analysis phase, which was expected to be completed early in 2018. Feasibility work has been headed by the American Institutes for Research, acting as the project intermediary; Montgomery County Department of Health and Human Services; and Prince George’s County Department of Social Services, which are the agencies that oversee the provision of homeless services and were integral to undertaking the project initially. During the project, health, finance, contracts, police, and corrections partners have also become key stakeholders, and these stakeholders reported increased awareness of the needs of the population and increased collaboration as a result of the project. Challenges in the context include the need to deal with each county’s own legislative and contracting requirements, distinct service and housing landscapes, and anticipated changes in political leadership because of upcoming elections of new county executives in each county. During 2017, much of the feasibility analysis was done within separate steering committees in each county, with the intention of eventually seeking project approval in 2018 from a combined executive committee and designing a combined two-county evaluation for success payments. The Criminal Justice Reform Context in Montgomery County and Prince George’s County Montgomery County has made a notable effort to prioritize community residential and nonresidential alternatives to incarceration, including the establishment of a prerelease and reentry services facility (Montgomery County Pre-Release and Reentry Services Division, 2013). Montgomery County has committed to the Data-Driven Justice Initiative along with 140 states, cities, and counties to use data- driven strategies for diverting people with low-level offenses and with mental illness from the criminal justice system and receive alternatives to pretrial incarceration (The White House, 2016). In Prince George’s County, several diversion programs have been institutionalized, including the Bad Check Program, Driving Diversion Program, Marijuana Diversion Program, Theft Diversion Program, and Mediation Program.17 On the national level, Prince George’s County is a recipient of the U.S. Department of Justice’s Smart Reentry: Focus on Evidence-Based Strategies for Successful Reentry from Incarceration to Community Program. This program is focused on developing strategies to address reentry challenges while increasing public safety and reducing recidivism for people who are identified as medium to high risk for recidivating. Maryland became a Justice Reinvestment Initiative site with the formation of a Justice Reinvestment Coordinating Council in 2015. In 2016, Governor Larry Hogan signed into law SB 1005, which made significant changes to sentencing, including reducing maximum penalties for convictions related to drug distribution charges, repealing mandatory minimum sentences for non-violent drug offenses, and establishing graduated sanctions for people who commit parole and probation violations, among others (Dresser, 2016). Homelessness Reform Efforts Montgomery and Prince Georges Counties are separate CoCs. The Montgomery County CoC released a 10-year plan to end homelessness in 2002 focused on prevention and the development of permanent housing for people experiencing homelessness. In 2014, the county released a revised plan to end homelessness focused on coordination within the homeless assistance system and with other public systems, including criminal justice, health, and child welfare. In recent years, the CoC has become focused on systemwide performance and system transformation for reducing disparities in services offered. The Prince Georges County CoC is led by the Department of Social Services. In 2012, the county released a 10-year plan to end homelessness that focused on coordinating the system, including the development of a coordinated entry system and shifting from short-term emergency housing to permanent housing solutions targeted to people experiencing chronic homelessness and high utilizers of other public systems. 17 This had been described as “Diversion Programs” on the Prince George’s County, Maryland, website, at https://www.princegeorgescountymd.gov/1213/Diversion-Programs. But as of October 2018, with a redesign of the website, the document appeared in a search, but the resulting link could not be followed. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 91  Montgomery County is more heavily invested in PSH than Prince George’s County and, from 2010 to 2017, whereas Montgomery County has increased PSH capacity by about 300 beds, Prince Georges County has had a decrease in PSH capacity of about 200 beds. Key Project Partners ■ American Institute for Research: Intermediary, grantee ■ Third Sector Capital Partners: Intermediary adviser ■ Montgomery County Department of Health and Human Services: Government partner, data source ■ Montgomery County Department of Corrections: Government partner, data source ■ Montgomery County Office of Management and Budget: Government partner, potential end payor ■ Prince George’s County Department of Social Services: Government partner, data source ■ Prince George’s County Department of Corrections: Government partner, data source ■ Prince George’s County Office of Management and Budget: Government partner, potential end payor ■ ICF: Evaluator Project Milestones November 2017—Preliminary feasibility results presented to each county’s executive committee Target Population Criminal justice    Homelessness   “High needs” Criminal justice involvement, potentially including arrests, detention, and/or reentry from sentences of incarceration At least 1 or more days in each of last 3 years, or preceding the most recent incarceration History of receiving costly services Anticipated Benefits Increased awareness of a vulnerable and in-need population. The feasibility analysis phase provided a broader awareness of the target population for government stakeholders for whom they were not previously a priority. Strengthened and formalized partnerships. Although smaller and more informal coordination efforts to identify and assist the target population occurred before the PFS project, participation in the PFS feasibility analyses formalized those processes and provided a platform to engage a larger group of stakeholders. This was particularly noted by Montgomery County stakeholders. This process also helped solidify some stakeholder relationships that will serve as building blocks for future collaboration. Challenges Two-county approach. The initial approach was described by multiple interviewees in both counties as an extension of other efforts to do some regional work, which would have included the District of Columbia, but such a joint project never progressed as anticipated. The logistics of establishing PFS agreements in two counties with different contracting requirements and decisionmakers, operating in differing service and housing landscapes, led to the creation of separate steering committees in each county to handle much of the feasibility analyses work through 2017. Local political context. Both counties face changes in leadership with the electoral change in 2018. Both elected county executives are in their last year (under term limits), and both county councils will experience significant turnover because of retirements, members running for other offices, or term limits. The Prince George’s County agencies, particularly the Department of Health and Human Services, exist in a resource-scarce environment, particularly in comparison with Montgomery County. This exacerbates issues over competing priorities and crises that could take priority over this PFS project. Legal hurdles for PFS. County law in both places does not permit a current county executive to bind a future council or executive to future funding. In addition, the Montgomery County charter prohibits binding future county councils or dedicating future funds for noncapital projects. Next Steps as of December 2017 ■ Complete feasibility analyses ■ Develop a draft evaluation plan EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 93  Lane County, Oregon Third Sector Capital Partners Site Summary as of December 2017 As of December 2017, the Lane County, Oregon, site was in the feasibility analysis phase. Despite this, they have been moving forward with engaging end payors and considering other financial structuring activities. Additionally, they have been running a pilot program to gain experience with Housing First, as the model had previously not been adopted in the jurisdiction. This site has been facing some key challenges in securing end payors because of a unique state budgeting process but has indicated that the PFS project has already resulted in increased programming and services available for the target population through strengthened partnerships. This site is the only site in the Demonstration targeting a prison reentry population, rather than a jail reentry population. The Criminal Justice Reform Context in Lane County The State of Oregon Department of Corrections operates 14 prisons across the state, any of which can return individuals to Lane County, Oregon, for their reentry into the community. In terms of national efforts, Oregon joined the Justice Reinvestment Initiative in 2012. Shortly thereafter, the Oregon legislature passed House Bill 3194, which created a Justice Reinvestment Grant Program to use savings from reduced prison growth and apply it to community-based programs to aid individuals reentering the community after being released from prison. These grants can be applied to community-based programs such as work release programs or reentry courts, evidence-based programming for the recidivism reduction. In addition, this bill allows the state to reduce prison sentences for people with theft offenses and recategorize first-time penalties for possession of certain drugs from felony to misdemeanor (DePledge, 2017). The Oregon Criminal Justice Commission (CJC) is responsible for overseeing Justice Reinvestment Initiative work and has played a critical role in facilitating data access and integration for data-driven decision making and outcome-oriented projects in the state. Homelessness Reform Efforts The Lane County CoC is led by the Lane County Department of Health and Human Services. In 2006, the Lane County Board of Supervisors adopted a 10-year plan to end chronic homelessness heavily focused on the development of additional housing for people experiencing chronic homelessness as well as better coordinating the system, improving data infrastructure, and improving prevention efforts. In 2014, Lane County formed the Poverty and Homelessness Board (PHB) to better coordinate efforts on issues of homelessness and poverty. The PHB released an updated strategic plan in 2016. This plan called for an additional 600 units of supportive housing for people experiencing chronic homelessness and increased emergency shelter capacity. From 2010 to 2017, PSH capacity increased fairly steadily from under 200 beds to over 500 beds. Key Project Partners ■ Third Sector Capital Partners: Intermediary, grantee ■ Lane County Probation and Parole: Service provider, referral source ■ Sponsors Partners: Service provider ■ Homes for Good: Housing provider ■ New York University: Evaluator Project Milestones ■ July 2017—Received Department of Corrections data ■ September 2017—Pilot project began Target Population Criminal justice    Homelessness   “High needs” Prison reentry under current supervision with medium to high recidivism risk Currently homeless or leaving transitional housing Not yet defined EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 95  Supportive Housing Model Housing. A mixture of project-based and scattered-site PSH units. The project-based units are for single individuals only. The scattered-site units can be occupied by single individuals or families with children. Services. Services to be determined Anticipated Benefits Increased attention on performance. The structure of PFS pushed partners to be more analytical with data and to use it for performance management instead of solely compliance and reporting. New partners. PFS created opportunities to develop new relationships and engage in new efforts. HACSA and Probation and Parole are relatively new partners, but in addition to PFS, they were able to begin work on a housing site specifically for women on probation. Capacity building. PFS has created an infrastructure that will benefit partner organizations beyond the PFS project, including building or reinventing databases such as grant management systems and creating tools and templates for data analysis and budgeting. Challenges Delays in project start. There were delays early in the project, in finalizing the contract and in work plan approval, that had negative impacts on the project. The delay had a particular impact on the timeline for end payor negotiations. Because Oregon operates in a biannual budget cycle, end payor conversations are now pushed to occur during an off-budget cycle, creating difficulties for the project. Additionally, partners filled the supportive housing units targeted for this PFS project and will now have to wait for turnover. Time commitment. Investing the staff time necessary across PFS partners to keep the project moving forward was challenging, particularly the outreach, engagement, and education to potential partners that occur outside formal PFS meetings and activities. Partner capacity. Even with funding to support their time, some partners do not have the capacity for special projects such as PFS. Partners often relied on executive-level leadership for much of the workload. Next Steps ■ Complete feasibility report ■ Secure end payor(s): Potential end payors are the County Board of Commissioners, Health and Human Services and State Department of Corrections, and the state legislature or the governor’s office. Meetings with potential end payors are scheduled for January. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 97  State of Rhode Island Rhode Island Coalition for the Homeless Site Summary as of December 2017 As of December 2017, the Rhode Island site had completed the feasibility analysis and was awaiting the governor’s approval to move forward with transaction structuring. After a delay caused by grant administration and internal challenges at the grantee, the Rhode Island Coalition for the Homeless, the project advanced rapidly through feasibility. Social Finance, the intermediary, completed the feasibility analysis pro bono. A project oversight committee was also convened and answered key questions on feasibility. The Criminal Justice Reform Context in Rhode Island Rhode Island is one of six states with a unified correctional system18 and was the first to adopt this system in 1956. This unified system means that the state Department of Corrections has jurisdiction over all correctional facilities, as well as probation, parole, pretrial services, and community corrections. This is distinct from how most correctional systems operate, in which jails are run locally by the county corrections agency or the sheriff’s office. A state unified correctional system means that people detained in state facilities can fall under different status categories, including post-arraignment and pretrial, sentenced with jail, and sentenced with prison. The State of Rhode Island has one designated jail facility, the Intake Service Center, a maximum-security facility for males. Other facilities serve people with both short- and long-term sentences (Krauth, 1997). The seven correctional facilities in Rhode Island are called adult correctional institutions. In terms of national efforts, the Rhode Island Department of Corrections (RIDOC) received funds from the Bureau of Justice Assistance in 2012 through the Second Chance Act Statewide Recidivism Reduction Program to implement evidence-based practices to reduce recidivism in corrections, probation, and parole (National Reentry Resource Center, n.d.). Also, the State of Rhode Island has participated as a Justice 18 The six unified correctional systems in the United States include are the Alaska Department of Corrections, the Connecticut Department of Corrections, the Delaware Department of Corrections, the Hawaii Department of Corrections, the Rhode Island Department of Corrections, and the Vermont Department of Corrections.  Reinvestment Initiative site since 2008. The Council of State Governments Justice Center has provided technical assistance in this effort and identified several trends in Rhode Island’s system. Rhode Island has the second highest probation rates in the country—91 percent higher than the national average (Reyes, 2017). Approximately 60 percent of sentenced admissions are attributed to probation violations and one- third of pretrial admissions to alleged probation violations (Council of State Governments Justice Center, n.d.1). Although RIDOC’s budget accounts for 45 percent of the state’s total budget for public safety, only a small portion goes to probation and parole services (Council of State Governments Justice Center, 2015). In 2015, Governor Gina Raimondo issued an executive order to establish the Justice Reinvestment Working Group, which operates as a bipartisan, interbranch effort to address the state’s criminal justice system, and more specifically, to focus on issues related to probation (Council of State Governments Justice Center, 2015). As part of this initiative, in October 2017, the Rhode Island Senate and House, with Governor Raimondo, passed and signed six bills to reform the parole and probation system statewide. This package includes allowing people to appeal a probation violation in court and reducing the maximum sentence for felony assault crimes. Last, the State of Rhode Island, along with 140 states, cities, and counties, has committed to the Data- Driven Justice Initiative. Participating local governments will use data to develop strategies that divert people with low-level offenses and mental illness from the criminal justice system and that employ alternatives to pretrial incarceration (The White House, 2016). Homelessness Reform Efforts Rhode Island has one statewide CoC, which is led by Rhode Island Housing. The State of Rhode Island developed a plan to end homelessness in 2006 that focused on establishing political will and leadership, improving data infrastructure and using data to inform decisionmaking, developing housing resources, and coordinating a systemic response to homelessness. Part of the efforts to build political will and leadership included reactivating a State Interagency Council on Homelessness. In 2012, this interagency council released a strategic plan to end homelessness that aligned with the federal strategic plan developed by the U.S. Interagency Council on Homelessness, Opening Doors. This plan called for the development of more PSH to address chronic homelessness and the development of coordinated entry. Despite this, since the release of this plan in 2012, PSH capacity in Rhode Island has decreased. The state did implement coordinated entry with the Rhode Island Coalition for the Homeless, the entity that also serves as the HMIS lead for the state, operating the coordinated entry system in conjunction with another nonprofit, Crossroads Rhode Island. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 99  Key Project Partners ■ Rhode Island Coalition for the Homeless: Grantee and owner of HMIS data ■ Social Finance: Intermediary ■ Governor’s office: Government partner ■ Department of Human Services: Government partner, data source ■ Office of Housing and Community Development: Government partner ■ Rhode Island Housing: Government partner ■ Harvard Government Performance Lab: Adviser to the state ■ Department of Corrections: Government partner, data source ■ Department of Behavioral Health, Development Disabilities, and Hospitals: Government partner Project Milestones ■ Early 2017—Pro bono feasibility analysis work begun by Social Finance ■ April 2017—Project oversight committee convened ■ October 2017—Feasibility analysis drafted and passed to the governor’s office for a decision on moving forward Snapshot of the Feasibility Report for Rhode Island Social Finance completed the feasibility study and will continue to work with the Rhode Island Coalition for the Homeless and the state of Rhode Island during the transaction and structuring phase. The problem being addressed. In 2016, about 3,000 individuals were homeless in Rhode Island, based on admissions to the shelter system. Although more than 21,000 subsidized housing vouchers are currently available in the state, Social Finance notes a need for subsidized housing and supportive services for individuals experiencing homelessness. The target population. Social Finance brought together data from HMIS, RIDOC, the Executive Office of Health and Human Services (EOHHS) (emergency department visits, inpatient admissions, psychiatric inpatient admissions, and substance abuse admissions), and Medicaid claims. Individuals were matched between HMIS and RIDOC (n=1,400), HMIS and EOHHS /Medicaid (n=4,500), and HMIS, RIDOC, and EOHHS/Medicaid (n=1,300). Rhode Island defined the target population as frequent users of the Department of Corrections, homeless shelters, and Medicaid services. Financial feasibility. Data showed the status quo per person annual expenses for the 150 most expensive users in each system: $2,194 in HMIS; $11,045 in RIDOC; and $42,710 in Medicaid. The estimated cost of the intervention was $15,000 to $20,000 per person per year to deliver housing and support services. The expected benefit of the project is a 70-percent reduction in shelter stays, a 40-percent reduction in Department of Corrections days, and a 27-percent reduction in Medicaid costs. Further, the project anticipates a positive cost-benefit analysis based on baseline costs, expected benefit, and cost of the intervention, but the report did not offer a specific estimate because total cost avoidance could not be estimated. Feasibility of Permanent Supportive Housing. In addition to a description of PSH and its primary components, Social Finance included information on the model’s evidence base. In particular, they provided an overview of the expected effect size of different outcomes based on a review of prior studies. The type of supportive housing to be used was still under development. Program outcomes. The feasibility report recommended three project outcomes: housing stability, criminal justice system interaction, and inpatient healthcare use. Leveraging existing resources. To drive down the capital raise and associated amount of state repayment, Social Finance worked with the state to identify existing resources that could be repurposed to offset a portion of service delivery costs (that is, Medicaid or certain waiver programs to provide supportive services; existing state-funded housing subsidies or CoC vouchers to offset a portion of the total housing costs). Recommendations Social Finance recommended that Rhode Island move forward and base target population eligibility on shelter interaction, homelessness status, current access to housing and services, criminal justice involvement, and health care service needs. Social Finance recommended three next steps: (1) determine project scale, (2) define a mechanism to make payments, and (3) identify state resources (for example, vouchers) to be leveraged with the project. Further, they recommended additional data analysis: the project needs to integrate data from Medicaid, the Department of Corrections, and HMIS to determine cost avoidance. Anticipated Benefits Increased attention to a population in need. Despite general agreement in Rhode Island on the importance of serving this population through housing and services, limited progress had been made to date. The platform this Demonstration provided—for collaboration across government agencies, particularly strong leadership from the governor’s office—offers an opportunity to break that status quo. By helping to drive the feasibility process, the governor’s office has given the project visibility among key stakeholders and provided impetus. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 101  Increased attention on performance. Because of the PFS structure, partners considered how the project could complement broader ongoing efforts to improve the state’s public management. These efforts include those aligned with the Government Performance Lab fellows embedded in state government. Challenges Grantee capacity. Because of organizational leadership changes and capacity constraints, the Rhode Island Coalition for the Homeless did not play an active leadership role in the feasibility process. One exception was the coalition’s HMIS administrator, who attended working group meetings and contributed by providing and matching data. Data sharing. Rhode Island was unable to match individual-level data across all three systems of interest (housing, criminal justice, and health) because of Medicaid data-sharing restrictions. The target population, therefore, is not clearly defined and will need refining in the transaction structuring phase. Next Steps ■ Obtain approval from the governor’s office to advance to transaction structuring phase in early 2018 ■ Include the PFS project in the 2018–2019 budget ■ Develop transaction structuring work plan and identify the working group and steering group composition ■ Finalize resources that will be repurposed for PFS project Austin/Travis County, Texas The Ending Community Homelessness Coalition (ECHO) Site Summary as of December 2017 As of December 2017, the Austin/Travis County, Texas, site was in the transaction structuring phase. Austin/Travis County was able to build on some existing data infrastructure to complete feasibility analysis in early 2016 and identify the 500 costliest users of criminal justice, homeless, and health services. The site has struggled to secure end payors, in part because of unfamiliarity with and uncertainty about the PFS model. Some potential end payors would prefer to pay for programming directly. Others would like more clarity around potential success measures as they relate to their sector’s interests. The City of Austin and Central Health’s Community Care Collaborative committed to being end payors, but the site still seeks additional end payors to meet the target project size. Another struggle has been planning for the evaluation, particularly the tension between the desire for a rigorous evaluation (for example, a randomized control trial) and ability to provide services to all individuals who need them. The Criminal Justice Reform Context in Austin/Travis County The State of Texas became involved in justice reinvestment strategies in 2005, working with the Council of State Governments (CSG) Justice Center and the Bureau of Justice Assistance. The strategy ultimately focused on reducing probation revocations, increasing treatment program capacity for people on probation and parole (including community-based substance abuse and mental health services), and parole approvals (Council of State Governments Justice Center, 2009). Texas had identified an increase of 18 percent of people being revoked to prison from probation between 1997 and 2006. Some reinvestment initiatives included increasing slots for substance use disorder and mental health treatment and providing diversion options for probation and parole (Council of State Governments Justice Center, n.d.2). Recent significant criminal justice reform in Texas included the implementation of Smart-on-Crime policy, 130 crime-reduction strategies that focus on diversion and treatment programs, progressive sanctions for drug offenses, risk assessments to determine probation conditions, reductions in penalties, and reentry investment (Texas Criminal Justice Coalition, n.d.). EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 103  Homelessness Reform Efforts The Austin/Travis County CoC is led by the Ending Community Homelessness Coalition (ECHO). Austin/Travis County released a plan to end homelessness in 2010 that focused in large part on addressing long-term homelessness and systemwide coordination, which included the development of a coordinated entry system. In February 2017, the CoC was recognized by the federal government for having effectively ended veteran homelessness. During 2017, ECHO worked closely with community stakeholders, including the Austin mayor’s office and county staff, to develop an updated action plan to end homelessness in the county. The Austin/Travis County CoC is already heavily invested in PSH with nearly as much PSH capacity as emergency and transitional housing capacity. Additionally, investment in PSH has grown drastically in recent years. From 2010 to 2017, Austin more than doubled its PSH capacity from 540 beds in 2010 to 1,191 beds in 2017. Key Project Partners ■ Ending Community Homelessness Coalition (ECHO): Intermediary, grantee ■ CSH: Adviser to the intermediary ■ Social Finance: Adviser to the intermediary ■ City of Austin: Government partner, end payor ■ Central Health’s Community Care Collaborative: End payor ■ Travis County: Government partner, potential end payor Project Milestones ■ September 2017—City of Austin committed $1.2 million per year for 5 years ■ December 2017—Community Care Collaborative committed $1 to $1.2 million per year for 5 years Snapshot of the Feasibility Report for Austin/Travis County The Austin/Travis County feasibility study was completed by the Corporation for Supportive Housing (CSH), the adviser to the intermediary. CSH began working with ECHO in March 2015 and spent 1 year on the feasibility report. CSH found that the project was feasible and recommended that the project move forward but noted that additional work was necessary. The problem being addressed. The project focuses on individuals who are the most expensive homeless users of public services and cycle through hospitals, jails, and shelters. The project seeks to provide up to 250 units of supportive housing to this group. CSH noted that Austin/Travis County has undertaken several other efforts to end homelessness, including the Austin City Council funding 750 units between 2010 and 2018, a Medicaid-funded supportive housing project, and an ECHO effort to identify Medicaid clients who need housing. The target population. CSH and ECHO linked administrative data from the health information exchange, the Integrated Care Collaboration, the Travis County Sheriff’s Office, and HMIS. This brought together hospital use data (for example, emergency room visits and inpatient stays), jail data (bookings and jail bed days), and shelter use data. Ten thousand people were matched, and CSH focused on the 500 and 250 most expensive homeless individuals. Among the top 500 by total cost, the average individual had 13.3 emergency room visits, spent 14.4 days in the hospital, and was booked in jail 2.6 times. In other words, the target population was persons whose cumulative criminal justice, health care, shelter, and emergency medical costs placed them in a cohort of roughly the 500 most expensive homeless users of these public services. Criminal justice Homelessness “High needs” Persons whose cumulative criminal justice, health care, shelter, and EMS costs place them in a cohort of roughly the top 500 most expensive homeless users of these public services. Financial feasibility. CSH’s financial modeling considered intervention costs, administrative costs, status quo costs, and potential cost avoidance. The estimated cost of the intervention was $28,557 per person per year. The combined system cost was estimated to be $101,218 per person per year. Total per person cost avoidance would be $78,000 per year for public systems (the health care district, Travis County, and the City of Austin). CSH found that the project would lead to a total cost savings of nearly $43 million over 5 years. Feasibility of Permanent Supportive Housing. CSH noted several practical components of supportive housing, such as affordability thresholds, tenant leases, and connections between tenants and community- based resources. The report included the impact of supportive housing on housing retention, health care use and costs, and criminal justice outcomes. Further, CSH assumed the care coordination model would be intensive case management or Assertive Community Treatment but did not include any additional information about the models. The Supportive Housing model. The feasibility report recommends a mixture of project-based and scattered-site PSH units. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 105  Services. The feasibility report recommends using the Assertive Community Treatment model, with connection to clinical behavioral health, primary care, and criminal justice services. Program outcomes. CSH proposed using three success metrics: improved housing stability, reduced jail system use, and reduced emergency room use, medical services, and inpatient costs. Recommendations CSH recommended that ECHO continue with the project. Recommended next steps were to (1) identify financial support for transaction structuring; (2) solidify the commitment of an end payor; (3) select potential service providers, potentially using a Request for Proposals (RFP) or a Request for Quotes (RFQ); (4) select an evaluation partner; and (5) identify housing resources that could support the project. The City of Austin, Travis County, and Central Health were listed as potential end payors. Anticipated Benefits New partners. The work done to bring in potential investors, particularly the relationship created with hospitals through Central Health, has identified new partners that are not the typical organizations involved in homelessness or PSH. Increased buy-in for PSH. This project created awareness about the importance of PSH in providing housing and services for vulnerable populations, as well as the potential cost offsets or savings for other industries, especially among nontraditional entities such as county and health care organizations. Challenges Time commitment. Most partners in Austin/Travis County see this project as a major time commitment for themselves and their staff. Government partners estimate that 10 to 25 percent of their time was spent on this project; intermediary and adviser organizations estimate 25 to 75 percent, depending on the key tasks of focus at any given time. Many end payors. Partners with past PFS experience credited the large time commitment to having three end payors with distinct interests and concerns. The costliest users in the target population incur costs across sectors, which do not align across all end payors; consequently, separate conversations and negotiations were necessary, as was time to align the three. Planning for evaluation. There has been a tension between the desire for rigorous evaluation and concerns around denying services to persons with high needs. Next Steps ■ Start pilot for 24 people: February 2018 ■ Secure county end payor ■ Define the evaluation plan EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 107  Appendix B. NOFA-Delineated Activities by PFS Phase As described in the NOFA, the following are considered grant-eligible activities in each phase (HUD, 2015: 13–21). Feasibility analysis ■ Organizational/Programmatic Activities » Assess the likelihood of success of a PSH model financed by PFS in the local context » Assess the strength, expertise, and capacity of relevant nonprofit PSH providers to deliver desired outcomes consistent with HUD’s programmatic guidelines for PSH » Assess available local, state, tribal, and/or federal administrative data and other available evidence, data and information relevant to carrying out a potential PFS PSH Intervention, including data matching and analysis » Provide due diligence, program design, and advisory services to assist government entities in determining whether and how to engage in the planned PSH Intervention using PFS financing » Assess the policy priorities of the relevant government entity that are consistent with the provision of PSH targeted to the reentry population ■ Budgetary/Financial Activities » Identify and estimate potential local, state, and federal funding sources that will be impacted by the project, including costs and savings to each affected level of government and program. These funding sources should be used to estimate potential net savings, as well as opportunities to achieve outcomes more cost-effectively at each level of government through the implementation of a PSH intervention targeting the reentry population (estimates of total cost savings must account for the net effect of any cost shifting among levels of government) » Inform decision making, develop a framework and conduct analyses for estimating public sector savings, cost-effectiveness, and benefits and success payments for the PSH intervention » Identify options for a financing strategy to sustain and scale up the PSH intervention should the intervention be deemed feasible for implementation through PFS in the Demonstration site » Develop a budget, estimating the costs needed for the Transaction Structuring phase and ramp-up costs (if any) ■ Legal/Regulatory Activities » Identify statutory, regulatory, and programmatic barriers to and enablers of a PFS PSH intervention » Review agreements and contracts for legal requirements and protections » Assess and address appropriation risks (that is, the risk that the government entity may not be able to make future success payments) ■ Procurement Activities » Support government efforts to design and implement a process for collecting relevant information from the public or key audiences to inform PFS activities, regarding priorities, service delivery, Transaction Structuring, evaluation, or other relevant issues, priorities, concepts, and strategies » Support government efforts to design and publicize requests for proposals, notices of funding availability, or other relevant funding announcements/proposal solicitations for release by government entities or other end payors to solicit the services of coordinators, service providers, or evaluators » Support government efforts to assess solicited proposals, including respondents’ organizational capacity, past performance, operating model, strength of outcomes, efficiency, quality of management team, and suitability for PFS financing ■ Applicants may propose additional or alternative strategies under any of the task areas listed earlier which further the purposes of the HUD-DOJ Demonstration. Transaction Structuring ■ Provide overall transaction coordination and support » Design PFS transaction work plan, timeline, and task list » Coordinate planning and meetings of relevant transaction participants » Manage all transaction elements to meet the shared timeline of stakeholders » Identify an outreach plan to identify eligible PSH Intervention participants that includes coordination between correctional facilities, homeless assistance providers, crisis care services, applicable government agencies, and other key players involved in EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 109  implementing the PSH Intervention. Outreach plans should consider how persons with disabilities and limited English proficiency can be reached » Develop a plan to identify, select, train, and provide technical assistance for homeless assistance providers » Assess the strength, expertise, and capacity of selected homeless assistance providers, including a quantitative and qualitative assessment of respondents’ track record, operating model, strength of outcomes, and compatibility with the transaction » Address homeless assistance provider performance concerns or capacity gaps » Engage and educate homeless assistance provider staff to ensure that expectations of their role in the PSH Intervention are clear and feasible » Coordinate selection of qualified third-party evaluator(s) » Develop outcome measures and specify evaluation methodologies and data sources that can be accessed to valuate identified outcome measures » Ensure that the evaluation design and service delivery plans are fully compatible (that is, if the evaluation design were to require the random assignment of participants, does the service delivery plan accommodate this requirement?) » Ensure that all data necessary to identify the target population and measure outcomes will be made available by the government entity or other data source, and shared among relevant stakeholders, including the intermediary, outcome evaluator, DOJ, and HUD, in a timely manner and in accordance with all applicable confidentiality requirements » Coordinate or lead design of key PSH Intervention components, including detailed service provision, duration of services, outcome monitoring, and evaluation design » Assess the risk of and develop contingency plans for lack of stakeholder capacity or support, (that is, stakeholders not providing adequate feasibility checks or information and/or not agreeing to key PSH Intervention requirements identified in this NOFA) ■ Raise capital and develop capital structure » Identify sources of funding for success payments (including sources beyond federal funds) » Conduct financial modeling of the transaction, including analysis of possible payment terms and transaction structures » Develop investment and entity structure, especially regarding success payments, that mitigates relevant risks and establishes appropriate incentives » Develop relevant documentation, such as a term sheet, that includes outcomes pricing, capital structure, success payment triggers, and payment schedules » Ensure that, in the case that HUD-DOJ Demonstration grant funds cover the cost of any ramp-up activities, those costs are considered in the structuring of success payments and the measurement of net savings generated by the PSH Intervention » Market the transaction to investors to raise capital commitments necessary to fund the PSH Intervention ■ Mediate and facilitate the written agreement between transactional relevant parties » Coordinate the negotiation of all parties around economic and contract terms » Develop and finalize all contracts and supplementary documentation, including offering or loan documents as relevant, working with legal counsel as appropriate ■ Close the PFS Contract and prepare for the Contract Implementation phase. Preparation for Contract Implementation may include ramp-up activities needed to assist homeless assistance providers to scale up their operations for the PSH intervention. Examples may include activities that build the infrastructure of the PSH intervention prior to the delivery of housing and services such as: recruiting, hiring, and training additional homeless assistance provider staff; training current staff to ensure they have the capacity to deliver the intended intervention; refining the referral process; finalizing and implementing evaluation methodologies; equipment; facility set-up; setting up information technology (IT) for the systematic collection of program participant data; and programming Homeless Management Information System (HMIS) specifications. These activities are also relevant for PFS contracts which are “shovel-ready” at the time of application and require funding for costs related to closing the transaction. ■ Ensure that the PFS contract accounts for any potential changes in risk associated with the separate funding of ramp-up activities and allows for potential appropriate contract modifications based on lessons learned during the ramp-up phase, including potential modification of outcome targets, time horizons, programmatic changes, and the amount and structure of success payments. ■ Support early Contract Implementation activities up to the delivery of housing and services to ensure the PFS Contract is moving toward successful implementation. ■ Applicants may propose additional or alternative strategies under any of the above task areas which further the purposes of the HUD-DOJ Demonstration. Project Implementation ■ Outcome evaluation » Inform and agree to the outcome evaluation design and validation methodology » Conduct an independent validation of the outcome evaluation methodology, including to ensure that needed data can be accessed appropriately and timely EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 111  » Monitor the outcome measure(s) agreed upon in the PFS Contract » Assess whether the operationalization of the PSH Intervention is consistent with HUD’s programmatic guidelines for PSH » Determine whether the outcome goals have been met » Provide the documentation to trigger the release of Success Payments. ■ Success payments » Payments to the investor(s) (for example, financial institution, foundation) once the third-party evaluator verifies that an outcome goal, as established and agreed upon in the PFS Contract, has been met through the agreed-upon validation methodology approved by DOJ and HUD. HUD-DOJ will not pay investors directly. The end payors identified by the grantees are responsible for paying the investors based on the terms outlined in the PFS contract. References Aidala, Angela, William McAllister, Maiko Yomogida, and Virginia Shubert. 2014. Frequent Users Service Enhancement “FUSE” Initiative: New York City FUSE II Evaluation Report. New York, NY: Columbia University Mailman School of Public Health. Alaska Criminal Justice Commission. 2015. Justice Reinvestment Report. Anchorage, AK: Alaska Judicial Council. Alper Mariel, Matthew R. Durose, and Joshua Markman. 2018. Update on Prisoner Recidivism: A 9-Year Follow-up Period (2005-2014). Washington, DC: U.S. Department of Justice, Office of Justice Programs. Azemati, Hanna, Michael Belinsky, Ryan Gillette, Jeffrey Liebman, Alina Sellman, and Angela Wyse. 2013. “Social Impact Bonds: Lessons Learned So Far,” Community Development Investment Review 9 (1): 22–32. Bird, Mia, Sonya Tafoya, Ryken Grattet, and Viet Nguyen. 2016. How Has Proposition 47 Affected California’s Jail Population? San Francisco, CA: Public Policy Institute of California. Blum, Jessica, Michelle Ciurea, Marjorie Levin, Tyler Moazed, Allan Porowski, and Alicia Sparks. 2015. State of the Pay for Success Field: Opportunities, Trends, and Recommendations. Washington, DC: Corporation for National & Community Service. Brooks, James. 2018. “Goodbye Bail: Alaska Switches to New System of Criminal Justice,” JuneauEmpire.com, January 1. http://juneauempire.com/state/news/2017-12-20/goodbye-bail-alaska-switches-new-system-criminal-justice. Bureau of Justice Assistance. 2016. Orientation Webinar: FY 2016 Second Chance Act Smart Reentry Grantees. Bureau of Justice Assistance, National Reentry Resource Center. https://csgjusticecenter.org/wp- content/uploads/2017/01/Smart-Reentry-Orientation-Webinar.pdf. Bureau of Justice Statistics. 2018. Correctional Populations in the United States, 2016. Washington, DC: U.S. Department of Justice, Office of Justice Programs. Burt, Martha, Laudan Aron, Toby Douglas, Jesse Valente, Edgar Lee, and Britta Iwen. 1999. Homelessness: Programs and the People They Serve| Findings of the National Survey of Homeless Assistance Providers and Clients. Washington, DC: Urban Institute. Centre for Social Impact Bonds. n.d. “Social Impact Bonds.” https://data.gov.uk/sib_knowledge_box/developing-social- impact-bond-0. City and County of Denver. n.d.1. Denver Social Impact Bond Initiative: Supportive Housing. https://www.equitasproject.org/wp-content/uploads/2014/10/Denver-SIB-Summary1.pdf. Denver, CO. ———. n.d.2. Housing Opportunity: Chronically Homeless Individuals—Using a Social Impact Bond to Support a Broader City Homelessness Strategy. Denver, CO. Commonwealth of Massachusetts Executive Department. 2014. Massachusetts Launches Pay for Success Initiative to Reduce Chronic Individual Homelessness. Press release. Washington, DC: Commonwealth of Massachusetts Executive Department: Office of Governor Deval L. Patrick. CommUnityCare Health Centers. n.d. “Integrated Care Collaboration.” CommUnityCare Health Centers. http://communitycaretx.org/outreach/partnerships.html/title/integrated-care-collaboration. Council of State Governments Justice Center. 2015. Justice Reinvestment in Rhode Island: Overview. New York, NY: Council of State Governments Justice Center. July. https://csgjusticecenter.org/wp- content/uploads/2015/07/RhodeIslandOverview.pdf. ———. 2009. Assessing the Impact of the 2007 Justice Reinvestment Initiative: Justice Reinvestment in Texas Brief. New York, NY: Council of State Governments Justice Center. ———. n.d.1. “Rhode Island.” Council of State Governments Justice Center. https://csgjusticecenter.org/jr/ri/. ———. n.d.2. “Texas.” Council of State Governments Justice Center. https://csgjusticecenter.org/jr/tx/. Culhane, Dennis. 2018. Chronic Homelessness. Rockville, MD: Center for Evidence-based Solutions to Homelessness, Abt Associates. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 113  Culhane, Dennis, Stephen Metraux, and Trevor Hadley. 2002. “The Impact of Supportive Housing for Homeless People with Severe Mental Illness on the Utilization of the Public Health, Corrections, and Emergency Shelter Systems: The New York-New York Initiative,” Housing Policy Debate 13 (1): 107–163. Washington, DC: Urban Institute. Czerwinski, Stanley. 2010. Nonprofit Sector: Treatment and Reimbursement of Indirect Costs Vary Among Grants, and Depend Significantly on Federal, State, and Local Government Practices. Report no. GAO-10-477. Washington, DC: Government Accountability Office. DePledge, Derrick. 2017. “Justice Reinvestment; Oregon Takes New Steps to Reduce Prison Use,“ The Daily Astorian, July 17. http://www.dailyastorian.com/Local_News/20170717/justice-reinvestment-oregon-takes-new-steps-to- reduce-prison-use. Dresser, Michael. 2016. “Hogan Signs Bill to Overhaul Maryland Criminal Justice System,” Baltimore Sun, May 19. http://www.baltimoresun.com/news/maryland/politics/bs-md-justice-reinvestment-20160518-story.html. Galloway, Ian. 2015. “Pay for Success: Unlocking Government Funding for Prevention,” National Council on Crime and Delinquency blog, January 27, https://www.nccdglobal.org/blog/pay-success-unlocking-government-funding- prevention. Gillespie, Sarah, Devlin Hanson, Mary Cunningham, Akiva Liberman, and Janine Zweig. Forthcoming. Practical Considerations for Pay for Success Evaluation. Washington, DC: Urban Institute. Global Impact Investing Network. n.d. “What You Need to Know about Impact Investing.” https://thegiin.org/impact- investing/need-to-know/. Gouvis Roman, Caterina, and Jeremy Travis. 2004. Taking Stock: Housing, Homelessness, and Prisoner Reentry. Washington, DC: Urban Institute. Gustafsson-Wright, Emily, Sophie Gardiner, and Vidya Putcha. 2015. The Potential and Limitations of Impact Bonds: Lessons from the First Five Years of Experience Worldwide. Washington, DC: Brookings Institution. Hawkins, Rayanne, Mayookha Mitra-Majumdar, David McClure, and Ellen Paddock. 2017. Using Pay for Success in Criminal Justice Projects: Lessons Learned from PFS Project Stakeholders, Policy Analysis, and Landscape Analysis. Washington, DC: Urban Institute. Henry, Meghan, Rian Watt, Lily Rosenthal, and Azim Shivji. 2017. The 2017 Annual Homelessness Assessment Report (AHAR) to Congress, Part 1: Point-in-Time Estimates of Homelessness. Washington, DC: U.S. Department of Housing and Urban Development. Hunt, Dana. 2018. Criminal Justice Reentry. Rockville, MD: Center for Evidence Based Solutions to Homelessness, Abt Associates. Hunter, Sarah B., Melody Harvey, Brian Briscombe, and Matthew Cefalu, 2017. Evaluation of Housing for Health Permanent Supportive Housing Program. Santa Monica, CA: Rand Corporation. Larimer, Mary, Daniel Malone, Michelle Garner, et al. 2009. “Health Care and Public Service Use and Costs before and after Provision of Housing for Chronically Homeless Persons with Severe Alcohol Problems,” JAMA 301 (13): 1349– 1357. Liebman, Jeffrey, and Alina Sellman. 2013. Social Impact Bonds: A Guide for State and Local Governments. Cambridge, MA: Harvard Kennedy School Social Impact Bond Technical Assistance Lab. Lofstrom, Magnus, and Brandon Martin. 2015. Public Safety Realignment: Impacts So Far. San Francisco, CA: Public Policy Institute of California. Cunningham, Mary, Sarah Gillespie, and Jacqueline Anderson. 2015. “Rapid re-housing: What the research says.” Housing and Housing Finance: 3–15. Cunningham, Mary, Mike Pergamit, Sara Gillespie, Devlin Hanson, and Shiva Kooragayala. 2016. "Denver Supportive Housing Social Impact Bond Initiative: Evaluation and Research Design." Kessell, Eric R., Rajiv Bhatia, Joshua D. Bamberger, and Margot Kushel. 2006. “Public Health Care Utilization in a Cohort of Homeless Adult Applicants to a Supportive Housing Program,” Journal of Urban Health 83 (5): 860–873. Krauth, Barbara. 1997. A Review of the Jail Function Within State Unified Corrections Systems. Longmont, CO: National Institute of Corrections.  Lutze, Faith, Jeffrey W. Rosky, and Zachary K. Hamilton. 2014. “A Multisite Outcome Evaluation of Washington State’s Reentry Housing Program for High Risk Offenders,” Criminal Justice and Behavior 41 (4): 471–491. Mann, Cindy. 2011. “A New Era for State Medicaid and Children’s Health Insurance Programs,” Academic Pediatrics 11 (3): S95–S96. Martinez, Tia, and Martha Burt. 2006. “Impact of Permanent Supportive Housing on the Use of Acute Care Health Services by Homeless Adults,” Psychiatric Services 57 (7): 992–999. Metraux, Stephen, and Dennis Culhane. 2006. “Recent Incarceration History among a Sheltered Homeless Population,” Crime & Delinquency 52 (3): 504–517. Mondello, Melany, Anne Gass, Thomas McLaughlin, and Nancy Shore. 2007. Cost of Homelessness: Cost Analysis of Permanent Supportive Housing, State of Maine, Greater Portland. New York, NY: Corporation for Supportive Housing. Montgomery County Pre-Release and Reentry Services Division. n.d. “Montgomery County, MD.” National Association of Counties. https://www.naco.org/resources/montgomery-county-md-pre-release-and-reentry- services-division. National Alliance to End Homelessness. 2016. “Chronic Homelessness.” http://www.endhomelessness.org/pages/chronic_homelessness. National Reentry Resource Center. n.d. “Second Chance Act Statewide Recidivism Reduction Program.” Council of State Governments Justice Center. https://csgjusticecenter.org/nrrc/second-chance-act-recidivism-reduction- grant-program/. Office of National Drug Control Policy (ONDCP). n.d. Chapter 3: Integrate Treatment for Substance Use Disorders into Mainstream Health Care and Expand Support for Recovery. Washington, DC: Obama White House Archives. Perlman, Jennifer, and John Parvensky. 2006. Denver Housing First Collaborative: Cost Benefit Analysis and Program Outcomes Report. Denver, CO: Colorado Coalition for the Homeless. Popper, Nathaniel. 2015. “Success Metrics Questioned in School Program Funded by Goldman,” New York Times, November 3. https://www.nytimes.com/2015/11/04/business/dealbook/did-goldman-make-the-grade.html. Reyes, Stephanie. 2017. “Rhode Island General Assembly Passes Parole, Probation Reform Bills,” Brown Daily Herald, September 25. http://www.browndailyherald.com/2017/09/25/rhode-island-general-assembly-pass-parole- probation-reform-bills/. Roman, John. 2015. Solving the Wrong Pockets Problem. Washington, DC: Urban Institute. Roman, John, Matthew Eldridge, and Rayanne Hawkins. 2015. “Foundational Concepts and Terms of Pay for Success.” Washington, DC: Urban Institute. Roman, John, Kelly Walsh, Sam Bieler, and Samuel Taxy. 2014. Five Steps to Pay for Success: Implementing Pay for Success Projects in the Juvenile and Criminal Justice Systems. Washington, DC: Urban Institute. Sadowski, Laura, Romina Kee, Tyler VanderWeele, and David Buchanan. 2009. “Effect of a Housing and Case Management Program on Emergency Department Visits and Hospitalizations among Chronically Ill Homeless Adults: A Randomized Trial,” Jama 301 (17): 1771–1778. Safety and Justice Challenge. 2017. “Pima County 2017 Safety and Justice Challenge Fact Sheet.” Safety and Justice Challenge. http://www.safetyandjusticechallenge.org/wp-content/uploads/2017/12/Pima-County-SJC-2017-Fact- Sheet.pdf. ———. n.d.1. “Pima County, AZ.” Vera Institute of Justice. http://www.safetyandjusticechallenge.org/challenge- site/pima-county/. ———. n.d.2. “Los Angeles, CA.” Vera Institute of Justice. http://www.safetyandjusticechallenge.org/challenge-site/los- angeles-county/. Sager, Michelle. 2015. Pay for Success: Collaboration among Federal Agencies Would Be Helpful as Governments Explore New Financing Mechanisms. Report no. GAO-15-646. Washington, DC: Government Accountability Office. Seligson, A., S. Lim, T. Singh, et al. 2013. New York/New York III Supportive Housing Evaluation: Interim Utilization and Cost Analysis. New York, NY: New York City Department of Health and Mental Hygiene. EVALUATION OF THE HUD-DOJ PAY FOR SUCCESS PERMANENT SUPPORTIVE HOUSING DEMONSTRATION: BASELINE REPORT 115  Teicher, Perry, John Grossman, and Marcia Chong. 2016. Authorizing Pay for Success Projects: A Legislative Review and Model Pay for Success Legislation. Boston, MA: Third Sector Capital Partners. Texas Criminal Justice Coalition. n.d. “Adult and Juvenile Justice System Reforms in Texas.” https://www.texascjc.org/adult-juvenile-justice-system-reforms-texas. Third Sector Capital Partners. 2016. Developing the Cuyahoga Partnering for Family Success Program: Partner Perspectives & Lessons Learned. Boston, MA: Third Sector Capital Partners. Tucson Pima Collaboration to End Homelessness. 2015. Service Gaps & Opportunities in Ending Homelessness: Pima County, AZ. OrgCode Consulting, Partners. https://www.tpch.net/uploads/1/4/0/0/14008364/pima_county_service_gaps_report- final_edits_sept_21_2015.pdf. U.S. Department of Housing and Urban Development (HUD). 2015. Pay for Success Permanent Supportive Housing Demonstration. Report no. FR-5900-N-26. Washington, DC: HUD. U.S. Interagency Council on Homelessness (USICH). 2016. Reducing Criminal Justice System Involvement Among People Experiencing Homelessness. Washington, DC: USICH. Wagner, Peter, and Bernadette Rabuy. 2017. Following the Money of Mass Incarceration. Northampton, MA: Prison Policy Initiative. The White House (President Barack Obama). 2016. “Launching the Data-Driven Justice Initiative: Disrupting the Cycle of Incarceration,” fact sheet. https://obamawhitehouse.archives.gov/the-press-office/2016/06/30/fact-sheet- launching-data-driven-justice-initiative-disrupting-cycle. June 2019 REFERENCES 117