Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment

 Accelerating America’s Transition to a Performance-Driven Social Sector Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment Marion County & Multnomah County Final Report August 2016 Project Partners:  Accelerating America’s Transition to a Performance-Driven Social Sector About Third Sector Capital Partners, Inc. Third Sector Capital Partners, Inc. (“Third Sector”) is a nonprofit advisory services firm that leads governments, high-performing nonprofits, and private funders in building evidence-informing initiatives that address society’s most persistent challenges. As experts in innovative public-private contracting and financing strategies, Third Sector is an architect and builder of the nation’s most promising Pay for Success (“PFS”) projects including those in Commonwealth of Massachusetts, Cuyahoga County, Ohio, and Santa Clara County, California. Third Sector has been awarded more U.S. government mandates to construct PFS projects than any other firm. These projects are rewriting the book on how governments contract for social services: funding programs that work to measurably improve the lives of people most in need while saving taxpayer dollars. A 501(c)(3) nonprofit based in Boston, San Francisco, and Washington D.C., Third Sector is supported through philanthropic and government sources, including a grant from the Corporation for National and Community Service’s Social Innovation Fund. Third Sector is a proud grantee of the Corporation for National and Community Service’s Social Innovation Fund. This feasibility report has been prepared as part of Third Sector's sub-recipient award to Marion and Multnomah Counties. Learn more at www.thirdsectorcap.org. About National League of Cities The National League of Cities (“NLC”) is dedicated to helping city leaders build better communities. Working in partnership with the 49 state municipal leagues, NLC serves as a resource to and an advocate for the more than 19,000 cities, villages and towns it represents. The NLC members are municipalities from across the country. A city or town joins NLC and the elected officials and staff participate in NLC's programs, activities and governance. State municipal leagues are also active members of NLC, guiding the organization's priorities and serving as an important link to cities in their state. And, NLC offers membership opportunities for members of the private and non-profit sector. About Marion County Marion County (“Marion”) is Oregon’s fifth most populous county. With four of fourteen state correctional institutions, two state juvenile corrections facilities, and the Oregon State Hospital housing offenders with psychiatric diagnoses, the county population of state prisoners and local inmates is the highest per capita rate of any Oregon county. More than 27% of Oregon’s total prison population is housed in Salem, the county seat and Oregon’s capital. The Marion County Sheriff’s Office operates the third largest jail in Oregon. Learn more about Marion’s juvenile justice program by visiting http://www.co.marion.or.us/JUV. About Multnomah County Multnomah County (“Multnomah”) is the most populous county in Oregon, with a total population of more than 700,000. It contains Oregon’s largest city (Portland), as well as several smaller cities in the eastern portion of the county. It is the smallest in area of all the counties in the state, and the most densely populated. According to the Poverty in Multnomah County report, Multnomah County’s population is more highly educated with a higher median and per capita income, but also a higher rate of poverty compared to Oregon as a whole. Multnomah County is also Oregon’s most racially and ethnically diverse county. Learn more about Multnomah’s juvenile justice program by visiting https://multco.us/dcj-juvenile Collectively, both counties will be referred to herein as the “Counties.”  Table of Contents I. II. III. IV. OVERVIEW OF PAY FOR SUCCESS.....................................................................................................................................2 MARION/MULTNOMAH COUNTY FEASIBILITY ASSESSMENT OVERVIEW .........................................................................4 OVERVIEW OF SOCIAL INNOVATION FUND ............................................................................................................................................. 4 PARTNERS OF THIRD SECTOR ON SOCIAL INNOVATION FUND GRANT...........................................................................................................5 SUB-RECIPIENT OVERVIEW .................................................................................................................................................................5 PFS FEASIBILITY ASSESSMENT STRUCTURE AND TIMELINE ......................................................................................................................... 7 GUIDING OBJECTIVES FOR PFS FEASIBILITY ASSESSMENT .......................................................................................................................... 8 EXECUTIVE SUMMARY AND OVERVIEW OF WORK STREAMS ........................................................................................10 EXECUTIVE SUMMARY......................................................................................................................................................................10 READINESS SCALE AND OVERVIEW OF WORK STREAMS .......................................................................................................................... 11 RECOMMENDED NEXT STEPS.............................................................................................................................................................13 PFS FEASIBILITY ASSESSMENT WORK STREAMS.............................................................................................................15 DATA SOURCE & ACCESS ASSESSMENT: PROMISING .............................................................................................................................. 15 TARGET POPULATION ASSESSMENT: PROMISING ................................................................................................................................... 23 INTERVENTION ASSESSMENT: NOT YET DEMONSTRATED ........................................................................................................................ 37 ECONOMIC & END PAYER ASSESSMENT: PROMISING ............................................................................................................................. 52 LEGAL & REGULATORY ASSESSMENT: NOT YET DEMONSTRATED..............................................................................................................74 FUNDER DEVELOPMENT: NOT YET DEMONSTRATED .............................................................................................................................. 89 APPENDICES ...........................................................................................................................................................................92 APPENDIX A: MEMBERSHIP IN FEASIBILITY ASSESSMENT WORKING GROUPS .............................................................................................. 92 APPENDIX B: JUVENILE CRIME PREVENTION (JCP) ASSESSMENT 2006.1 RISK INDICATORS FOR ELIGIBLE YOUTHS ............................................. 96 APPENDIX C: DESCRIPTIVE OVERVIEW OF JUVENILE JUSTICE PLACEMENTS .................................................................................................. 98 APPENDIX D: ORGANIZED SKEPTICISM OF OREGON PAY FOR SUCCESS.....................................................................................................100 APPENDIX E: TOUGH QUESTIONS ON PAY FOR SUCCESS: CORE CRITICISM OF THE MODEL........................................................................... 107 APPENDIX F: SERVICE PROVIDER REQUEST FOR INFORMATION (RFI) TEMPLATE ........................................................................................ 114 APPENDIX G: EVALUATOR PROCESS & REQUIREMENTS ........................................................................................................................ 118 APPENDIX H: COST OF PLACEMENT IN OYA RESIDENTIAL AND YCF ........................................................................................................ 121 APPENDIX I: OREGON VICTIMIZATION COSTS BY CRIME TYPE................................................................................................................. 122 APPENDIX J: COUNTY ADMINISTRATIVE COST FOR NEW CRIMINAL REFERRALS .......................................................................................... 123 APPENDIX K: RESTITUTION AMOUNT PER CRIMINAL REFERRAL .............................................................................................................. 124 APPENDIX L: CRIMINAL REFERRALS PER YOUTH ON PROBATION BY PLACEMENT TYPE ................................................................................. 125 APPENDIX M: TOTAL COST ASSUMPTIONS IN ABSENCE OF AN INTERVENTION (BY SCENARIO) ...................................................................... 126 APPENDIX N: LITERATURE REVIEW: QUANTIFYING SOCIAL COSTS ........................................................................................................... 129 APPENDIX O: REQUIREMENTS FOR ESTABLISHING A SPECIAL PURPOSE ENTITY (SPE) ................................................................................. 134 APPENDIX P: OREGON REGISTRATION FORM OF A NONPROFIT 501(C)(3) ............................................................................................... 135 APPENDIX Q: LAUNCHED PROJECT EXAMPLE – SANTA CLARA COUNTY .................................................................................................... 138 APPENDIX R: UNDERSTANDING THE CONDUITS FOR FUNDING AND STRUCTURING A PROJECT ...................................................................... 141 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 1 I. Overview of Pay for Success The Pay for Success (“PFS”) feasibility assessment for Marion County (“Marion”) and Multnomah County (“Multnomah”) (collectively referred to as the “Counties”) is a collaborative effort to evaluate the development of a juvenile justice initiative under an outcomes-based contracting model. The Counties’ juvenile justice initiative is an effort to (i) divert youth in the juvenile justice system from further escalating to county secure or out of home programs, or into state custody, (ii) provide culturally appropriate services to youth overrepresented in the juvenile justice system, and (iii) thereby strengthen and restore families. The technical assistance provided for this assessment is led by Third Sector Capital Partners, Inc. (“Third Sector”), with support from staff in juvenile services, procurement, legal, regulatory, and finance departments in both Counties. This work was made possible through a grant from the Corporation for National and Community Service’s Social Innovation Fund. Elements of Pay for Success Pay for Success (“PFS”)1 is a form of performance-based contracting that strives to raise the bar on its three main elements: (1) A clear set of objectives and indicators (2) Guidelines to collect data on the progress of selected outcomes and indicators, (3) Performance-driven incentives with “upside” (bonus payments) or “downside” (injunctions) for service delivery2 In a PFS project, government and/or private payers (“end payers”) pay a service provider(s) based on specific outcomes achieved. Before a PFS contract is implemented, end payers and service providers mutually agree to the specific terms and conditions of the project including outcomes, evaluation plans, and payment structures. Once launched, service providers begin delivering services while an independent evaluator rigorously assesses the impact of the services; end payers make payments only when outcomes are achieved. Third Party Funding is an optional component of PFS to capitalize a project when government resources do not fund a service provider at the onset of an intervention (or partially withhold payments). This form of funding is drawn down by a service provider while intervention results are being evaluated and until success payments are made by an end payer. Outside funders – like philanthropic organizations, private financial institutions, and/or individual grant makers – can provide the upfront funding for services and evaluation. Depending on the initial PFS agreements entered into by all parties, government end payments for services can be used to repay funders or recycled back into the project to provide more services. This mechanism allows end payers to withhold success payment 1 The term Pay for Success is utilized instead of the original name for this mechanism, Social Impact Bonds (or SIBs), because it better conveys how the process works 2 “What is Performance-Based Contracting?” World Health Organization. Retrieved July 2016 http://www.who.int/management/resources/finances/Section2-3.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 2 until results are measured, while also ensuring cost coverage for service providers. As described above, it is important to note that performance-based contracting and third party funding are complementary but distinct elements of a Pay for Success project. Subject to the funding needs of a service provider and sources of funding for government end payers, in addition to relationships in the community, not every PFS project requires (or is well-suited for) third party funding. Benefits of PFS Principles Though it is not a silver bullet for pressing social issues, PFS offers a number of benefits to stakeholders – end payers are able to ensure their funds only go toward programs that are working (based on outcomes achieved), service providers can focus on achieving impact, and if required, funders are able to help build government end payer and service provider capacity. The process thereby extends the time frame for measuring outcomes, enabling all parties to work together to drive better outcomes for vulnerable families. Given this potential, Pay for Success has become a national movement with eleven projects launched in states across the country (California, Colorado, Connecticut, Illinois, Massachusetts, New York, North Carolina, Ohio, and Utah) and dozens more projects in development3. The eleven PFS projects launched in the United States all utilize performance-based contracts and third party funding, however, the specifics of each project are vastly different. For instance, some projects are funded mostly by philanthropy, while others are funded mostly by private financial institutions. 3 “Pay for Success Projects.” Nonprofit Finance Fund. Retrieved via http://www.payforsuccess.org/provider-toolkit/pfs-projects Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 3 II. Marion/Multnomah County Feasibility Assessment Overview Third Sector provided technical assistance to the Counties to explore a PFS project in juvenile justice in accordance with the SIF grant competition. The following section provides an overview of the grant for the project, Third Sector, the Counties, and other project partners. Overview of Social Innovation Fund The Social Innovation Fund (“SIF”), a key White House initiative and program of the Corporation for National and Community Service (“CNCS”), combines public and private resources to grow the impact of innovative, community-based solutions that have compelling evidence of improving the lives of people in low-income communities throughout the United States. In June 2014, SIF released a Notice of Funding Availability of up to $12 million for the 2014 SIF PFS grant competition. Congress sought an honest, impartial experiment about PFS to learn more about where and how it works best. Grantees, including Third Sector, focused on supporting potential PFS opportunities in SIF’s priority areas of economic opportunity, healthy futures, and youth development. Furthermore, SIF’s goal with its PFS competition was to accelerate the field by diversifying the sector, in terms of social issue areas, locations, and implementing organizations. Third Sector was awarded $1.9 million to provide technical assistance to assess feasibility and develop PFS capacity. Third Sector ran a competitive process for jurisdictions to apply for technical assistance awards. Third Sector chose nine sub-recipients for Round 1 of the grant and developed three levels of technical assistance awards, based on the readiness of the sub-recipient to develop a Pay for Success project:  Accelerating Assistance Award: This award was presented to applicants who had already initiated some of the steps required for a Feasibility Assessment. These applicants received smaller, “Accelerating” awards, since it was anticipated that they would require fewer resources and less time to complete the Feasibility Assessment process.  Intensive Assistance Award: This award was presented to applicants who were just at the beginning of the Feasibility Assessment process and demonstrated a high-level of readiness for Pay for Success. These applicants required more comprehensive “Intensive” awards.  Developmental Assistance Award: This award was presented to applicants who demonstrated particular promise based on their outstanding performance on selected aspects of PFS readiness and their perceived potential to improve their readiness over time. These applicants received smaller “Developmental” awards to assist in building their capacity and advancing their readiness to launch a PFS project. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 4 The Counties applied for technical assistance from Third Sector to assess a juvenile justice initiative and received a Development Assistance Award for up to $123,000 of services.4 Partners of Third Sector on Social Innovation Fund Grant For implementation of its SIF grant, Third Sector partnered with leading national organizations that assisted in developing the technical assistance competition, conducting outreach efforts, sharing lessons learned and delivering services to sub-recipients. These organizations are America Forward, Abt Associates, the National League of Cities (“NLC”), the National Governor’s Association (“NGA”), and the National Association of Counties (“NACo”).  Abt Associates provided evaluation and programmatic experts who worked alongside Third Sector staff in delivering technical assistance, thus not only benefiting the sub- recipients, but also building other partners’ capacity to bring PFS expertise to their much wider audience of governmental leaders.  NLC, NGA, NACo, and America Forward educated their constituencies about the benefits of PFS and the opportunities presented by the SIF PFS technical assistance. They also brought a cadre of experts in governmental structure and the SIF priority program issue areas to assist in providing high-quality technical assistance. This partnership ensured that these organizations’ resources and insights were brought to bear in building the knowledge base necessary to scale PFS nationally. Sub-Recipient Overview The Counties indicated they were seeking technical assistance to explore a Pay for Success model for justice involved youth, using a proven, evidence-based intervention; in this case an intensive in-home program for probation youth at risk of out-of-home placement. Such an intervention would need to be well suited for an outcomes-based contract and committed by an end payer willing and able to pay for achieved results. As potential end payers and/or administrators of a project, the Counties indicated their willingness to partner with state agencies (e.g. Oregon Youth Authority) and state-level parties (e.g. the Oregon Legislative Assembly and/or Governor’s Office). The partnership would identify a (i) funding source for project end payments for an evidence-based program and (ii) a service provider to successfully execute the intervention. The intervention must successfully address the needs of county probation youth in a cost-effective manner, and, among other outcomes, achieve fewer new criminal referrals, reduce out-of-home placements, engage or reengage youth in education or vocational training, and support a stable home. Marion County 4 Retrieved from http://www.thirdsectorcap.org/wp-content/uploads/2015/04/SIF-Sub-Recipients_FINAL_03.11.15.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 5  The Marion County Juvenile Department, led by Faye Fagel, encompasses four programs: (1) Case Management, (2) Accountability and Risk Reduction, (3) Counseling, and (4) Administration. The Juvenile Department oversees a $12 million budget with 103 employees, and manages grants averaging $1.1 million annually. Marion County’s Business Services, Information Technology, and Finance departments oversee financial, purchasing, and contracting processes. Contracts are reviewed and approved by the Chief Administrative Officer, legal counsel, and the elected Board of Commissioners. Marion County adheres to the highest standards in financial reporting and processes, receiving awards for budget and finance documents. Marion County has been recognized by the U.S. Bureau of Justice Assistance for collaborative work in the field of prisoner reentry. Through evidence-based strategies and partnerships involving public sector agencies, faith organizations, nonprofits, businesses, and elected leaders, Marion County reduced recidivism for adults released from Oregon prisons from 36% in 2002 to 14% in 2014. This work required significant community engagement, quality programs, trained staff, and systems alignment among housing, substance abuse treatment, education, parole and probation, and employment agencies and institutions. Close working relationships with Congressional and state legislators garnered resources and policy changes that supported reentry work. Similar approaches to expand the continuum of youth services available at the community level to strengthen family systems will be applied through performance-based contracts with youth service providers, and with support from business, faith, and community leadership. Innovative programs include a partnership with Child Welfare to prevent youth with histories of abuse and neglect from crossing over and penetrating deeper into the juvenile justice system. Executive leadership and oversight for the project was provided by Commissioner Janet Carlson and her staff. Multnomah County The Multnomah County Juvenile Services Division is housed in the Department of Community Justice. The division provides a continuum of services ranging from informal handling (diversion) and formal probation, shelter care, electronic monitoring, mental health assessments and care coordination, outpatient treatment staff and detention. With a 2016 budget of $31 million and 184 support and front line full time staff, the division completed over 1,300 detention screens in 2013 and annually diverts roughly 800 youth from court. In addition to the team assembled for this project, other county commissioners and departments have researched the efficacy of PFS efforts initiated through the Corporation for National and Community Service. The Chief Financial Officer produced a report5 in early 2014 analyzing the PFS model’s alignment with county systems based on existing pay for performance projects, including historical background, financing options, legal challenges, and impacts on budgeting, outlining a path forward that requires additional input and policy analysis – the work of this project. In addition, the county Budget Office has a lengthy history of implementing innovative budgetary processes, including the use of Priority Based Budgeting. 5 “Social Impact Bonds: A look at the potential benefits and challenges for Multnomah County.“ Multnomah County. August 2014 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 6  Multnomah County has been at the forefront of innovative public safety initiatives for the past two decades. The County formed a Local Public Safety Coordinating Council in 1995 as a forum to discuss pressing public safety issues across jurisdictions and agencies, spawning the creation of Decision Support System-Justice in 1998 as a common public safety data warehouse to integrate public safety data from local sources. Multnomah’s Department of Community Justice Juvenile Division integrates a number of evidence-based services into their system design. Functional Family Probation is a family-focused, integrative model for supervision and case management of youth and families that replaces the traditional individual probation model with supervision provided in the community. Multnomah County is also one of five model sites around the country participating in the Juvenile Detention Alternative Initiative, serving as a learning laboratory for objective risk assessment, supporting alternatives to detention such as police diversion, community monitoring and a youth reception center, collaborating with system partners including juvenile court, law enforcement, public health, community-based providers, advocates, youth and families; and combating racial and ethnic disparities. Finally, the County completed the federal Office of Juvenile Justice and Delinquency Prevention Comprehensive Gang Model assessment in 2014 and is working across multiple jurisdictions to increase efficiency and effectiveness on a range of strategies and services to reduce and prevent youth violence and trauma. Executive leadership and oversight for the project was provided by Commissioner Judy Shiprack and her staff. PFS Feasibility Assessment Structure and Timeline Third Sector announced its sub-recipient awards on March 11, 2015, and began working with a core team at both counties during the second quarter of 2015. The project Steering Committee and Working Groups (detailed in Appendix A: Membership in Feasibility Assessment Groups) focused on the following:  Creating and executing a work plan for the feasibility assessment  Reviewing findings and providing detailed feedback and guidance  Coordinating all activities via weekly meetings Key Milestones  June - July 2015: Project due diligence data gathering from Marion, Multnomah, and Youth Villages6  July: PFS Educational Webinars conducted with all Working Groups  August: Kick-off meeting  October: Completed Special Purpose Vehicle Formation project guidelines 6 Youth Villages operates the Intercept program, a national evidence-based, intensive in-home service that looks to strengthen and restore families, and help children safely remain in their homes or reunify with their families. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 7  October-November: Prioritization and identification of risk factors for selection and payment-linked outcomes  February 2016: Project priority shifted toward serving Marion County youth only since baseline data for Multnomah County youths highlighted proposed project would not be financially viable. Multnomah continued in an educational capacity.  February: Completed PFS financing and appropriation project guidelines  March: Finalized Target Population Assessment  March: Participants in the first SIF Learning Community: Data & Evidence hosted by Oregon Health & Science University to share feasibility study lessons learned to other organizations conducting PFS feasibility studies nationwide  May: Completed Counterfactual and Cost/Benefit Analysis  June: Completion of service provider Request for Information template and Guideline for Evaluator Requirements  July: In partnership with the Oregon Community Foundation, hosted a philanthropic convening, “State of Oregon Pay for Success & The Role for Philanthropy,” to discuss project alignment with philanthropy priorities and opportunities for promoting innovation and building common infrastructure Guiding Objectives for PFS Feasibility Assessment This PFS feasibility assessment was customized to fit the County’s needs. Below is an overview of each major work stream and the guiding questions that were addressed: Data Source & Access Assessment: Do we have ongoing access to quality information to track outcomes of interest? Target Population Assessment: Who exactly do we want to serve and why? How is the population currently being served and what is its performance against "baseline" outcomes of interest to end-payers? Intervention Assessment: Are there interventions that can positively impact the population we want to serve? What is the scaling opportunity? Do they track outcomes that are a priority for potential end-payers? What is the recommended evaluation method? End Payer Assessment & Economic Model: What is the economic case for the PFS project? Are there end payers committed to paying for success achieved? Legal & Regulatory Assessment: What are the options for end payer(s) and service provider(s) to enter into multi-year PFS contracts? What are the options for end payer(s) to commit and appropriate success payments? Funder Development: Is there funder interest in continuing to develop a PFS project and invest in the up-front funding required by a PFS contract? Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 8 At the onset of the project, the Counties reached a preliminary understanding with Youth Villages of Oregon to assess the juvenile justice PFS model as a “proof of concept.” Youth Villages operates the Intercept program, a national evidence-based, intensive in-home service that looks to strengthen and restore families, and help children safely remain in their homes or reunify with their families. Given the required time and resource investment to assess both County’s target population’s risk criteria and baseline outcomes, and the cost / benefit implications of a provider’s cost of service delivery and successful impact, neither Youth Village nor any other service provider or specific intervention were selected as definitive parties to contract with. Section: Intervention Assessment focused on developing government capacity and tools to select an appropriate intervention for the project and aligned objectives of project participants. In the absence of a service provider, Section: Legal & Regulatory Assessment and Section: Funder Development focused on developing guidelines for the Counties to rely on in the event a project were to commence. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 9 III. Executive Summary and Overview of Work Streams Executive Summary The feasibility assessment provided various opportunities to foster innovation within the Counties. Under the leadership of Marion County Commissioner Janet Carlson and Multnomah County Commissioner Judy Shiprack, the Counties continue to recognize the challenge of annual budgeting cycles to fully support critical multi-year social service functions. Through learnings of the feasibility assessment, the Counties are committed to pioneer alternative funding strategies that use evidence-based practices to connect public funds to program outcomes rather than outputs. The Counties have a shared goal of using evidence-based models that can be replicated locally as well as across systems statewide. Whether it is juvenile justice or other public service areas, both governments wish to incorporate lessons learned from the study to improve internal capacity and data analytics, and various executive, administrative, and departmental processes. The feasibility study process brought together a diverse set of representatives, including elected county and state officials, county juvenile units, service providers, state agencies, academia, data analysts, and county counsel. The project team also worked and shared information with the other Oregon based Pay for Success studies. This team-oriented approach was intended to promote a new collaborative and outcome-based procurement culture. The promise of Pay for Success (funding programs over a period of years based on outcomes with in depth evaluation) continues to serve as an appealing option for the Counties. For example, Multnomah County is participating in another feasibility study in concert with the City of Portland, examining how the model could be used to fund low income housing improvements. For just over a year, a multi-disciplinary team from Marion and Multnomah engaged in a thorough evaluation of their juvenile justice system, as well as an analysis of the legal and procurement requirements for instituting a Pay for Success model. This was not a simple or easy process. It required detailed and rigorous examination of the current methods for assessing criminogenic risk factors for youth and available services to address them; assessment of the current costs of services; the potential economic and programmatic benefits of alternatives; and determining the legal and administrative framework for a Pay for Success project. The feasibility assessment found a significant investment in time and relationship building is required by the Counties to identify a willing and able state-level entity to serve as a potential revenue source for project end payments. End payments could be structured as a direct appropriation (annually or multi-year), a partial withhold contingent on service provider performance, and/or an authorized commitment via new legislation. During the engagement, it was concluded that the best path to securing an end payer would be via a joint project proposal with the Oregon Youth Authority (OYA) to the Oregon State Legislature, Executive Office (or other state agencies), or philanthropic organizations. The Counties believe that an intensive in-home intervention for emotionally and behaviorally troubled probation youth and their families is suitable for the project. Assuming project outcomes are an Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 10 improvement over existing government programs for the target population, such end payments would cover full project cost. Key conclusions from the highest priority works streams include: Marion and Multnomah have different risk criteria for selection and placement, which make a joint-project difficult to achieve at this stage. Marion is best suited to proceed given its substantial target population headcount and precise screening criteria. The target population is defined as youth 13-18 years of age previously placed on probation. The individual must have been identified as “Medium” to “High” risk, and at risk of out-of-home placement. High priority project outcomes include: (i) reductions in out-of-home placements to county and OYA facilities, (ii) reduction in criminal activity, and (iii) improvement in social impacts of juvenile criminal activity. Scalability of the project is promising. Annually an estimated 115 Marion youth on probation would meet criteria for intervention referral. Given voluntary consent and service provider admission estimates, total annual headcount is projected at 75. OYA is an essential partner in making the case for state-level end payments for an intervention to reduce the costliest placements and achieve better outcomes for youth and their families. OYA and the state stand to benefit the most from a successful intervention serving the target population. State-level contributions to end payments are crucial to the advancement of the project. End payments could either directly flow to a county or be administered by OYA (or another state agency), and would be tied to successful diversion from OYA placements, and potentially GAP placement and/or county detention. A county-only funded project does not result in a favorable cost/benefit assessment. The benefits that would accrue to the county are insufficient to justify end payments large enough to treat the entire target population. Results from conducting a non-binding request process would help the county demonstrate there are viable service providers operating in the community that can potentially meet the outcome expectations of the project. Once identified, the project end payer can work closely with the county, OYA, and other project partners to ultimately select a preferred service provider and better assess the project’s cost / benefit proposition. Readiness Scale and Overview of Work Streams Each work stream received a rating to reflect readiness to move forward with implementing a PFS project. The readiness levels (described below) reflect the amount of time, funding, and groundwork required to develop and launch a Pay for Success project. Further, the levels are intended to show the relative readiness of certain work streams against others. The selection of a readiness level was done by Third Sector and reviewed by the Steering Committee for the project. Po pul ati on En d Pa yer Int erv ent ion Not Yet Demonstrated Possible Promising Strong Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 11  Not Yet Demonstrated: Progress on the goals of the work stream was made, but goals were not completed, due to challenges in data access and/or timing constraints of the feasibility assessment. Further work is required to assess readiness for this area.  Possible: A significant amount of work is needed to implement a PFS project for this work stream.  Promising: A moderate amount of work is needed to implement a PFS project for this work stream.  Strong: A minimal amount of work is needed to implement a PFS project for this work stream. Exhibit 1 presents an overview of the work stream areas assessed, the readiness level, and the major findings from the feasibility assessment of that work stream: Exhibit 1: Summary Findings of Work Streams Assessment Readiness Level Scale Summary Findings Data Source & Access Promising  Since Counties administer all juvenile justice activity at youth intake, they are well equipped to access and analyze the data required for a performance-based contract evaluation.  Oregon’s Juvenile Justice Information System, namely the Juvenile Crime Prevention Assessment, is the appropriate data source for any project requiring risk criteria for youth selection, outcome tracking, and program evaluation. These data sources are suitable due to its rigorous tracking of relevant project data and common usage among county and state-level entities.  The Counties, in partnership with OYA, have a wealth of information from government, but limited corresponding data from service providers. Target Population Promising  The target population is defined as youth 13-18 years of age living in Marion or Multnomah and previously placed on probation. The individual must have been identified as “Medium” to “High” risk using the Juvenile Crime Prevention Risk Assessment, and at risk of out-of-home placement.  High priority project outcomes include: o Reductions in out-of-home placements, particularly commitment to OYA which is costly and longer term o Reduction in criminal activity o Improvement in social impacts of juvenile criminal activity  There are an estimated 115 and 135 probation youth that would be eligible for intervention enrollment for Marion and Multnomah, respectively. On average approximately 30% of Marion’s annual target population escalate to OYA level placements. For Multnomah, the estimate is approximately 10%.  Marion and Multnomah have different risk criteria for selection and placement, which make a joint-project difficult to achieve at this stage. Marion is best suited to proceed given its substantial target population headcount and precise screening criteria. Intervention Not Yet Demonstrated  Scalability of the project is promising. Given voluntary consent and service provider admission estimates, annual headcount for enrollment could be 75 in Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 12 Marion and 100 in Multnomah.  A Request for Information (RFI) document has been created so that at the Counties’ choosing, a vetting process can be initiated to understand Oregon’s landscape of service providers and their capabilities to meet project goals (Appendix F).  In the absence of a specific service provider and designated end payer, a specific evaluation plan was not developed. However, this report includes a description of the roles and essential work streams to be undertaken by a project evaluator (Appendix G). Economic & End Payer Promising  The costliest placements are at the state level. OYA is an essential partner in finding a source of state level end payments to capture the benefits of reduced escalation to state-level placements.  The cost/benefit analysis demonstrates the important interaction between assumptions about the status quo outcomes of the target population, an intervention’s cost, and the intervention’s effectiveness. The analysis provides a framework for selecting an intervention that can reasonably be expected to succeed.  County-level measures for the feasibility study do not predict the risk of escalation from probation to a residential placement by placement type (e.g. detention, residential, and OYA). This means some youth would receive intervention who are not at risk for the highest cost outcomes valued by potential end payers, which currently dilutes the cost/benefit analysis. Legal & Regulatory Not Yet Demonstrated  Three potential project funding and payment approaches to consider are Traditional PFS, Partial PFS, and Performance-Based Contract with Deferred Payments. The first two options would require third party funding while the last alternative could be funded using traditional procurement mechanisms.  Three strategies for guaranteeing end payments include full faith and credit, appropriations (multi-year or annual), or rating agency trigger funding mechanisms  For this project, the holding mechanism for success payments or if performance- based contracts that cross biennia are possible in the state are unknown. Funder Development Not Yet Demonstrated  Due to not having a committed end payer and identified intervention model or service provider, it is unclear if the project would require third party upfront funding. Detailed engagement with local third party funders is at this phase premature  As the Counties identify a potential end payer, continued identification of prospective funders can help determine interest in financially supporting specific project work streams (e.g. data building efforts or evaluation) or providing upfront funding for the entire project (if needed). Recommended Next Steps  Secure commitment of state-level end payer to assist in the development and design of the project, with the objective of contributing to all or a portion of project end payments.  Conduct a non-binding RFI process to demonstrate there are viable service providers operating in the community that can potentially meet the outcome expectations of the project. Once identified, the project end payer can work closely with the county, OYA, and other project partners to ultimately select a preferred service provider and better assess Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 13 the project’s cost / benefit proposition. This process, despite the fact that an end payer has not yet been secured, would help propel the project forward.  Conduct RFI process to gauge organizational data capacity for service providers and cost- effectiveness requirements. The cost/benefit framework of the feasibility assessment was developed without reference to a specific intervention. As such, it may not reflect the cost structure and measured impact of providers able to serve the county. The county should use an RFI process to learn more about the capabilities and cost of interested providers.  Evaluate the benefit of incorporating OYA’s predictive data resources into risk criteria used for referring a youth into an intervention. Predictive data could help target youth with the highest need and identify those likely to escalate to the costliest out-of-home placement.  Incorporate lessons learned from evaluation and target population work streams to improve existing government programs serving probation youths.  Continue identifying funders who have exhibited strong interest in supporting a potential project. Potential PFS projects should leverage the local funder community to provide project funding or work stream support (e.g. evaluation), and to demonstrate community support for the project goals and intervention. For more detail on the findings and next steps for each of the work streams in the feasibility assessment, please refer to the detailed sections below. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 14 IV. PFS Feasibility Assessment Work Streams Data Source & Access Assessment: Promising Not Yet Demonstrated Possible Promising Strong Do we have ongoing access to quality information to track outcomes of interest? Work Stream Goals:  Assessment of data needs & data sources  Data plan development for collecting, analyzing, and acting on data  Incorporation of data from service provider(s) Conclusions:  Since Counties administer all juvenile justice activity at youth intake, they are well equipped to access and analyze the data required for a performance-based contract evaluation.  Oregon’s Juvenile Justice Information System’s (“JJIS”) is the appropriate data source for any project requiring risk criteria for youth selection, outcome tracking, and program evaluation  For the feasibility assessment, the Working Groups exclusively relied on risk scores derived from JJIS’ Juvenile Crime Prevention (“JCP”) Assessment 2006.1. The JCP is a suitable data foundation for the project due to its rigorous tracking of relevant project data and common usage among county and state-level entities.  The Counties, in partnership with OYA, have a wealth of public sector data, but limited corresponding data from service providers. Since a specific service provider or intervention was not selected for pilot development (See Section: Intervention Assessment), the feasibility assessment was unable to fully evaluate service provider data, any provider’s access to government administrative information, or intervention impact to treat the target population. Recommended Next Steps:  Conduct Request for Information (“RFI”) process to gauge organizational data capacity for service providers  Evaluate the benefit of incorporating OYA’s predictive data resources into risk criteria used for referring a youth into an intervention  Integrate juvenile justice outcomes with educational and/or health outcomes  Consider expanding internal county capacity for data analysis or forming new data partnerships Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 15 The Data Assessment & Access work stream was focused on evaluating the following: I. Assessment of data needs & data sources: Data needed from county and state in order to build a performance-based contract. II. Development plan for collecting, analyzing, & acting on data: Methods for efficient and reliable processing of government data. III. Incorporation of data from service providers: Collection and analysis of data from service providers, including youth recruitment and tracking. I. ASSESSMENT OF DATA NEEDS & DATA SOURCES The juvenile departments at each county devoted substantial time and resources accessing county and state level data of their target population from a centralized database, Oregon’s Juvenile Justice Information System (JJIS). JJIS is a collaborative initiative of the state’s Oregon Youth Authority (“OYA”), the 36 county juvenile departments, and other juvenile justice and public safety partners. JJIS offers a single source of case record information for all youth within the juvenile justice system. The law enforcement referrals on every youth taken into custody or referred through a police report come to the local County Juvenile Department. Juvenile Departments have the responsibility to determine, in conjunction with the District Attorney’s Office, the appropriate response to address the behavior that brought the youth to police attention, mitigating risk factors, and ability to increase skills for long term youth success and public safety. Roughly 94% of all youth referred stay with the local County Juvenile Department. A small portion of youth referred are court committed to OYA either for placement in a Youth Correctional Facility or placement in a residential treatment program. OYA funds, and initiates and oversees the system of residential programs across the state. JJIS entries are initiated by whoever has contact with a youth, either county juvenile departments or OYA. JJIS’s consistent tracking of data for over 15 years and common use by Marion, Multnomah, and OYA supports its use as the appropriate data source for any project requiring risk criteria for youth selection, outcome tracking, and program evaluation. The data system essentially accounts for three key functions that any performance-based contract project would require:  Aids in planning, developing, and evaluating services designed to reduce juvenile crime  Provides comprehensive case information about juveniles and the services they receive from county juvenile department and Oregon Youth Authority.  Assists county juvenile departments and Oregon Youth Authority in managing individual juvenile cases and in tracking juveniles through the system For all Juvenile Justice System youth, JJIS provides information such as demographics, allegations, adjudications, court conditions, incident reports, and treatment services, and commitments to Youth Correctional Facilities. In Marion County individual youth case plans are Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 16 also completed and kept in JJIS. Of most relevance to the project, the data platform tracks risk assessments of county probation youth (administered at intake and every six months thereafter), new criminal referrals, probation violations, residential commitments (including length of stay), and court ordered restitution amounts and amount paid. JJIS tracks an individual youth’s status, progress, program involvement and outcomes through the justice system. The database enables juvenile justice personnel at the county and state level to analyze the risk criteria, outcomes, and cost data requirements associated with any performance-based contract and/or evaluation. II. DATA PLAN DEVELOPMENT FOR COLLECTION, ANALYZING & ACTING ON DATA JJIS contains the Juvenile Crime Prevention Risk Assessment Instrument (JCP) used by all Juvenile Departments, and the OYA Risk/Needs Assessment, used by OYA and some Counties. The JCP and the Risk and Needs Assessment generate information that assist in decision- making for appropriate supervision levels, interventions, programs, services and dosage, readiness for transition, and support program evaluation.7 For the feasibility study, the Working Groups exclusively relied on risk scores derived from the Juvenile Crime Prevention Risk Assessment (JCP) 2006.1. The JCP was used to determine the risk and needs profile of youth for selection into the intervention, and would serve as the information basis for any program evaluation. The JCP is a suitable data foundation for various reasons: Rigorous Risk and Outcomes Tracking: JCP assessment includes 30 risk factors organized into seven social areas, all of which are relevant to the youth risk factors addressed with the project:  Behavioral issues, including data on school behavior, prior criminal activity, runaway history, violent and/or aggressive behavior, and weapon usage  Substance use  Peers and peer activities  Family dynamics  Attitudes, Values and Beliefs  Education / academic issues  Mental health The risk criteria required for this project are also contained within JJIS. These include number and type of criminal referrals, probation violations, response to interventions and services, placement out of home in the County Shelter Care or OYA placements in residential programs or Youth Correctional Facilities, and length of stay. Supplementary outcomes that are educational (e.g. school attendance, academic performance), 7 “OYA Recidivism Risk Assessment: Modeling Risk to Recidivate” Oregon Youth Authority. Jan. 2011. Retrieved via https://www.oregon.gov/oya/research/recidivismriskassessment_modelingrisk.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 17 behavioral (e.g. improved pro social skills and functions), or health oriented (e.g. hospitalizations) are not currently stored in JJIS. Credible Data Source: The JCP was implemented in 2000 for the state of Oregon. The tool is one of many risk assessment instruments used throughout the country with high marks in validity, reliability, and equity, including working equally well across different racial and ethnic groups.8 Common Usage: The JCP is used to assess the risk level of youth based on criminogenic risk factors tied to criminal behavior. The JCP is unique to Oregon in its format but was developed using data and research on factors common in other risk tools around the country. All 36 counties and the Oregon Youth Authority have agreements to use JJIS. They work together to define how information is entered and determine what is mandatory, while partnering on a steering committee that establishes policies and standards. Common usage provides the project ample flexibility for implementation in Marion, Multnomah, or any other county (or combination of counties). Note that the feasibility study did not assess Department of Corrections data bases, or any information related to adult activity of individuals previously involved with the juvenile justice system. III. INCORPORATING DATA FROM SERVICE PROVIDERS At the onset of the project, the Counties reached a preliminary understanding with Youth Villages of Oregon to use the juvenile justice PFS model as a ”proof of concept.” Youth Villages operates the Intercept program, a national evidence-based, intensive in-home service that looks to strengthen and restore families, and help children safely remain in their homes or reunify with their families. As discussed in Section: Intervention Assessment below, Youth Villages is not the sole service provider that could be considered for a performance-based contract under this study. However, the organization was an active partner that provided the feasibility assessment with a realistic and experienced perspective for various work streams. The internal data sets maintained by Youth Villages’ Intercept program were not reviewed or evaluated to validate impact. From a data perspective, Youth Villages’ provided general insights on its internal data collection process, assessment of youth and family needs, and the recording of outcomes achieved by youth that successfully complete the Intercept program. The Counties, in partnership with OYA, have a wealth of public sector data, but limited corresponding data from service providers. Since a specific service provider or intervention was 8 “A Comparison of Risk Assessment Instruments in Juvenile Justice.” National Council on Crime and Delinquency, August 2013. Retrieved via https://www.ncjrs.gov/pdffiles1/ojjdp/grants/244477.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 18 not selected for pilot development (See Section: Intervention Assessment), the feasibility assessment was unable to fully evaluate service provider data, any provider’s access to government administrative information, or intervention impact to treat the target population. Recommended Next Steps: 1) Conduct Request for Information (“RFI”) process to gauge organizational data capacity for service providers As detailed in Section: Intervention Assessment, surveying the service provider landscape will identify which Oregon organizations may be suitable to partner with on a performance- based contract. Their participation in the planning process may also highlight any data shortcoming that government and service providers need to address before finalizing project design. A common challenge in reviewing service provider data is the potential gap between government outcomes of interest and provider-tracked outcomes. For example, outcomes of interest for this project are length of stay per County and/or OYA out of home placement, criminal referrals, and victimization costs. However, service providers typically only collect output data, which includes who they serve (i.e. participant information), fidelity measures (e.g. program completion), and leading indicators or proxies for project outcomes (e.g. interactions with the law vs. criminal referral). An RFI process would help address this potential gap. Examining the data capabilities of potential providers will help clarify how much effort is needed to create data sharing agreements between them and government. Such agreements are typically developed in any performance-based contract so outcomes delivered by service providers can be confirmed with datasets from the end payer (whichever government entity it may be). Provider data linked to state systems help in the evaluation of performance based contracts. These steps will compare County systems to those of service providers, enabling the project to:  Align prioritized JCP risk criteria and service provider intake requirements  Confirm service provider outcomes are equivalent to project outcomes, as prioritized by an end payer  Validate success measures of an intervention 2) Evaluate the benefit of incorporating OYA’s predictive data resources into risk criteria used for referring a youth into an intervention Criteria for selecting project participants is currently based on the JCP assessment. By supplementing the JCP with the Escalation to OYA tools, the counties might strengthen project design and evaluation. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 19 Escalation to OYA Escalation scores estimate the likelihood that a youth will escalate from county probation to OYA, either in a community placement or a youth correctional facility. While OYA uses several tools to estimate a youth’s likelihood of recidivism (e.g., Juvenile Crime Prevention Assessment (JCP), Risk Needs Assessment (RNA), etc.), OYA research has found that what predicts recidivism is not the same as what predicts escalation. The Escalation to OYA tool is a data driven approximation with the capability to estimate a youth’s likelihood to escalate to OYA. Each youth on county probation with an updated JCP will have different scores within the Escalation to OYA tool:  Escalation on a new crime - Predicts the likelihood a youth will escalate from county probation to OYA on an allegation that was not associated with placement on county probation  Escalation on a probation violation - Predicts the likelihood a youth will escalate from county probation to OYA on the same allegation associated with placement on county probation  Percentile Ranks - Indicates the percentage of youth on county probation in the same county who will have an equal or lesser Escalation to OYA score for escalation on a new crime and probation violation OYA is developing an improved Escalation to OYA score that predicts the risk of OYA placement that does not distinguish between probation violations and new crimes as causes for escalating. That tool is not yet available to counties throughout the state, but, when available, will be a recommended risk score to evaluate as a complement to the Counties’ JCP score to identify and select youth well suited for this project. It is recommended to evaluate the use of Escalation to OYA scores to assess if the Counties can more accurately prioritize youth for enrollment. As discussed in Section: Target Population Assessment, the current criteria make it likely more youth would be eligible than there is capacity to serve. Predictive tools could help target youth with the highest need. Second, and perhaps more importantly for justifying the project, Section: Economic & End Payer Assessment highlights that the more targeted an intervention enrollment can be, the better the cost / benefit economics. As discussed in Section: Economic & End Payer Assessment, Marion and/or Multnomah (or any other county) need to continue to work with OYA for a successful launch of a project. From a data perspective, that would also include leveraging existing OYA predictive tools for youth enrollment into the program. By fully leveraging data resources currently available from the state, the Escalation to OYA Score tools might strengthen the data quality of the project—both for program design and evaluation purposes. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 20 3) Integrate juvenile justice outcomes with educational and/or health outcomes The feasibility assessment is primarily based on JJIS information. Section: Target Population Assessment below describes youth with a specific risk profile that would generate benefits beyond juvenile justice outcomes with successful diversion of an out of home placement. Whether it is improved school attendance, academic reading, or reduction in drug abuse treatment services associated with successful completion of the intervention, the project data sets currently cannot validate those additional outcomes. A project could be successful if primarily anchored by JJIS data, but the upside could be larger if integrated with other administrative datasets. Should the Counties look to improve the cost/benefit analysis, or document wider benefits to promote state support, it is recommended they establish data sharing agreements or intergovernmental partnerships with the Oregon Department of Education, Department of Human Services, and/or Oregon Health Authority. Integration of various administrative sources would provide a unified data narrative of the Target Population (beyond JJIS) for youth that access various government services. The Oregon Youth Authority (an active project participant) and the Center for Evidence- based Policy at Oregon Health and Science University (in-state PFS practitioner) have already undertaken a significant cross-governmental data merging effort for their own internal analyses. Both are recommended partners for further data integration. 4) Consider expanding internal county capacity for data analysis or forming new data partnerships While two to four Working Group members led the data gathering and analysis efforts for the feasibility assessment, a launched project would require increased internal capacity due to the data demands of a performance-based contract. Whether it is risk score assessments for enrollment, scenario analyses, evaluation, contract compliance, and/or program improvement, more data staff would be required. The Counties might partner with OYA’s Research & Evaluation team or an organization like the Center for Evidence-based Policy at Oregon Health & Science University. OYA’s staff includes various statistical and evaluation experts that can improve the data capacity and analytics of the project. The Center for Evidence-based Policy is already intimately familiar with PFS principles,9 and has previously demonstrated interest in supporting Marion and Multnomah’s performance-based contract initiative. Through its own PFS project, the Center for Evidence-based Policy’s data efforts uniquely position it as a national leader among research institutions in integrating disparate state-level datasets and consolidation of youth outcomes. If the Counties wish to pursue Next Step #3 above (“Integrate juvenile justice outcomes with educational and/or health outcomes”), the Center for Evidence-based Policy has 9 The Center for Evidence-based Policy completed its PFS feasibility assessment in 2016 with Third Sector Capital Partners. The engagement was similar Marion and Multnomah’s engagement described in this report. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 21 already negotiated data sharing agreements with various state agencies, including the Oregon Department of Education and OYA, and can facilitate data integration with JJIS data histories of the target population Though it may be less cost effective, a final option would be to engage third party data consultants or private-side data specialists. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 22 Target Population Assessment: Promising Not Yet Demonstrated Possible Promising Strong Who exactly do we want to serve and why? How is the population currently being served and what is its performance against "baseline" outcomes of interest to end- payers? Work Stream Goals:  Examination of participant flow  Defining the target population Summary of Findings:  The Counties seek to reduce the number of high-risk probation youth, particularly, youth of color, from escalating further in the juvenile justice system. In partnership with Multnomah, Marion strives to do so through an expansion of county-level alternative programming, including those that are therapeutic in nature for youth adequately suited for such services.  Target population is defined as youth 13-18 years of age living in Marion or Multnomah and previously placed on probation. The individual must have been identified as “Medium” to “High” risk using the Juvenile Crime Prevention Risk Assessment, and at risk of out of home placement.  High priority project outcomes include: (i) Reductions in out-of-home placements, particularly commitment to OYA which is costly and longer term (ii) Reduction in criminal activity (iii) Improvement in social impacts of juvenile criminal activity  There are an estimated 115 and 135 probation youth that would be eligible for intervention enrollment for Marion and Multnomah, respectively. On average approximately 30% of Marion’s annual target population escalate to OYA level placements. For Multnomah, the corresponding figure is estimated to be approximately 10%.  Marion and Multnomah have different risk criteria for selection and placement, which make a joint-project difficult to achieve at this stage. Marion is best suited to proceed given its substantial target population headcount and precise screening criteria. Recommended Next Steps:  Evaluate benefit of OYA’s predictive data analysis resources into risk selection for target population for both Counties  Conduct multi-year forecast of the target population referred to an intervention  Modification of Multnomah’s risk criteria for target population selection is required for a PFS project, regardless of project structure (stand-alone or in partnership with Marion). Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 23 The Target Population Assessment was focused on addressing the following: I. Examination of population flow: Assessing how this population is currently being served and defining outcomes of interest a project end-payer is willing to fund. II. Defining the target population: Determining who the Counties want to serve and why. Selection process and size of potential cohort is developed by assessing the needs and risk profile of the youth to be served. I. EXAMINATION OF POPULATION FLOW Various risk factors can be identified at a youth’s first interaction with the juvenile justice system that can help prevent further penetration. The funding for statewide residential services rests with the Oregon Youth Authority (OYA). Marion County has a limited number of county residential placements options (only 30 slots available) and no alternatives that are therapeutic in nature for youth suited for such services. To access other residential programs, a youth must be committed by the court to OYA. This results in a youth being removed from the home and local community. There are currently no OYA contracted residential programs in Marion County. 10 The following provides summary information on Marion youths placed in OYA facilities in 2014:11  Historically, Marion county probation youth eventually committed to OYA facilities face various challenges before being removed from their homes o 82% are failing academically o 70% have been suspended or expelled o 46% have already dropped out of school. o 60% of youth also qualified for free or reduced lunch programs in 2013 o 84% were assessed as providing poor supervision at home o 71% reported serious family conflict o 66% have a family history of child abuse/neglect or domestic violence o 57% have a family member struggling with substance abuse).  These youth demonstrate chronic aggressive behavior that began prior to age 13 (68%), have runaway history (50%), behaviors that hurt or put others in danger (84%) and/or themselves (79%). Hispanic youth are disproportionately affected by this lack of resources. Marion County has the state’s second highest Latino/a population (81,851), which accounts for 25% of the county’s total population. Marion has the largest number of Hispanic youth in the juvenile justice system in the state. In 2013, Latino/a youth comprised approximately one third of all Marion County criminal referrals. 10 Marion County 11 Marion County. Similar trends are comparable for 2015. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 24 This data analysis is consistent with observations in Multnomah County. In Multnomah, African American, Latino and Native American youth are disproportionately impacted by the juvenile justice system.12  African American youth, specifically, are brought to a detention facility at higher rates in comparison to white youth (20.3% to 14%), and are less likely to be released (48.8% compared to 68.3%).  African-Americans are four times more likely to be referred to juvenile court than white youth, and nearly two times more likely to have cases resulting in placement in a secure juvenile correctional facility.  Hispanics are 1.5 times more likely than white youth to be involved in a case requiring secure detention.  Despite many efforts to the reverse disparities over the last ten years, the situation appears to have worsened in the last few in both counties. Multnomah County’s minority population is 205,031 or 28% of the county’s total. In 2013, of 3,288 criminal referrals to Multnomah County, 1,812 or 55% were for youth of color (African American, Hispanic, Native American and Asian). Eligibility Criteria for Project The population to be served in this proposed project is comprised of a subsection of the youth described above. For the purposes of the feasibility study, the project eligible population includes youth on county juvenile department probation and at-risk of out-of-home placements: OYA Commitments  OYA Youth Correctional Facility (“YCF”)  OYA Residential Treatment (“OYA Residential”) County Placements  County Residential (In Marion, known as Guaranteed Attendance Program, or “GAP”)  County Detention (“Detention”) For a description of each of these placements, please see Appendix C. A successful project would be one where at disposition to county probation youth would be referred into an intense, in-home intervention that would serve as an adjunct to probation as usual. 12 Multnomah County and Feyerherm, W. (2012, November). “Disproportionate Minority Contact in Oregon’s Juvenile Justice System: Identification and Assessment Report.” State of Oregon. Retrieved July 2016 via http://www.oregon.gov/oya/dmcsummit/2014/Materials/OregonDMCReport2012FINAL.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 25 The four out-of-home placements and corresponding decision-making are highlighted in Exhibit 2. It is important to note the population being assessed in the feasibility study excludes youth with Ballot Measure 11 adult charges (mandatory remand to adult court). EXHIBIT 2: Overview of Oregon’s Juvenile Justice System Source: Oregon Youth Authority (emphasis and formatting added) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 26 EXHIBIT 3: Project Focus Within the Juvenile Justice System  Juvenile justice departments receive all referrals from local law enforcement.  The JCP assessment tool (discussed in Section: Data Source & Access Assessment) is typically completed by probation officers when a youth is proceeding to adjudication. All counties enter youth information to the state-wide JJIS system throughout the youth’s active case, and use that information for direct service supervision and case planning  Juvenile departments, in partnership with local district attorneys and juvenile courts, determines whether to: (i) Proceed informally in lieu of formal court action, or (ii) In the case of more serious referrals, file petitions before the juvenile court  The feasibility assessment focused on the group of youth processed in the latter (ii) determination  When youth are referred for more serious offenses (ii, above), juvenile department staff work with the local district attorneys to file delinquency petitions in the juvenile court so that the case can be adjudicated  As discussed in the Section: Data Source & Access Assessment, it is important to note that for the feasibility study, the JCP alone may not provide the Counties the most predictive forecast of the likelihood that a youth may ultimately be placed in out-of- home placements.  Youth would be referred to an intervention once adjudicated as delinquent and ordered on probation  Should the juvenile court establish jurisdiction (i.e. determine guilt for an act which would be a crime if committed by an adult), the judge may place the youth under formal County Probation.  Youth on county-level probation are subject to formal sanctions and requirements that are designed to provide a consequence for the crime committed, mitigate risk factors, and increase skills while including a range of supervision, accountability, and reformation services administered by the county.  If a judge approves the use of County Detention for a youth, an individual may be kept in detention custody as a sanction for probation violations, pre-adjudication to ensure a youth is present for court proceedings, or as the least restrictive alternative to ensure public safety.  Depending on county size, frequency of juvenile criminal activity, budget, and/or policy decisions, counties may have in-county programs (e.g. County Residential) for youth placed on County Probation.  For the purposes of the feasibility assessment, Marion’s County Residential program is the GAP Program, a voluntary service offered to families that is not court ordered.  Youth may be committed to the custody of OYA by a juvenile court if they: (i) Are unsuccessful in meeting conditions of County Probation, (ii) Are in need of a residential placement to establish structure, supervision and accountability not available in the home (iii) Commit very serious offenses, and/or (iv) Are found to be serious risk to community safety  OYA administers state Youth Correctional Facilities, manages contracts with community residential programs, provides treatment foster care, and provides parole and probation supervision for youth in those placements. These programs generally are not available to youth in their home counties. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 27 Detention, GAP and OYA commitments are costly services for government (e.g. facilities, staff, administrative processing) and traumatic for youth and their families. The target population has complex needs that cannot be adequately addressed using the existing continuum of resources. It is a project priority to ensure the target population participates in services designed to address the whole person, focusing on educational, health, and social/emotional outcomes, including: parent skill building, family engagement, collaborative and customized treatment, trauma informed care, and education interventions. The potential target outcomes the Counties strive to improve with a performance-based contract are listed in Exhibit 4 below. The listed outcomes are considered a top priority for any state-level government body that would serve as the end payer (as determined by the Steering Committee, including County and state-level leadership represented in the group): EXHIBIT 4: Outcomes of Interest End Payers Would be Willing to Pay For Placement Behavioral / Other Reduction in out-of-home placements OYA Commitments  OYA YCF  OYA Residential County Resources  County Residential / GAP  County Detention Reduction in criminal activity  New criminal referrals Improvement in social impacts of juvenile offending  Reduction in alcohol and other substance use  Increase in school attendance, credit accumulation, grades, pro-social behavior, and positive peer association II. DEFINING THE TARGET POPULATION It was agreed at the onset of the project that participants would meet the following minimum requirements: EXHIBIT 5: Minimum Requirement for Program Eligibility 13 “Medium to High” includes youth with Medium, Medium High or High designations Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 28  Living in Marion or Multnomah Counties  13-18 years of age  Placed on probation  Identified as “Medium to High” risk on the Juvenile Crime Prevention Risk Assessment (JCP)13  At risk of out-of-home placement  Note: See Appendix B for the definition of JCP risk criteria. As presented in Exhibit 6 below, selection for Marion’s target population included an analysis broken down into four specific subject areas for screening: Individual, Peers, Family, and School risk factors derived from the JCP risk assessment tool discussed in Section: Data Source & Access Assessment. EXHIBIT 6: Marion Target Population: Eligibility Criteria for Intervention Referral In addition to the Minimum Requirements for Program Eligibility in Exhibit 5, Marion participants must meet eight or more of the following 15 JCP items to be considered for eligibility: Individual – Early initiation of antisocial behavior, Early initiation of drug use, Rebelliousness, Substance use  JCP 6.1 Substance use beyond experimental use  JCP 6.2 Current substance use is causing problems in youth's life  JCP 7.3 Youth accepts responsibility for behavior  JCP 7.6 Youth pre-occupied with delinquent or anti-social behavior Peers – Interaction with antisocial peers, peer substance abuse  JCP 3.1 Friends disapprove of unlawful behavior  JCP 3.2 Friends engage in unlawful or serious acting-out behavior  JCP 3.3 Has friends who have been suspended or expelled or dropped out of school  JCP 3.5 Substance abusing friends Family-Family conflict/violence, neglectful parenting, parent history of mental health difficulties, parent stress, parental attitudes favorable to antisocial behavior, parental attitudes favorable to drug use, poor gamily management, violent discipline  JCP 5.1 Communicates effectively with family members  JCP 5.2 Poor family supervision and control  JCP 5.10 Has close, positive, supportive relationship with at least one family member School - Low school commitment and attachment, poor academic performance  JCP 2.1 Significant school attachment/commitment  JCP 2.2 Academic failure  JCP 2.7 Family actively involved in helping youth succeed in school  JCP 4.5 Involved in extra-curricular activities Note: Please see Appendix B for the definition of each of the JCP risk criteria listed above. The basis for using these 15 JCP items as risk criteria for youth selection include:  An historic analysis of Marion probation youth that had been committed to OYA showed the 15 items were highly prevalent (>65%) at the closest date of first commitment to OYA Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 29  These risk factors overlap with key screening criteria for enrollment into an intense treatment intervention like Multi-Systemic Therapy Based on the 2010-2015 historical probation youth that meet the eligibility criteria, 83% (184) of all Marion youth placed in OYA Residential and YCF had 8 or more of the 15 JCP risk criteria, 89% (198) had 7 or more, and 93% (206) had 6 or more. The analysis found that Marion youth who went to OYA Residential or YCF and had less than 8 of the 15 JPC risk criteria, had special needs only available through OYA commitment: (a) sex offense youth going to sex offender-specific treatment only, (b) focused substance treatment youth, and (c) youth with very specific treatment needs such as arson charges or serious mental health needs that were suicidal in nature. For Multnomah’s target population, the risk criteria from the JCP for selection are listed in Exhibit 7 below. EXHIBIT 7: Multnomah Target Population: Eligibility Criteria for Intervention Referral In addition to the Minimum Requirements for Program Eligibility in Exhibit 5, Multnomah participants must meet at least two of the six following JCP items to be considered for eligibility:  JCP 5.2 - Poor Family Functioning  JCP R3.2 - Friends unlawful behavior  JCP R3.3 - Friends suspended or expelled  JCP R6.1 - Substance use  JCP R5.4 - History of child abuse  JCP R4.10 - Behavior endangers others Note: Please see Appendix B for the definition of each of the JCP risk criteria listed above. Multnomah has an existing services contract with Youth Villages Oregon to utilize its Intercept Program.14 Eligibility criteria for the feasibility assessment was drawn from the risk profile of Multnomah youth previously identified as eligible for placement and referred for Intercept. Multnomah’s target population referral criteria was also considered for Marion’s probation population. The Marion data analysis yielded a low headcount for the highest need / most expensive youth to serve (those committed to OYA Residential and YCF). The low number of youth that would escalate to OYA would thereby structurally cap the diversion impact potential of the project. Ultimately, Marion ended up with a more specific set of risk criteria (15 JCP risk factors) which include all of Multnomah’s six JCP risk criteria for selection except JCP R5.4 History of child abuse and JCP R4.10 Behavior endangers. 14 Youth Villages operates the Intercept program, a national evidence-based, intensive in-home service that looks to strengthen and restore families, and help children safely remain in their homes or reunify with their families. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 30 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 31 Quantifying the Target Population To determine size of any annual cohort, the Working Groups evaluated historical population groups for both Counties that would have met the eligibility requirements presented above in Exhibit 6 and Exhibit 7 starting in 2010. Given state and national declining trends in criminal activity, and accounting for any juvenile justice policy changes, the Steering Committee recommended that the most suitable time frame to estimate potential annual size of the target population would be from 2013-2015. Among the Marion population and on the risk criteria in Exhibit 6, it is expected that in the future there would be approximately 115 youth eligible per year for program enrollment. For context, county-wide 338 total youth were placed on probation in 2015 (regardless of risk criteria profile). Of the 115 individuals, the majority would be Latino/Hispanic (~45% of total eligible youth) and White (~45%). Approximately 25 youth, representing half of the total Latino/Hispanic eligible youth, would require specific bi-lingual and bi-cultural specialists in any intervention. EXHIBIT 8: Overview of Marion Youth Eligible for the Intervention Source: Marion County Among the Multnomah population and on the risk criteria in Exhibit 7, it is expected that in the future there would be approximately 135 youth eligible per year for program enrollment. For context, county-wide 200 total youth were placed on probation in 2015 (regardless of risk criteria profile). From a demographic perspective, of the 200 youth, 37% are Multicultural/Other, 36% African American, and 19% Asian. Roughly 15 youth and families per year would require specific bi- lingual/bi-cultural specialists in any intervention. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 32 EXHIBIT 9: Overview of Multnomah Youth Eligible for the Intervention Source: Multnomah County A historical analysis of Marion’s target population’s pathway through the juvenile justice system highlights the different commitments for youth within a two-year time frame of being placed on probation. EXHIBIT 10: Marion Eligible Youth by Juvenile Justice Commitment Probation disposition began: Probation Eligible Youth Unique Youths Detention County Residential OYA Residential OYA YCF %Placed Detention County Residential OYA Residential OYA YCF Commitments Detention County Residential OYA Residential OYA YCF Days / Youths Detention County Residential OYA Residential OYA YCF Source: Marion County 2013 2014 114 113 2015 '13-'14 Avg 138 114 104 106 102 46 47 26 22 18 7 12 11 3 91% 94% 40% 42% 19% 16% 11% 10% 74% 93% 19% 41% 5% 18% 2% 10% 195 168 100 51 48 21 30 24 7 16 22 15 182 50 27 19 21 22 80 71 208 151 316 192 20 21 72 75 111 180 140 254 As demonstrated above, the majority of Marion youths are committed to Detention or County Residential. For youths that escalate to OYA, 18% and 10% are committed for OYA Residential and YCF, respectively. Note that the analysis does not account for youths with multiple types of placements, so consider an aggregate Marion OYA escalation rate of 30%. No estimate was provided for Multnomah youth that are placed in County Detention or County Residential. Multnomah, unlike Marion, has various alternative programs in place to potentially Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 33 105 47 20 12  divert youth that might end up in County Detention. Any intervention Multnomah would examine as part of the project would exclusively look to target youth at risk of escalating to OYA Residential and OYA YCF. For Multnomah’s target population, it is estimated that 10% would annually escalate to OYA Residential and YCF. Individuals that escalate to OYA placements are likely the highest-need youth and account for the largest financial burden for the state due to mandated supervision over extended lengths of stay (see Section: Economic & End Payer Assessment ). Marion prioritizes diversion from OYA placements as well. However, outside of treating these high-need youths through home removals and placement in state facilities, Marion County has limited number of County Residential placements options (only 30 slots available) and no alternatives that are therapeutic in nature for youths adequately suited for such services. Due to its limited resources for alternative programs, Marion is also interested in the ability of a project to reduce use of county detention and residential placements. Such an approach would also suitably work within the data limitations of the project. As discussed in Section: Data Source & Access Assessment, the risk identification processes currently prevent the Counties from reliably distinguishing at intake youth that will end up in County Placements vs. OYA. If a project were to exclusively focus on youth escalating to OYA, there is a meaningful difference in placement outcomes. On average approximately 30% of Marion’s annual target population escalated to state level placements via OYA Residential and YCF. However, for Multnomah, the corresponding figure is estimated to be approximately 10% of the target population of the target population in OYA Residential or YCF. The headcount discrepancy is in large part driven by the different risk criteria for selection by each county. From a project design perspective, Marion’s target population is sizeable for project consideration while Multnomah’s low headcount fundamentally limits the diversion impact potential for the project. With the different risk criteria for youth selection and varied placement outcomes for each county, a joint-county project would be difficult to achieve at this stage. Unless Multnomah refines its risk criteria for selection (to increase headcount) or considers referring youth that would go to county placement to the project, Marion is best suited to proceed in constructing a project. Ultimately, both counties also agreed that the subsequent sections of the feasibility study would be primarily tailored for Marion’s target population but nevertheless broadly applicable to Multnomah. Recommended Next Steps: 1) Evaluate benefit OYA’s predictive data analysis resources into risk selection for target population for both Counties Both Counties’ analysis for the feasibility study might benefit from use of the Escalation to OYA tool. As highlighted in Section: Data Source & Access Assessment, it is recommended that the Counties explore expanding their collaboration with OYA to test the value of incorporating the Escalation to OYA predictive score into the youth selection Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 34 process. OYA would also need to work with the counties to address any concerns with their predictive analytic tools. Incorporating predictive analytics could enable Marion to prioritize intervention enrollment for youth that have the highest targeted needs and are costliest to serve by the state. To the extent Multnomah would strive to partner in a project and only enroll youth that would escalate to OYA Residential and YCF, enhanced targeting of probation youth at intake would reduce the need to enroll and pay for all eligible youth, and instead focus on those likeliest to escalate to OYA Residential and OYA YCF. 2) Conduct multi-year forecast of the target population referred to an intervention The first hurdle a launched project typically faces is ensuring a service provider (and funders, if any) that there are a sufficient number of youth eligible for services in the project. A forecast would not only address this, but also serve as the basis for project economics and prove the project to be suitable for the chosen evaluation method. A formal forecast of the target population was not conducted as part of this feasibility study. Project leaders should consult with internal forecasts devised by county juvenile departments or the state. Using existing partnerships between County Juvenile Directors and OYA to conduct forecasting provides an opportunity to evaluate the demand for close custody and community placement services from OYA. While standard forecast exercises conducted by Counties or OYA apply to all youth, future collaboration would entail (i) detailing the demographic and risk profile of the target population, (ii) outlining the appropriate time frame for forecast, (iii) defining the forecast metrics (# of youth, criminal referrals, crime type, etc), and (iv) identifying the impact of the forecast along key junctures of the juvenile justice system (e.g. intake, adjudication, and level of placement). 3) Modification of Multnomah’s risk criteria for target population selection is required for a PFS project, regardless of project structure (stand-alone or in partnership with Marion) If Multnomah wants to target only youth that escalate to OYA Residential or OYA YCF, new JCP risk factors (or other criteria) are required to better identify such individuals and increase the cohort size. With Multnomah’s selection criteria and focus on OYA diversion, the project design is currently not financially viable.  For a Multnomah program to be conceived, the project would need to pay for an intervention of all 135 Multnomah eligible youth (regardless of county or OYA placement) in order to reach the 15 youth that would escalate to OYA Residential and OYA YCF. The cost of serving all eligible youth is only compensated with “benefits” tied to successfully diverting the small number of OYA placed youth. A Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 35 detailed breakdown of how project economics are diluted with insufficient targeting is presented in Section: Economic & End Payer Assessment.15  The current target population’s definition yields less than 15 combined youth in OYA Residential and OYA YCF per year. The Multnomah headcount for impact is insufficient (i) for Multnomah to justify launching a new alternative program and (ii) to maintain fidelity to rigorous evaluation standards. 15 Since Marion County is willing to enroll all youth that would be committed in any of the County and OYA placements, Marion’s cost / benefit is not diluted as much as Multnomah’s. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 36 Intervention Assessment: Not Yet Demonstrated Not Yet Demonstrated Possible Promising Strong Are there interventions that can positively impact the population we want to serve? What is the scaling opportunity? Do they track outcomes that are a priority for potential end-payers? What is the recommended evaluation method? Work Stream Goals:  Determination of intervention scale  Service provider readiness for a performance-based contract  Development of a procurement strategy  Guidelines for evaluation design Summary of Findings:  Scalability of the project is promising. Given historical voluntary family consent rates, the total headcount estimate for enrollment could be 75 and 90 per year for Marion and Multnomah, respectively.  When considering which community organizations would best qualify as project partners for an outcomes-based contract tied to a rigorous evaluation, the following must be prioritized: o Data-driven intake process to validate risk criteria for selection o Documented evidence-base to validate impact o Focus on outcomes impacting government budgets o Ability to scale and address needs of target population  Request for Information (RFI) document has been created so that at the Counties’ choosing, a vetting process can be initiated to understand Oregon’s landscape of service providers and their capabilities to meet project goals. See Appendix F for the RFI document tailored for Marion.  In the absence of a specific service provider and designated end payer, a specific evaluation plan was not developed. See Appendix G for a description of the roles and essential work streams to be undertaken by an evaluator for the project. Recommended Next Steps:  Initiate Service Provider RFI process in order to understand the provider community’s capacity to meet project outcome benchmarks  Incorporate evaluation components into project design Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 37 The Counties’ historical analysis of probation youth and the JJIS database set the stage for issuing an RFI. Youth Villages was invited to join the feasibility discussion in order to provide a valuable service provider perspective.16 This helped the counties assess the characteristics of a proven in-home service. The Intervention Assessment work stream was focused on evaluating the following: I. Determination of intervention scale: Based on conclusions from Section: Target Population Assessment, the Working Groups identified the intervention referral process and potential scale of the project. II. Service provider readiness for a performance-based contract: Specific requirements for an intervention were explored, including suitability of provider data capacity, ability to measure outcomes, staffing and other resource needs, and timeline for execution. III. Development of a procurement strategy: Identifying the factors that would be most important to evaluate when selecting a service provider. IV. Guidelines for evaluation design: In concert with analyzing service provider capacity and its intervention model, develop recommendation for a plan to evaluate impact including selection of an evaluator. I. DETERMINATION OF INTERVENTION SCALE The scalability of the project is promising due to (i) use of the JCP risk assessment to anchor the youth selection process and (ii) informed decision-making that can occur at the intake stage of the juvenile justice system. Data-Driven Process: As discussed in Section: Data Source & Access Assessment, the JCP risk scores used for the project’s youth selection process are a mandatory requirement for all individuals adjudicated delinquent and put on probation. The data set is comprehensive and consistently applied to all youth on probation. By not solely relying on staff discretion for program referral, the JCP scores enable timely referrals to be made, a data-informed justification for a referral, and consistency in youth risk profile. Defined Referral Pathway: There is one defined juncture at which a youth would be referred to an intervention (as presented in Exhibit 3): once adjudicated as delinquent and ordered on probation. Juvenile staff assigned to a case would be informed by the JCP risk scores (and any other assessment) and be able to make the intervention recommendation before the youth has the opportunity to escalate to an out-of-home placement. The program would be voluntary with the youth remaining under the jurisdiction of a probation officer for completion of the conditions of their court ordered probation. Voluntary Consent & Enrollment: Although there are more than 100 youth in Marion County that would be considered eligible for the program on an annual basis (and more in Multnomah), 100% admission would not be considered suitable or would volunteer for 16 Youth Villages operates the Intercept program, a national evidence-based, intensive in-home service that looks to strengthen and restore families, and help children safely remain in their homes or reunify with their families. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 38 admission into an intervention. Based on the risk criteria in Exhibit 6 in Section: Target Population Assessment, juvenile probation officers17 would make recommended referrals without a court mandate, which make participation voluntary for youth and their families. Multnomah’s existing contract with Youth Villages’ Intercept Program to treat high risk youth functioned as a good proxy to determine enrollment yield of the eligible population. Based on the Working Group’s analysis, approximately 25% of Multnomah’s referred youth decided not to accept the referral to Intercept. Moreover, it is estimated that roughly 7% of referred youth that accepted the referral did not meet the program requirements as determined by Intercept’s Pre-Placement Assessments. When accounting for youth and family consent, and a provider’s own enrollment criteria, the consolidated enrollment yield is estimated to be 68%. By applying Multnomah’s current consent and enrollment rate with Youth Villages to eligible youth in the feasibility assessment, the annual total headcount estimate for enrollment (assuming round figures) could be 75 and 90 for Marion and Multnomah, respectively. As noted in the Section: Target Population Assessment a formal forecast of the target population was not conducted in the scope of the feasibility assessment but would be required before constructing a project. Lastly, once a service provider is selected, voluntary consent and admission yields need to assessed and forecasted. II. SERVICE PROVIDER READINESS FOR A PERFORMANCE-BASED CONTRACT While sound data, a defined referral pathway, and voluntary enrollment are determining factors in the number of youth that could receive alternate treatment to placement, there are many variables that determine how a provider can successfully deliver services. Governments seeking to enact performance-based contracts strive to partner with service providers with experience implementing an evidence-based intervention, capacity to execute the intervention on a larger scale, and ability to collect and report data for evaluation. This is important for two reasons. First, when governments truly contract for outcomes, those agreements generally include a requirement for impact to be rigorously evaluated (in the case of PFS this helps determine “success payments.”) Second, once evaluation allows governments to demonstrate the effect of a program, such results add to the evidence-base to inform future programs, policy priorities, and/or academic research. When governments consider which community organizations would best qualify as an applicant to an outcomes-based contract tied to a rigorous evaluation, staff should focus on the following service provider capacities: Data-driven intake process to validate risk criteria for selection: The Counties have illustrated the specific risk criteria in Section: Target Population Assessment for referring the eligible population to a proposed intervention. An intervention program would need to 17 Known as Juvenile Court Counselors in Multnomah County Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 39  demonstrate their intake criteria for enrollment directly overlaps with the government’s risk criteria for referral. At a minimum, the organization must demonstrate why the needs of youth previously served are fully or partially aligned with those that will be referred by government. The Intercept Program, for example, currently does not have data sharing agreements in place with government to validate that each of its enrolled youth has specific JCP risk factors. However, the organization conducts various intake assessments and knows that over 40% of its youth from 2006-2010 have demonstrated suicidal ideation, which is comparable to JPC 4.10 (“Behavior hurts or endangers youth”). Similarly, almost 40% of Intercept youth have a history of child abuse, which is comparable to JCP 5.4 (“History of reported child abuse”). During the organization vetting process, the onus lies on the project managers and evaluators to validate that at a minimum, (i) the majority (if not all) of the risk factors for referral determined by government overlap with a service provider’s intake criteria for enrollment and (ii) the majority (if not all) of an intervention’s past participants demonstrate the specific risk criteria. Demonstrated consistency of risk criteria across various cohorts served and a decision-making flow chart of the treatment cycle should be required from providers in order to gauge scalability of a program. Documented evidence-base to validate impact: One of the largest challenges in the service provider community is demonstrating what would have happened to the population served in the absence of the delivered intervention. Service providers typically rely on self-reported survey tools (instead of evaluations conducted by third parties) and/or monitor internal program metrics with limited to no validation from an unbiased dataset (e.g. government information). Self-evaluations and unproven internal metrics are two common “evaluation practices” that require substantial diligence if governments want to implement performance-based contracts with evaluations. The Counties and end payer should work to partner with a service provider with a documented track record of providing successful outcomes (not outputs). Outputs are descriptive statistics that typically highlight what an organization does (e.g. number of people served, successful discharge, types of service provided within a timeframe, etc.). Conversely, outcomes demonstrate the differences achieved by a program, and are typically metrics that can be compared to what would have happened in the absence of the intervention (e.g. decrease in new criminal referrals, reduction of drug rehabilitation services, or lower length of stay in detention facilities). The change compared to what would have happened anyway (i.e. status quo) would be termed “impact.” Should the project elect to proceed with a service provider without a demonstrated track record of outcome delivery validated by a formal evaluation, it is highly recommended to allow for the service provider to develop the capacity to collect and report data for evaluation. Partnering with philanthropic partners and other community based funders could allow projects to build in a trial period or perform a small scale pilot with an evaluation before transitioning to full project execution. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 40 Focusing on outcomes impacting government budgets: Identifying outcomes is important, but what determines the viability of a project is understanding the impact such outcomes have on government resources. The Counties and end payer must distinguish between a service provider’s leading indicators and outcomes that impact budgets, and build a project around the latter. Doing so enables a project end payer to clearly gauge how it would directly benefit from a successful intervention. A leading indicator or intermediary outcome is defined as an important metric that validates successful service delivery, or demonstrates changes in behavior or usage of services that may not directly impact government expenses (e.g. program completion, interactions with the law, or enhanced involvement in pro-social activities). In this instance, outcomes impacting government budgets that could directly be identified with a successful result would include (i) reduced number of placements or criminal referrals for a juvenile justice end payer or (ii) improved average daily membership (attendance) for an education end payer. Service providers that can identify and validate outcomes impacting budgets are well suited for a performance-based contract and help build the case for an end payer to commit funding. Assessing Intercept’s outcome reporting is a particularly helpful exercise for the feasibility assessment since the organization has an active partnership agreement with Multnomah to serve youth at risk of escalating to OYA. When assessing the outcomes evaluated by Intercept, there is a lot of alignment with the Counties’ priorities. However, not all outcomes can be tied to directly impacting the government budget of a potential end payer. Exhibit 11: Outcomes evaluated by the Intercept Program Intercept Program Outcome Commentary a. Successful discharge (e.g. Youth & family safety) Leading Indicator: Unclear impact to government budget b. Living stability within 12 months (e.g. family reunification or independent living) Unclear: Depends on who would be the end payer (DHS, OYA, or Coordinated Care Organization). Likely a longer term indicator of successful diversion of out of home placement c. No trouble with the law Leading Indicator: # of criminal referrals or probation violations are better outcomes that demonstrate budget impact d. Reduced use of drugs and alcohol Leading Indicator: Decreased use of drug and alcohol abuse treatment is a better outcome that demonstrates budget impact. e. Reduction in out of home placement (e.g. residential treatment, detention/correctional facility, & foster care) Strong Outcome: Direct budget impact to county juvenile justice, OYA, and/or state-level budget f. Placement for appropriate mental health service Leading Indicator: Likely to increase cost to government but important in meeting population needs g. Enhanced involvement in pro-social activities and/or with pro-social associates Leading Indicator: Difficult to quantify into a budget impact without making broad lifespan assumptions Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 41 h. School status (e.g. GED completion or graduation) Leading Indicator: Difficult to quantify into a budget impact without making very long term (lifespan) assumptions i. Academic progress Leading Indicator: Difficult to quantify into a budget impact, unless strongly correlated with improved school attendance j. Improved school attendance Strong Outcome: Direct budget impact to a school district (via Average Daily Membership metric) With the exception to outcomes (c), (e), and (j), the other outcomes are not directly traceable to affecting a budget line item of a potential end payer for the intervention. From a project design perspective, a program can be anchored by outcomes tied to government budgets like (c), (e), and (j), and also be supplemented by other outcomes that may be long term in nature (e.g. (b) living stability), valued by society but difficult to quantify (e.g. (g) enhanced involvement in pro social activities), or operational in nature (e.g. (a) successful discharge). Negotiating the type of outcomes to be evaluated and/or paid for in a project is a critical discussion when engaging an end payer Section: Economic & End Payer Assessment delves into how the specific outcomes for this project impact the budgets of Marion, Multnomah, the Oregon Youth Authority, and other state-level entities. Ability to scale and address needs of target population: The ability for a provider to scale up (or down) to account for ongoing intervention delivery (or course correction) is very beneficial. Since performance-based contracts are contingent on successful outcome delivery, funding for a service provider is not fully guaranteed in the traditional sense. Therefore, the ability to flexibly manage budgets, staffing, recruitment, and retention is important. For example, the Intercept Program has instituted a service delivery model based on the utilization of a specialist. Up to four cases are assigned per family intervention specialist, with the specialist utilization rate historically averaging 90%. Since Intercept lasts 4-6 months on average, one specialist can account for two youth cohorts on an annual basis. With sufficient lead time, new specialists can be assigned to cover a geographic region. For an organization that may not be operational in the geographic area of preference, prompt and sustainable ramp up processes are essential. Intercept, for example, requires approximately two months lead time after contract signing with government to hire/(re)assign and train case specialists, and allocate leadership staff. An additional month is needed to develop the internal capacity to serve the first eight youth. All in all, a service provider expanding to a new area would require 2-3 months (or more) since contract signing before a youth can be served. For Youth Villages, an additional eight youth can be referred every month thereafter. The total cost for Youth Villages to start operations in a new area can be approximately $350,000, inclusive of relocation assistance, hiring incentives and bonuses, open houses, new office space, depreciated computer equipment, and office furnishings. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 42 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 43 III. DEVELOPMENT OF A PROCUREMENT STRATEGY The first step in developing a procurement strategy would be to issue a Request for Information (RFI) for the project. This is the best path to understanding what service providers are capable of delivering. The process will allow project partners to explicitly outline mandatory requirements (programmatic or administrative) for service providers, in addition to assessing how the organization will meet the needs of the target population and the outcome delivery for which an end payer would be willing to pay for. Initial engagement with organizations will also help providers understand what is required to succeed in an outcomes-based contract. Lastly, it allows project partners to evaluate an organization or the intervention model’s ability to meet the goals, standards, and evaluation needs of a performance-based agreement. Issuance of an RFI is only the first stage of the procurement process. Following this information gathering phase, government will typically issue a Request for Proposal (“RFP”) to initiate vendor selection. The Intervention Assessment work stream did not reach this stage of analysis for the feasibility assessment. However, Section: Legal & Regulatory Assessment does identify key elements of a performance based contract that would assist in its development. With guidance from the Steering Committee, the Counties developed programmatic conditions that complement the general service provider principles outlined above in Section II (“Service provider readiness for a performance-based contract”). Those requirements include (partial list):  Intensive services delivered in an in-home setting throughout the designated county  Bilingual/bicultural competency in service delivery  Willingness to participate in rigorous impact evaluation (e.g. propensity score matching) A full copy of the RFI document, which was explicitly designed for Marion County, can be found in Appendix F. As the Counties assess the best path for procuring a potential service provider, there are two project-specific factors to consider for the RFI process: (i) bilingual/bicultural competencies and (ii) cost of service delivery Bilingual/bicultural competency in service delivery As demonstrated in Section: Target Population Assessment, Marion County has a disproportionate number of Latino/a youth in the target population. Of the 115 eligible individuals per year, almost half are Hispanic. Of those youth, approximately 25 people (or half of the total Latino/a adolescents) would require specific bilingual/bicultural specialists in any intervention. Assuming the 68% consent and enrollment rate holds for Hispanic youth, 15- 20 families might require bilingual/bicultural support annually. While it is unclear how many organizations in the state have the bilingual/bicultural presence to serve that many families in Marion County during year one, the selection process must be mindful of any recruiting and retention factors for specialized staff, and any associated costs to meet this demographic requirement. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 44 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 45 Cost of service delivery While the Counties did not highlight a preferred range or maximum cost of service delivery to enroll a youth, it is a critically important factor (especially in the cost/benefit analysis). Based on the risk profile of the target population, the chart below presents potential intervention models suitable for the project to consider. It is based on an estimated annual cost of service conducted by the Washington State Institute for Public Policy for the Washington State legislature. A broad spectrum of costs are listed ranging from $1,500 to more than $10,000 per youth, depending on service type.18 EXHIBIT 12: Costs Range for Potential Invention Suitable for Target Population Intervention Type Annualized Cost per Participant Family Integrated Transitions (youth in state institutions) $11,809 Multidimensional Treatment Foster Care $8,282 Multisystemic Therapy for substance abusing juvenile offenders $7,739 Multisystemic Therapy $7,731 Functional Family Therapy (youth in state institutions) $3,427 Functional Family Therapy (youth on probation) $3,427 Source: Washington State Institute for Public Policy. Review of evidence-based policies for the Washington State legislature (accessed July 2016) There are a number of limitations with this table, but it is helpful in ascertaining which interventions could be considered and the range of costs. The following are critically important factors for any program selected as a result of the feasibility assessment: 1. Interventions available in Marion, Multnomah, or any Oregon county 2. Datacapacityoftheorganization 3. Matching of the enrollment criteria of an intervention to the risk criteria of the project’s target population 4. Identifying of specific intervention outcomes directly impacting an end payer’s budget with successful service delivery 5. Cost for service delivery related to the program effectiveness (“success rate of attaining outcomes that directly impact government budgets”) 6. Varianceincostofservicedeliverybygeography,humanresourceneed,andother environmental constraints (travel, communications, etc) While the WSIPP report broadly defines program costs, any intervention the project would consider for selection must perform due diligence to forecast the following cost items:  Salaries, wages, and benefits 18 WSIPP accessed July 2016 http://www.wsipp.wa.gov/BenefitCost/Pdf/1/WSIPP_BenefitCost_Juvenile-Justice For technical details, see http://www.wsipp.wa.gov/TechnicalDocumentation/WsippBenefitCostTechnicalDocumentation.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 46   Operations & Maintenance, including travel, communications, rent, and supplies  Corporate Overhead  Ramp up costs (if required) Whether it is determining which service providers can meet specific project requirements or understanding the potential cost of service, conducting an RFI process will enable the project to assess the suitability of partnership with an Oregon service provider. These components, however, underscore the importance of government the needs of its target population and the outcomes it strives to address before starting the RFI process. IV. GUIDELINES FOR EVALUATION DESIGN: An evaluation plan was not developed for the project due to the absence of a specific service provider or intervention model. The Counties intend to determine objectives and partners associated with any evaluation at a later date. As for the expertise required for conducting an evaluation, the Counties and their project partners felt well positioned to select an evaluator through a competitive process, sole procurement, or via internal government units. Evaluation Work Flow Based on guidance from the Steering Committee, there are a number of local experts in the state that have prior experience evaluating juvenile justice programs, have a working relationship with project partners, or are experienced practitioners in the academic or service provider sectors. Potential evaluation partners include:  Portland State University  University of Oregon  Oregon Social Learning Center  NPC Research (Portland) The Working Groups have summarized expected roles and processes to be undertaken by an evaluator for this project. See Appendix G for a description of the roles and essential work streams. Depending on how a project is administered, staff within the Counties’ juvenile department and the Oregon Youth Authority’s research and evaluation unit should be consulted either for guidance or direct involvement in project evaluation. Data Quality The statewide use of JCP scores allows for a direct data link between risk profile and outcomes. This ensures a basic foundation for any evaluation methodology employed to determine program effectiveness. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 47 In fact, the data quality of JJIS and the JCP was sufficient enough for OYA to recently complete an evaluation report in the spring of 2016 on Youth Village’s program with Multnomah County without the need for a physical control group.19 By using various actuarial formulas based on JCP and Escalation Scores, OYA developed a status quo case (no treatment) to highlight baseline outcomes. Those outcomes then served as a reference point for comparison with the outcomes of the treatment group. Preliminary Views on Evaluation Designs Once a service provider or intervention model is selected, there are a number of evaluation methodologies that can be chosen. Each has different characteristics and requirements, and the design selection is influenced by a number of factors related to the goals of the evaluation, quality of data and resources available. Below are three key evaluation options to consider, with several different designs available.  Non-Experimental (measurement of participant outcomes before/after program; no comparison group)  Quasi-Experimental (measurement of participant outcomes with a “control” for bias)  Experimental (includes randomization of participants to “treatment” or “control” groups) Conceptual Evaluations Designs 19 Oregon Youth Authority. Research and Evaluation. Evaluation of rates of escalation to OYA among Multnomah County youth enrolled in the Youth Villages Intercept Program. March 2016 Table 7 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 48 Examination of these evaluation designs would be required by the project evaluator to assess the feasibility of validating results for a performance-based contract. There are, a number of factors (e.g. data access, ethical design, resources) that need to be considered when designing a future evaluation for a performance-based contract, all of which are contingent on (i) the data capabilities of a service provider, (ii) the outcome thresholds an end payer is willing and able to pay for, and if applicable, (iii) a third party funder’s risk tolerance. The Steering Committee has ruled out conducting a randomized control trial (“RCT”) as an evaluation methodology for the project. A RCT is a study in which people are allocated at random to receive the intervention treatment. One group, the “treatment group,” will be formally enrolled and accepted into the intervention (and subsequently notified). The other group, the “control group,” will not receive the intervention treatment, and will instead have access to other regular social services and support. For this project the control group would be comprised of youth that continue living their lives “as is” with the risk of further penetrating the juvenile justice system without enrolling in Youth Villages. The Steering Committee was not comfortable with knowingly denying a probation youth access to treatment that would potentially lead to poor outcomes. The table below includes potential alternative evaluation options to an RCT that withstand comparable statistical rigor to validate impact EXHIBIT 13: Evaluation Methods for Assessing Impact Recommended Next Steps: 1) Initiate Service Provider Request for Information (RFI) process Conducting a non-binding RFI process to demonstrate there are viable service providers operating in the community that can potentially meet the outcome expectations of the project. Once identified, the project end payer can work closely with the county, OYA, and other project partners to ultimately select a preferred service provider and better assess the project’s cost / benefit proposition. This process, despite the fact that an end payer has not yet been secured, would help propel the project forward. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 49 A full copy of the RFI document prepared for Marion County can be found in Appendix F. Conducting an RFI in the short-to medium-term will enable project partners to: a) Determine of the number of organizations interested in participating in the project, b) Identify organizations most ready to participate in that span, and c) Identify capacity gaps of an organization, and help them successfully participate if the project is scaled Depending on the quality of responses, the project may or may not consider referring potential providers to targeted training and/or technical assistance programs to increase their capacity and competency. Alternatively, additional education can be provided to applicants so that they obtain a better understanding of the contracting requirements of a performance-based contract. 2) Incorporate evaluation components into project design Though a specific intervention has not yet been selected, a work stream can be focused on preparing the project for evaluation. Such efforts could include:  Evaluation Methodology: Identifying all possible evaluation methods (including those in Exhibit 13) that would be most appropriate for the project. That could also include a review of other evaluations conducted in criminal justice or juvenile justice and the implications of applying them to this project.  Data Quality: Assessing the statistical quality of raw data sourced from JJIS (namely risk criteria and outcomes information). This would entail reviewing the risk scores and outcomes of the target population defined in Section: Target Population Assessment. Evaluating risk criteria for selection would be particularly important. For Marion, the fact that only some of the risk criteria (8 of 15 factors in Exhibit 6) are required for referring a youth could compromise the evaluation of successful diversion. Evaluators must opine on how youth with 8 of 15 risk factors differ from those with 10 of 15 (or youth that have a different set of 8 risk criteria).  Project Design: Recommendation of data categories, proxies, and collection frequency to be incorporated in the design of the intervention. For example, evaluators should examine whether or not it is suitable to have a two-year time frame to assess new criminal referrals. The same would apply with success metrics (e.g. evaluators should comment on whether it is a realistic benchmark to have no placement and no new criminal referrals among probation youth with the risk profile Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 50 defined by the project). Lastly, a determination could be made if the project would be constrained by a minimum number of youth referred from the target population. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 51 Economic & End Payer Assessment: Promising Not Yet Demonstrated Possible Promising Strong What is the economic case for the PFS project? Are there end payers committed to paying for success achieved? Work Stream Goals:  End payer identification and assessment  Conduct cost / benefit analysis Summary of Findings:  Primary outcome of interest to potential end payers of a performance-based contract is a reduction in out-of-home placements  A potential list of priced outcomes includes a reduction of: o Escalations to OYA commitment (Residential Treatment and Youth Correctional Facilities) o Use of County Detention and County Residential (Marion only) o Number of criminal referrals (metric for criminal activity) o Victimization costs (metric for social impact of juvenile offending)  The costliest placements are at the state level. OYA is an essential partner in finding a source of state level end payments to capture the benefits of reduced escalation to state-level placements.  The cost/benefit analysis demonstrates the interaction between assumptions about the status quo outcomes of the target population, an intervention’s cost, and the intervention’s effectiveness. The analysis provides a framework for selecting an intervention that can reasonably be expected to succeed.  County-level measures for the feasibility study do not predict the risk of escalation from probation to a residential placement by placement type (e.g. detention, residential, and OYA). This means some youth would receive intervention who are not at risk for the highest cost outcomes valued by potential end payers, which currently dilutes the cost/benefit analysis Recommended Next Steps:  Secure commitment of state level end payer  Determine feasibility of cost-effectiveness requirements for available service providers  Set expectations for risk sharing between end payer and provider  Solidify willingness to pay for outcomes without a direct budgetary impact  Explore incorporating OYA Escalation data to improve evaluation and enrollment targeting Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 52 The development of any PFS project is predicated on (i) the identification of a government entity willing to pay for the delivery of project outcomes and (ii) viability of securing government funding from such entity. Without an end payer that is able and willing to pay for outcomes engaged in a project, it is very difficult for the initiative to materialize. The Economic & End Payer Assessment work stream was focused on evaluating the following: I. End Payer identification and assessment: Develop strategies for securing an end payer for improved outcomes for juvenile offenders. II. Cost/Benefit analysis: Determine framework for establishing cost-effectiveness of proposed interventions I. END PAYER ASSESSMENT: From the outset, the Steering Committee established the key project outcome as reducing escalation to state and county placements for the target population. The Working Groups examined the financial impact of reduced escalation for each of the four placements (e.g. detention, GAP, OYA Residential, and YCF) on government funding streams that would otherwise pay for such placements. Before delving into how each entity would be suited as end payers, it is important to note that cost is defined as average cost (i.e. total spending that would not decline linearly with reduced placement) and not as marginal cost (i.e. direct operational cost savings with fewer placements). The three potential end payers considered for this project are the State of Oregon, Marion County, and Multnomah County. The following analysis examines how a successful diversion from placement would impact the respective budget of a particular end payer. This analysis ensures that each prospective end payer would pay for a diversion outcome aligned with the benefit to be gained with a successful intervention. Simply put, state-level dollars primarily fund OYA Residential, YCF, and GAP, and would therefore benefit the most from diverting commitments to those placements. State-Level End Payer One of the Counties’ primary aims is to reduce escalation to OYA Residential and YCF among the target population (as defined in Exhibits 6 and 7 in Section: Target Population Assessment). This goal addresses the highest need youth and those who are costliest to serve due to long stays. While the Counties process the intake of all youth that may escalate to OYA, the Counties are not responsible for the direct costs of OYA commitments. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 53 Exhibit 14: Marion Target Population Direct Cost of Placement per Youth Source: Marion County and Oregon Youth Authority Note: Length of Stay is inclusive of all re-entries. Federal funds inclusive of Medicaid and other federal funds  As shown in Exhibit 14, state-level funding accounts for approximately 60% and 100% of the direct cost of placement in OYA Residential and YCF.20  As the tier of government which stands to be the main beneficiary of successful reductions in these two OYA placements, the state would therefore be the logical source for project end payments. As presented in the cost/benefit calculations, the conclusion remains the same when accounting for cost of criminal referrals and social costs.  Should the project also factor in GAP placements, the state would still stand as the primary beneficiary of those successful diversions. Although Detention is a 100% county- funded commitment, successful Detention diversion does not fully offset the state’s gain in also successfully diverting GAP and OYA placements. Should the project exclusively seek to serve youth at risk of escalating to OYA, those individuals would be a minority of youth on probation. Statewide, regardless of risk criteria, 21% of youth on probation before age 17 escalate.21 By using the additional targeted risk criteria associated with eligibility for this project, approximately 30% of Marion’s target population in the feasibility study escalated to OYA (the remainder ultimately utilize County Detention or County Residential without OYA escalation).22 Unless the intervention could be targeted to exclusively focus on the 30% of Marion’s youth at risk,23 the cost of serving (paying enrollment for) all youth who meet the current target population criteria but do not escalate weakens the cost effectiveness for the state to pay for an intervention if the project is only focused on OYA benefits of diversion. The preceding discussion assumes that the state or one of its departments (e.g. OYA) would be willing to administer success payments for diverted placements. OYA staff participated in the Working Group and Steering Committee meetings, but no firm commitment was made by the 20 The chart above does not include the administrative costs of criminal referrals or cost to society. As demonstrated in the latter half of this section in the cost/benefit calculations, such additional costs are minor relative to the direct cost of placement and minimally inform which end payer’s budget would benefit from successful placement diversion. 21 Oregon Youth Authority. Research and Evaluation. Evaluation of rates of escalation to OYA among Multnomah County youth enrolled in the Youth Villages Intercept Program. March 2016 p.4 22 Marion County and Multnomah County 23 A solution to this challenge is proposed in Section: Data Source & Access Assessment, Next Step #1 and Target Population Assessment, Next Step #1 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 54 conclusion of the feasibility assessment. Because of the high cost associated with OYA placements, state-level contributions to success payments are crucial to any project. Enabling legislation may be required to grant such funds, which could either directly flow to a county or be administered by OYA (or another state agency). Marion County The Marion County representatives on the Steering Committee prioritized serving youth at risk of escalation to OYA. The county also wanted to explore the option of providing an alternative service for youth at risk of escalating to County placements (Detention and GAP).There are benefits to looking at both options:  As presented in Exhibit 14 above, any diversion from County Detention or GAP would predominantly benefit the county since it accounts for 100% and 33% of the direct costs of placement, respectively.  As the government which stands to be the main beneficiary of reductions in Detention and GAP, the county would be the logical end payer for those outcomes. The conclusion remains the same when accounting for cost of criminal referrals and social costs.  Marion currently has minimal justice alternatives that do not remove these youth from home and is seeking to expand its options for keeping families intact.  Although Marion would like to incorporate diversion to OYA Residential and YCF, the county does not directly fund such placements and would have limited fiscal benefit from successful diversion.  As presented in Section: Target Population, the analyses below assume that 30% of Marion’s target population would escalate to OYA placements.  From a technical perspective, the current intervention referral criteria in Exhibit 6 in Section: Target Population Assessment do not distinguish at intake youth at risk of county placements from those at risk of escalating to OYA. So, similar to the key cost effectiveness consideration mentioned above for the state, if the county serves all eligible youth (regardless of county or OYA placement) but only benefits from diverting placements the county funds, the cost effectiveness for Marion is materially weakened. A Marion-only end payer scenario was evaluated (see Scenario A below). Under this scenario, all youth in the target population would be served, but only reductions in placements that affect direct Marion expenditures were considered (County Detention and GAP). To be thorough and to present a comparable analysis to Scenario A, an alternate case was contemplated with the same Marion target population and assumptions listed above, but with the State of Oregon serving as end payer in conjunction with Marion County with how end payments are funded is to be determined. (see Scenario B below). Under Scenario B, each end payer would be responsible for outcomes that affects their respective budgets, so Detention and GAP for the county with GAP, OYA Residential, and YCF for the state. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 55 Multnomah County The Multnomah County representatives on the Steering Committee prioritized serving only youth at risk of escalation to OYA for the project. Unlike Marion, Multnomah presently has a more diverse set of alternative programs for youth at risk of county placements (including a contract with Youth Villages as an in-home alternative). Other alternative programs, not including this project, would be considered for youth likely to be placed in County Detention. A breakdown of how youth placed in county detention are served was therefore not included by Multnomah in the feasibility study. Since the outcomes of interest for Multnomah County are OYA and YCF diversion, and those primarily impact state budgets, Multnomah ultimately was not considered in an independent end payer scenario. A scenario for a hypothetical bi-county project with a state-level end payer was assessed (see Scenario C below). This scenario is similar to Scenario B, except project headcount is increased by Multnomah’s youth. For purposes of the analysis, Marion’s 30% OYA escalation rate is hypothetically applied to Multnomah’s total headcount. However, a Multnomah focus on outcomes only impacting OYA placements dilutes the cost/benefit analysis. This is due to the fact that in an effort to reach Multnomah youth at risk of OYA placements, youth that would not escalate to OYA would need to served (and paid for in an intervention) without accounting for the financial benefits of detention or county residential diversion. II. COST / BENEFIT ANALYSIS: The Working Groups conducted a cost/benefit analysis to determine what the required effectiveness of an intervention would need to be in order to justify an end payer’s commitment to contribute success payments under each of the three scenarios (A though C) described above.  The first step determined the status quo costs of the negative outcomes prioritized by potential end payers.  One outcome was the (i) placement cost associated with Detention, GAP, OYA Residential, and YCF commitments, which accounted for the majority of the value opportunity.  Other outcomes considered in the cost/benefit calculations were the (ii) cost of processing criminal referrals and (iii) victimization costs (social costs), albeit at fairly conservative estimates.  Once status quo costs were determined for the three categories of outcomes, a hypothetical success rate of diversion was implied to determine the potential costs to be avoided with success.  The calculated cost avoidance is ultimately compared to the total cost of service delivery. Exhibit 15 helps illustrate the cost / benefit framework. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 56 EXHIBIT 15: Framework of Cost / Benefit Analysis (1) Defined as evaluator costs, financing costs and all other project management costs Calculating Cost Effectiveness Using Average Cost It is important to note that while the Working Groups assessed the cost effectiveness of potential interventions, this is not the same as cost savings. The difference comes down to the choice to use average costs24 rather than marginal costs25 to describe the benefits of reduced escalation to juvenile placements. The marginal cost figure only captures direct operational savings, such as reduced food and clothing expenditures associated with fewer placements. This could be thought of as an immediate budgetary saving. The average cost figure reflects total spending, including a portion of capital and other expenditures that do not decline linearly with reduced placements. Using average costs for capital intensive operations like juvenile placements overstates the immediate budget impact of reduced escalation. However, in the medium to long term it is a better gauge of the costs associated with the status quo and a strong proxy to directly align outcomes per public dollar spent. The Steering Committee gave explicit guidance to use average cost because the aim of this project is to make the best use of current funds being spent to achieve the best outcomes for the target population, and not necessarily underwrite a project solely to produce costs savings. This approach is consistent with the project design of all 11 PFS projects launched since 2012. Time Horizon for Assessing Cost Effectiveness The Steering Committee chose to measure outcomes for youth up to two years after being placed on probation. The two-year window was chosen based on precedent set by existing contracts in Multnomah County with service providers. A provider like Youth Villages, for example, is experienced with tracking outcomes over this time frame, and would be the 24 Average Costs = Total Costs / Number Served 25 Marginal Costs = The cost added by serving one additional person Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 57 expectation for other organizations that work with probation youth. This is a conservative estimate of the medium-term benefits of improved outcomes for the youth served and ensures improvements are not temporary. From a budgeting perspective, a two-year time window also allows project evaluation to be aligned with the biennial funding timeline in the state. Because intervention benefits for this project are proposed to be measured over two years, not all costs will be comparable to annual budget amounts. For example, the length of stay and number of criminal referrals per youth are averaged over a two-year period. This methodology does not mirror spending from an annual budget, because it accounts for some of the criminal referrals that would occur in the following year. If this project was restricted to outcomes impacting a single year’s budget it would be difficult if not impossible to justify investing in interventions with medium-term benefits. Another nuance to be aware of when conducting a cost/benefit analysis is the appropriate timeframe in which to measure both costs and benefits. As this intervention is a formal diversion from the four identified placements, the costs of intervening are borne immediately whereas the benefits of improved outcomes may accrue for years if not a lifetime.26 This approach reflects a conservative measurement of project potential, but can be reasonably negotiated with a potential end payer when discussing what outcomes can be achieved over a specific time frame for every public dollar invested in the project. Other cost/benefit analyses, such as those from the Washington State Institute for Public Policy, produce a much larger measure of impact by valuing improvements in a youth’s outcomes over many years or even a lifetime.27 Although measuring outcomes over 40+ years captures a broader spectrum of improvements and makes a case for more interventions, relying on benefits that far in the future poses several challenges. To start, lifetime benefits make a case for “societal benefits” that do not directly translate into “budget benefits” that would make the case for government to act today. Second, validating any long time horizon drastically increases the costs of evaluation. If the benefits are back-loaded this can make it difficult to determine if there is justification for success payments from a specific end payer. Lastly, because this project focuses on juvenile department spending, the long-term outcomes for all youth that could impact other budgets (e.g. the adult criminal justice system) were not considered for the feasibility study. While benefits might conceivably accrue to the adult system in future years, they were not used to justify end payments by the juvenile system in the near term. The following sections detail how the selected outcomes were valued, and the relative cost effectiveness of each of the end payer scenarios. Selected Outcomes The Steering Committee determined that the primary outcome for a project is a reduction in out- of-home OYA placements for the target population. A successful intervention, however, would 26 Washington State Institute for Public Policy. Benefit-Cost Results. http://www.wsipp.wa.gov/BenefitCost 27 Ibid Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 58 reduce the rates of escalation to County juvenile placements. The Working Groups investigated the costs and usage associated with each placement to determine how to value a reduction in placements and how much each end payer would benefit. The Steering Committee also requested that the cost of criminal referrals be included to reflect the direct benefits to government of reduced juvenile offenses. The social costs associated with juvenile offenses were also considered, with a focus on justice-related costs as opposed to follow-on effects in education or other realms. The following diagram illustrates the framework for calculating a value for each of these outcomes. The following sections detail the sources and results of this analysis. State Commitments County Placements  OYA Residential  OYA YCF  Detention  County Residential / GAP EXHIBIT 16: Cost / Benefit Illustrative Framework Cost of Placement Outcome Placement Criminal Referrals Social Cost Placement Methodology = Length of Stay x # of Days = Arrest Cost + = Tangible Costs to + Victims = Length of Stay x # of Days Cost per Day $ per Youth Source Adjudication Costs Other Social Costs Cost per Day $ per Youth Source The Working Groups estimated the cost of one juvenile placement by multiplying the average cost per day by the average length of stay. For OYA commitments (OYA Residential and YCF) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 59 prior state research had already established a cost per day value28. For the Marion County placements (GAP and Detention), the Working Groups used total 2015 funding for each placement and total actual bed days. The funding was broken out by source, such as County General Fund, State General Fund, and Medicaid, wherever possible. A detailed breakdown of county placements, by funding source, is included below. EXHIBIT 17: Marion Target Population Daily Placement Cost per Youth Funding Source Placement County State Federal Total Detention $352 - - $352 GAP $103 $160 $41 $304 OYA Residential - $115 $81 $196 YCF - $290 - $290 Source: Marion County and Oregon Youth Authority All length of stay estimates are based on actual bed days for Marion County youth who entered probation between 2013 and 2015 and match the eligibility criteria. Although data was reviewed since 2010, this date range was chosen based on Steering Committee guidance that older data might not accurately reflect current needs of the target population and recent national trends in juvenile criminal activity. Total days of placement per youth were tracked for two years from when the youth entered probation. EXHIBIT 18: Marion Cost of Placement Assumptions per Youth Source: Marion County and Oregon Youth Authority Cost of Criminal Referrals Arrest Cost + 28 Oregon Youth Authority. Research and Evaluation. Evaluation of rates of escalation to OYA among Multnomah County youth enrolled in the Youth Villages Intercept Program. March 2016 Table 7 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 60 Placement Average Length of Stay Average Cost per Day Benefit of a Diverted Youth A B =AxB Detention 21 days $352 $7,392 GAP 75 days $304 $22,800 OYA Residential 180 days $196 $35,280 YCF 254 days $290 $73,660 Criminal Referrals = Adjudication Costs  The Working Groups calculated the direct costs of criminal referrals by looking at the cost of arresting juvenile offenders and juvenile department staff time associated with processing youth referrals. The cost of an arrest was based on the internal analysis of the Oregon Criminal Justice Commission’s Cost Benefit Methodology (see Appendix I).29 The Working Groups estimated the administrative costs for a county to process a new criminal referral, which were based on Marion County Juvenile Department estimates of the time needed for intake staff to process a new referral alongside hourly salary including benefits (see Appendix J). The Working Groups estimated how many criminal referrals could be expected by a youth in each of the four placements based on historical 2013-2015 Marion County data. See Appendix L for detailed breakdown. EXHIBIT 19: Marion Cost of Criminal Referral per Youth Avg # of New Criminal Referrals Arrest Cost per Referral Juvenile Dept Referral Processing Cost per Referral Cost of Criminal Referral Placement A B C = A x (B+C) Detention 2.0x $742 $392 $2,268 GAP 5.1x $742 $392 $5,783 OYA Residential 5.4x $742 $392 $6,124 YCF 4.7x $742 $392 $5,330 Source: Marion County Social Cost Social Cost = Tangible Costs to + Other Social Victims Costs The Working Groups also explored additional social costs such as victimization costs and other values that do not have a direct budgetary impact to the county or state budgets (e.g. school attendance, drug abuse treatment, or mental health consultations). The Working Groups considered various way of incorporating these considerations into the cost/benefit analysis since they are important for the youth’s future, their families, and for policy makers. Ultimately the project settled on a fairly conservative approach by only using tangible costs to victims per new criminal referral. The tangible costs to victims were estimated using the restitution amounts for youth of all crime offenses leading to restitution orders.30 For Marion County youth, this averaged $3,200 from 29 Arrest (per arrest) of $701 is adjusted for inflation to $742 for the purpose of this analysis 30 All offenses include all misdemeanors and felonies for target population youth from 2013-2014. The most common offenses are burglary, theft, criminal mischief, and substance/alcohol. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 61 2013-2014 per criminal referral (See Appendix K). For the cost/benefit analysis, the average $3,200 in restitution is applied to each criminal referral per placement. Additionally, all models assume that only 40% of youth have an offense which can be tied to restitution. This is done to reflect the actual level of these offenses and avoids overstating the negative impacts of other nuisance offenses (e.g. substance abuse and criminal mischief) for which youth are referred. Restitution costs are inherently a lower bound of the true social costs of juvenile offending and escalation. The Working Groups reviewed several existing methodologies that account for the full cost to society of juvenile offending. This literature review included the Washington State Institute for Public Policy Benefit-Cost Analysis, Oregon Criminal Justice Commission Cost-Benefit Methodology, and the Justice Policy Institute’s “$ticker $hock” report.31 Some of the additional outcomes considered for inclusion were educational attainment, improvements in mental health, and reduction in substance abuse. A description of these methodologies and their applicability to this feasibility assessment can be found in Appendix N. The Working Group’s analysis of this existing research suggests valuing the social costs, inclusive of restitution, at $10,000 to $15,000 per youth. However, unlike the analysis conducted for this study, social costs referred to in the literature review apply to an undefined level of criminal activity and may not be applicable to the specific target population of this study. The costs figures of the feasibility study are specific to OYA and County placements for probation youth. The social cost estimates from the literature review have differing assumptions around the timeframe to track benefits, in some cases as many as 50 years.32 Most calculations include felony offenses, which are not well aligned to the target population, especially given that Measure 11 youth are excluded in this assessment.33 Furthermore, the value to be placed on positive social outcomes is ultimately a determination to be made by a committed end payer relative to the outcomes delivered by a selected service provider. The social cost figures utilized in this analysis are essentially a beginning point for a conversation with an end payer. As such, only the restitution figures are accounted for in the cost/benefit analysis and serve as a “floor” for determining positive social value with a successful intervention. By maintaining a conservative estimate, this analysis seeks to avoid criticism that the cost/benefit justification depends on inflated social cost estimates that may not directly apply to the specifically defined target population. Despite the focus on juvenile criminal activity, the Steering Committee chose not to include an estimate of adult offending or other reasonably predictable costs occurring beyond the two-year measurement window. This approach should keep any funding request specific for juvenile justice and highly targeted to partners at the state level. Cost/Benefit by End Payer Scenario 31 Justice Policy Institute. $ticker $hock: Calculating the full price tag for youth incarceration. December 2014 32 Washington State Institute for Public Policy. Benefit-Cost Results. http://www.wsipp.wa.gov/BenefitCost 33 Oregon Criminal Justice Commission. Cost-Benefit Methodology. July 2011 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 62 Based on guidance from the Steering Committee, three end payer scenarios were evaluated at length to determine the implications for selecting a cost effective intervention. The goal was to understand which combination of government entities would realize sufficient benefits from a successful intervention and be willing to pay for the cost of service delivery. Scenario A: Marion County Youth – Marion County End Payments  End Payer: Features only Marion County as an end payer.  County Placements Implications: o Any diversion of County Detention or GAP would predominantly benefit the county since it accounts for 100% and 33% of the costs of placement, respectively. o The avoidance of associated new criminal referrals and victimization, which are based on county costs, are also considered as a county benefit with successful diversion.  State Commitments Implications: o While youth who might escalate to an OYA placement are served (e.g. service provider must be paid to enroll them), successful diversion of OYA placements is a value that does not accrue to the county but rather a state- level entity (which in this scenario is not the end payer) since OYA Residential and OYA YCF are primarily funded by state funds. o The avoidance of associated new criminal referrals and victimization, which are based on county costs, are also considered as a county benefit with successful diversion. Scenario B: Marion County Youth - State-Level End Payments  Same as Scenario A  End Payer: The state is considered as the end payer in conjunction with the County (funding to be determined).  County Placements Implications: o Diversion of County Detention placement is accounted for as in Scenario A. o The benefit of GAP diversion can be attributed the state and county. State funds account for the roughly 53% of placement costs, with the remaining 13% coming from federal Medicaid match dollars. o The avoidance of associated new criminal referrals and victimization, which are based on county costs, are also considered as a county benefit with successful diversion.  State Commitments Implications: o Successful diversion of OYA Residential and OYA YCF is favorable to a state end payer since it directly funds approximately 60% and 100% of each placement, respectively. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 63 o The avoidance of associated new criminal referrals and victimization, which are based on county costs, are also considered as a county benefit with successful diversion. Scenario C: Marion County & Multnomah County Youth - State-Level End Payments  Scenario C adds Multnomah County’s target population to Scenario B.  End Payer: The state is considered the end payer in conjunction with the Counties (funding sources is to be determined).  County Placements Implications: o Diversion of County Detention and GAP placement is accounted for Marion and the State as in Scenario A. o For the Multnomah cohort, any successful diversion of County Detention or County Residential would not be factored into the project design o The avoidance of associated new criminal referrals and victimization, which are based on county costs, are also considered as a benefit to both counties with successful diversion.  State Commitments Implications: o Successful diversion of OYA Residential and OYA YCF is favorable to a state end payer since it directly funds approximately 60% and 100% of each placement, respectively. o The avoidance of new criminal referrals and victimization, which are based on county costs, are also considered as a benefit to both counties with successful diversion. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 64 EXHIBIT 20: Cost / Benefit Scenario Overview (1) Scenario C assumes headcount associated with Multnomah’s target population and the County’s objective of building a project only around OYA outcomes. However, the status quo outcomes of that population (e.g. length of stay, # of criminal referrals, and victimization costs) are of Marion County’s population. Cost Effectiveness Methodology & Conclusions The cost effectiveness of any program is a function of three variables.  The first is the benefit derived from an intervention. The prior section established how the benefit of reduced escalation, criminal referrals, and social costs is assessed.  The second is the intervention cost, which is based on the range for cost of service delivery in Exhibit 12 in Section: Intervention Assessment.  The last is a measure of efficacy. For the feasibility study, that is defined as how effective the intervention is at reducing escalations and criminal referrals among youth served. This measure of escalation prevention should be based on improvement above an expected escalation rate. For example, if 100 youth are served with the expectation that 40 would escalate without the intervention, reducing the number who escalate to 20 would be a 50% impact. The estimates here assume a uniform efficacy across placement types, meaning than an intervention is equally effective at diverting youth from detention as from YCF. A more realistic efficacy assumption can be made once an intervention is selected. The formula below describes the conditions under which an intervention “breaks even” or produces benefits that match its cost at a given rate of success. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 65 The following example, assuming the structure of scenario B (Marion County Youth - State-Level End Payments), illustrates how cost effectiveness is assessed for the feasibility assessment, assuming the state reaps the benefits of a successful program.  Assume that the current status quo costs in the absence of an intervention to serve 75 youth from the Marion target population is $2.6 million. This is inclusive of the cost of placement, cost of criminal referral, and social costs associated with their status quo outcomes.  Assume an intervention can be provided to all 75 youth for $8,000 each, or $600,000.  Under these parameters, an intervention would need to successfully divert from placement at least 23% of probation youth relative to expected escalation rate to be cost-neutral.  A project should be considered as viable under these assumptions if the cost of service delivery could be lower, if the successful diversion rate could be higher, or both. Consistent with the target population identified for Marion discussed in Section: Target Population Assessment, and the voluntary acceptance of a referral and acceptance rate of a service provider discussed in Section: Intervention Assessment, the chart below shows the expected baseline outcomes for a cohort of 75 Marion youth based on the actual outcomes for the 2013-2015. The scenarios that follow will be based on how effective the intervention is at improving outcomes above this baseline. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 66 EXHIBIT 21: Scenario B Status Quo Overview of cost Note: Headcount and length of stay are based on Exhibit 10 with adjustment for voluntary entry Cost per day figures are based on averages for Marion County and Oregon Youth Authority (2015) New referrals during probation period is based on Marion County youth (2012-2014) Restitution amount and incidence based on Marion County youth (2013-2015) Exhibit 22 below presents the necessary diversion rates to cover the intervention’s cost under end payer Scenario B assuming that the intervention is given to all eligible youth in the target population. This does not account for the cost of evaluation, project management, or other expenses beyond the intervention. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 67 EXHIBIT 22: Scenario B Breakeven Diversion Impact as a function of Intervention Cost The feasibility assessment did not evaluate a specific provider, so the intervention’s cost and effectiveness are hypothetical variables to be diligently evaluated in the RFI service selection process described in Section: Intervention Assessment and eventual RFP process. The graph above demonstrates the relationship between these two variables. The line reflects the formula for interventions that are cost neutral. Interventions with a cost effectiveness coordinate above the line should be considered whereas those below the line are worse than the status quo. Based on the status quo costs detailed in the previous section, this graph illustrates how costlier interventions require a higher diversion success rate to justify. Having defined three potential end payer scenarios (A through C), each case will result in a different cost/benefit matrix because of the number of eligible youth and possible sources of end payments per outcome. The matrices below display the cost-effectiveness implications for each scenario assuming all youth in the target population are served. Positive numbers reflect benefits in excess of program costs. The key insight is that scenario A (Marion County Youth – Marion County End Payments) has a narrower range for which to select a cost effective provider. An intervention would need to be both low cost and very successful to have even modest net benefits with minimal buffer for indirect costs. However, with cooperation from the state as an end payer to benefit from this program, costlier interventions are still viable for this project because the state has more to gain from successful diversion of costly placements to sufficiently justify the costs of service delivery. An overview of the cost / benefit matrices for scenarios A through C is presented below assuming various cost and success rate ranges: Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 68 A. Scenario A: Scenario A: Marion County Youth – Marion County End Payments Youth Enrolled: 75 Marion youth per annual cohort Status Quo Costs34 in the Absence of an Intervention: $1.3 million Cost / Benefit matrix depending on cost and impact of selected intervention: % Diversion Impact of Intervention See Appendix M for detailed cost description. Scenario B: Marion County Youth - State-Level End Payments Youth Enrolled: 75 Marion youth per annual cohort Status Quo Costs35 in the Absence of an Intervention: $2.6 million Cost / Benefit matrix depending on cost and impact of selected intervention: % Diversion Impact of Intervention See Appendix M for detailed cost description. 10% 20% 30% 40% 50% $4,000 $ (169,400) $ (38,800) $ 91,800 $ 222,400 $ 353,000 $8,000 (469,400) (338,800) (208,200) (77,600) 53,000 $12,000 (769,400) (638,800) (508,200) (377,600) (247,000) $16,000 (1,069,400) (938,800) (808,200) (677,600) (547,000) B. 10% 20% 30% 40% 50% $4,000 $ (43,500) $ 213,000 $ 469,500 $ 726,000 $ 982,500 $8,000 (343,500) (87,000) 169,500 426,000 682,500 $12,000 (643,500) (387,000) (130,500) 126,000 382,500 $16,000 (943,500) (687,000) (430,500) (174,000) 82,500 34 Defined as cost of placement, criminal referrals, and social costs 35 Ibid Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 69 Intervention Cost Intervention Cost per Youth per Youth Scenario C: Marion County & Multnomah County Youth - State-Level End Payments C. Youth Enrolled: 75 Marion and 90 Multnomah youth per annual cohort Status Quo Costs36 in the Absence of an Intervention: $3.9 million Cost / Benefit matrix depending on cost and impact of selected intervention: % Diversion Impact of Intervention Only diversions from OYA Residential and YCF are counted for Multnomah youth See Appendix M for detailed cost description. 10% 20% 30% 40% 50% $4,000 $ (271,900) $ 116,200 $ 504,300 $ 892,400 $ 1,280,500 $8,000 (931,900) (543,800) (155,700) 232,400 620,500 $12,000 (1,591,900) (1,203,800) (815,700) (427,600) (39,500) $16,000 (2,251,900) (1,863,800) (1,475,700) (1,087,600) (699,500) The matrices above are primarily illustrative. The precise cost effectiveness of an intervention would need to account for indirect costs such as evaluation and data-centric project management and include specific social benefits that could be reasonably expected based on the type of intervention. Impact of Improved Targeting at Intake The main reason the positive cost / benefit range in Scenario C is narrower than Scenario B (despite having all factors remain equal except the number of youth served) is that Scenario C, contains Multnomah youth that are being paid for and enrolled into the intervention that have no corresponding benefit to a state-level budget in the event of successful diversion. Many of these individuals are detention youth, which in a successful intervention would accrue value to the county not state. As OYA escalation is the only placement outcome being tracked for Multnomah youth, serving them has minimal state benefits that are captured in this project. To the extent program enrollment could be limited to only those youth whose placement outcomes impact state budgets (i.e. OYA placements), the cost effectiveness of any given intervention would improve. It is important to recall that the target population was defined as youth on probation with medium to high risk scores. Although JCP risk assessments are correlated with higher escalation rates, the project currently does not account for the OYA Escalation Risk measure to identify the highest cost youth at probation intake that go to OYA placements. A more precise predictive tool could ensure the intervention is delivered only to youth at high risk of escalation, thereby improving the cost effectiveness of the project across all scenarios. The matrix below demonstrates the positive cost / benefit impact of having hypothetical improved targeting for Multnomah youths in Scenario C. Because Multnomah’s focus is on serving the youth at risk of escalating to state facilities, the hypothetical scenario assumes the total number of Multnomah youth to be served is 50 (instead of 90 as assumed in Scenario C). It is also 36 Ibid Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 70 Intervention Cost per Youth assumed that the OYA escalation rate of Multnomah youth increases to 50% (instead of 30% as assumed in Scenario C). The more targeted headcount could be facilitated by using a hypothetical refined referral tool that would predict those youth most likely to escalate. By serving the riskiest youth, the cost effectiveness noticeably improves. Hypothetical Scenario C: Marion County & Multnomah County Youth - State-Level End Payments with OYA Escalation Risk score targeting Youth Enrolled: 75 Marion and 50 Multnomah youth per annual cohort Status Quo Costs in the Absence of an Intervention: $3.9 million C. % Diversion Impact of Intervention Only diversions from OYA Residential and YCF are counted for Multnomah youth Recommended Next Steps 1. Secure commitment of a state-level end payer: County-only projects, paid for by county funds, are not viable given the presumed costs of providing services to the target population. The benefits that accrue at the county level are insufficient to justify end payments large enough to cover a successful intervention. The Counties would have to place a very large weighting on the social costs without any budgetary impact to close this gap. A solution is for the state legislature to make success payments for diversion from OYA placements (and potential GAP placement and/or County Detention). An appropriation can either be done directly to the Counties or through OYA. The Working Groups explored alternatives, such as repurposing OYA diversion funds in Marion County but 100% of existing funds are being used to support the GAP program to allow youth to stay in their home county while receiving out-of-home support. Seeking to amend grants from Oregon Criminal Justice Commission to focus on a juvenile justice initiative like this one could be explored, albeit it is not considered to be a viable option among stakeholders given the commission’s current charge from the legislature to focus on the adult prison population. Another potential option is for the state to increase the amount of Diversion funds that OYA can pass through to the Counties. This is a difficult option as the Diversion funds are spread across the state based on a fixed formula. Alternatively, the state could appropriate funds directly to a pilot serving youth in one or more counties where the payments will be tied to outcomes. In any scenario, the Counties should engage OYA and other state-level decision makers and determine the best path for collaboration. 10% 20% 30% 40% 50% $4,000 $ (111,900) $ 276,200 $ 664,300 $ 1,052,400 $ 1,440,500 $8,000 (611,900) (223,800) 164,300 552,400 940,500 $12,000 (1,111,900) (723,800) (335,700) 52,400 440,500 $16,000 (1,611,900) (1,223,800) (835,700) (447,600) (59,500) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 71 Intervention Cost per Youth 2. Determine feasibility of cost-effectiveness requirements for available service providers: The cost-effectiveness framework was developed without reference to a specific intervention. As such, it may not reflect the cost structure and measured impact of providers able to serve the Counties. As mentioned in Section: Intervention Assessment, the Counties should use an RFI process to learn more about the capabilities and cost of interested providers. Indirect costs (e.g. start-up costs), to the extent they vary by provider, should be incorporated to provide a complete picture of cost-effectiveness for any project. One assumption that should be addressed early on with providers is whether their intervention can deliver consistent results for youth at risk of county and state placements. These youth may have different risk profiles and needs. The models presented above assumed identical rates of diversion across county and OYA placements, but different rates by placement are more likely realistic. Depending on the type of intervention provided, one group may respond better to the treatment, with the analysis updated accordingly. Prospective providers may need support in evaluating their programs to the standards required by this project. The OYA escalation scores were discussed in Section: Intervention Assessment as one option for assessing risk and measuring success. Should OYA make escalation to OYA scores accessible to service providers, it may allow for an understanding of what outcomes can be achieved, how to change programming to meet governments priorities, and improve general performance. 3. Set expectations for risk sharing between end payer and provider: The cost/benefit analysis established minimum cost and impact requirements for an intervention to break even. If an intervention delivers results above this threshold, the surplus benefits can be shared between the end payer and the provider (or third party funders as applicable). Similarly, if an intervention does not meet the baseline targets to break even, success payments will not be sufficient to cover program costs. If the project cannot be initially capitalized by the government end payers, the project leaders should consider raising third party financing or grants from the philanthropic community in the state or among national foundations focused on criminal justice. 4. Solidify willingness to pay for outcomes without a direct budgetary impact: The cost effectiveness framework is primarily driven by outcomes that have a direct budgetary impact. As discussed in the social costs section, this means the assessment is the lower bound of the true benefit of improved outcomes for the target population. A higher weighting of the social costs of juvenile offending may be justified based on how much an end payer values these other outcomes beyond their budgetary impact. The cost/benefit analysis of any potential project improves with the willingness to pay for non-budget impacting outcomes, such as reduced use of drugs and alcohol, enhanced involvement in pro-social activities and/or with pro-social associates, academic progress, and/or improved Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 72 school attendance. The Counties should establish their own willingness to pay for improved social outcomes, as well as the willingness of state level partners. 5. Explore incorporating OYA Escalation Risk scores to improve evaluation and enrollment targeting: Using the OYA Risk of Escalation score could simplify the process of attributing impact to the intervention. OYA Escalation scores could be used to establish a baseline expectation of outcomes for participants without the need for an evaluation requiring random assignment (such as a randomized control trial). Utilizing OYA Escalation scores eliminates the need for a control group, making it simpler and cheaper to measure the impact of the intervention. Lastly, these scores could be used to deliver the intervention to the youth with the highest likelihood of escalation. For example, if the total project budget is limited, focusing on youth with high predicted likelihood of escalation to OYA placements would improve the cost-effectiveness of the project because of the increased likelihood of actually diverting those youth from costly placements. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 73 Legal & Regulatory Assessment: Not Yet Demonstrated Not Yet Demonstrated Possible Promising Strong What are the options for end payer(s) and service provider(s) to enter into multi- year pay for success contracts? What are the options for end payer(s) to commit and appropriate success payments? Work Stream Goals:  Identify key elements of a performance-based contract  Outline alternatives for managing project risk (upfront funding & end payments)  Explore of strategies to secure end payment commitments  Define the role of a Special Purpose Vehicle, if needed Summary of Findings:  PFS is a type of performance-based contracting that strives to raise the bar across three areas: specific project objectives, performance structuring, and rigorous evaluation  Three potential project funding and payment approaches to consider are Traditional PFS, Partial PFS, and Performance-Based Contract with Deferred Payments. The first two options would require third party funding while the last alternative could be funded using traditional procurement mechanisms.  Three strategies for guaranteeing end payments include full faith and credit, appropriations (multi-year or annual), or rating agency trigger funding mechanisms  For this project, the holding mechanism for success payments or if performance-based contracts that cross biennia are possible in the state are unknown  Formation of a Special Purpose Vehicle to independently govern project contracts and funding could be achieved by the formation of a Special Purpose Entity in the State of Oregon, mostly likely as a 501(c)3 organization Recommended Next Steps:  None at this moment. Once an end payer and service provider are identified, the Counties should create a cross-agency contracting and legal working group from the state legislature, committed end payer(s), and other legal and political experts familiar with the budgeting and procurement processes for human services. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 74 The objective of this section of the feasibility assessment is to establish the contractual requirements of entering into a performance-based contract and to compare potential structures. This assessment does not constitute legal advice. Advice of legal counsel will have to be obtained to determine what is legally feasible within the state. I. Identify key elements of a performance-based contract: Document the requirements of creating a PFS contract II. Outline alternatives for managing project risk of upfront funding and end payments: Identify strategies for managing funding and end payments for a performance-based contract III. Explore strategies to secure end payment commitments: Overview of appropriation strategies for securing project end payments, including case studies from relevant launched projects. IV: Define the role of a Special Purpose Vehicle, if needed: Depending on the structure of a project, multi-party contracts may be best managed by an SPV. General guidelines are included on the role of an SPV and how it could be formed within the state of Oregon. Pay for Success (PFS) contracts are multi-year, multi-party performance-based contracts with an evaluation of specific outcomes. These contracts typically include a government end payer, service provider, and intermediary. These key parties must work together to agree on specific terms. An existing contract example of a launched project is included for easy reference in Appendix Q to provide additional context to the guidelines listed below. The sections that follow lay out the items the government end payer should consider and the historical precedent for these types of contracts. I. IDENTIFY KEY ELEMENTS OF A PERFORMANCE-BASED CONTRACT When discussing performance based contracting, it is important to note it is a type of government contracting with:  A clear set of objectives and indicators  Guidelines to collect data on the progress of selected outcomes and indicators  Performance-driven incentives with “upside” (bonus payments) or “downside” (injunctions) for service provider performance PFS is a type of performance-based contracting that strives to raise the bar across three areas:  Specific Project Objectives: Government and/or other end payers agree to pay for specific outcomes achieved by a service provider(s).  Performance Structuring: Before a contract is implemented, end payers and service providers mutually agree to the specific terms and conditions of the project including outcomes, evaluation plans, and success payments.  Rigorous Evaluation: Once launched, service providers begin delivering services while an independent evaluator rigorously assesses the impact of the services; end payers make success payments only when outcomes are achieved. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 75 When discussing the elements of a performance based contract, independent of how government end payments are sourced or if third party funding is required, the core principles of the agreement typically fall into the categories listed below. Note, each contract looks different and may not need to include every concept listed below, while some may include additional items. Exhibit 23 identifies contractual elements and related details. EXHIBIT 23: PFS Contract Elements PFS Contract Category Questions that Require Answers Description • How is success defined and measured? • When can success payments be triggered? • How is success priced? Specifies the payment amounts made at varying levels of success (e.g. $100 for every day of placement avoided per youth) and the maximum amount of success payments possible. Success Payment Calculation Success Payment Mechanics & Distribution  When will appropriations be made?  How will funds be held during the evaluation?  What will “trigger” the release of success payments? Specifies the mechanism through which the government will appropriate potential success payments, how those funds will be secured, and the timeframe for disbursement (described in greater detail below). • How will intervention participants be enrolled into the evaluation? • Which outcomes should be measured? • What is the appropriate evaluation design? • What is the structure for project reporting, dispute resolution, and project governance? • How will intervention operations, partnerships, and decision-making be managed? • Is the formation of a special purpose entity or any other vehicle required? Specifies how success will be evaluated and reported. Evaluation Terms • Who is the target population? What intervention is being provided to them? • What resources will be needed to provide those services? • How will individuals be enrolled into the intervention? Specifies the conditions critical to successful implementation (e.g. clear referral pathway of participants to intervention, government collaboration, and execution of the evaluation plan to validate the level of success). Implementation Support Specifies the structure for project reporting, dispute resolution, and project governance. This includes how to manage operations, partnerships, and decision making. Oversight & Reporting • How can all parties (e.g. end payer, service provider, and evaluator) enter into a multi-year contract? • What are the communication mechanisms? Ensures that all parties can enter into a multi-year contract, specifies the publicity protocols (namely for government end payer and project administrators). Representation & Warranties Exit • How do project parties course-correct during Specifies ways in which project parties Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 76 Thresholds & Remediation the contract? • What would cause project shut-down? can course-correct for various difficulties during the contract, and conditions under which the project will shut down. II. OUTLINE ALTERNATIVES FOR MANAGING PROJECT RISK (UPFRONT FUNDING & END PAYMENTS) PFS projects include performance-contingent payments, whereby service delivery and outcome measurement occurs prior to payment for the observed results. Given this payment timing gap, upfront working capital is needed for the service provider. This raises two key questions regarding project funding: 1. How will the project be funded? (i.e., funding sources, repayment structure) 2. What is the type and level of risk being borne by the parties? The project’s financing structure should be designed to complement the needs and priorities of its stakeholders. The following describes several potential project funding and payment approaches and outlines key considerations for each. The intent is to facilitate a discussion within the Counties, the Oregon Youth Authority, and any other state-level stakeholders that would be involved in the negotiation, funding, and designing of a juvenile justice PFS contract. An overview of priorities, sensitivities, and needs with respect to the PFS project’s financing is also provided. EXHIBIT 24: Spectrum of Performance-based Solutions Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 77 Option A: “Traditional” PFS Projects: Third party funders provide the upfront capital needed to cover total project costs How it works: Banks, philanthropic organizations, foundations, or other community entities provide funding for total project costs, typically through loans, program related investments (PRIs), or grants to the service provider. Assuming outcomes are achieved and validated, success payments are made by government to repay these initial funders. Participant Project Funding Source Funder Repayment Type/Level of Risk Outside Funder(s)  Loans/PRIs/grants to service provider equal to total project budget costs  Repaid with success payments achieved  Loss of investment (faces appropriation, implementation, impact risk) Government End Payer  Payments subject to evaluation (contingent funding)  Issue success payments to repay funders  Reputational risk  Unsuccessful outcome Service Provider  Typically contribute “skin-in- game” investment by either (i) contributing to upfront funding or (ii) deferring full receipt of success payments until target performance is achieved  N/A  Reputational risk  Unsuccessful outcome  Loss of investment (share in impact risk) Risk: Outside funders bear appropriation, project implementation, and performance/impact risks. Key Consideration: This option typically involves the highest outside fundraising need, since outside funders cover the full project costs. Fundraising for this amount may prove challenging and/or require the involvement of multiple funders. There are several additional considerations based on this:  Transaction costs – meaningful resources (financial and nonfinancial) committed to outside funder development, due diligence, interest payments for loan (if needed), legal counsel, etc.  Extended negotiation of project terms - multilateral negotiations between government end payer (e.g. state-level), service provider, project partner (e.g. county) and outside funders may lengthen discussions on project terms (as compared to bilateral negotiations between government end payer/project partner and service provider).  Additional scrutiny of project assumptions and structures - outside funders provide another external perspective on key project assumptions.  Project reporting requirements - external funders occasionally require reporting processes unique to their own organizations, or negotiate for additional Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 78 governance provisions. This can increase the project’s reporting and compliance burden A graphical representation of Option A is listed below in Exhibit 25. EXHIBIT 25: Option A: “Traditional” PFS Projects Option B: Partial PFS Projects: Government end payer provides portion of upfront funding need; Outside funders cover remainder How it works: Government provides upfront, non-performance contingent funding to cover part of the project costs. Outside funders finance remaining funding need. Success payments are made by government to repay funders. Total maximum success payments are decreased by the amount of government upfront funding. Participant PFS Project Funding Source Funder Repayment Type/Level of Risk Outside Funder(s)  Loans/PRIs/grants to service provider equal to total PFS Project Budget costs less upfront funding from government/service provider  Repaid with success payments achieved  Loss of investment (faces appropriation, implementation, impact risk) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 79 Government End Payer  Combination of upfront cost coverage (non-contingent funding) and payments subject to evaluation (contingent funding)  Issue success payments to repay funders  Reputational risk  Unsuccessful outcome  Loss of upfront funding (share in impact risk) Service Provider  Typically contribute “skin-in- game” investment by either (i) contributing to upfront funding or (ii) deferring full receipt of success payments until outcomes are achieved  N/A  Reputational risk  Unsuccessful outcome  Loss of investment (share in impact risk) Risk: Government and outside funders risk losing upfront funding/investment (share performance/impact risk). Key Consideration: This option lowers the outside funding need by the amount of funding provided by the government. Government bears some of the risk of low performance by providing upfront funding. Many of the considerations noted in “Traditional PFS” still apply:  Transaction costs - similar to “Traditional PFS,” albeit with decreased fundraising need  Negotiation of project terms - same as “Traditional PFS”  Additional scrutiny of project assumptions and structures - same as “Traditional PFS”  Project reporting requirements - same as “Traditional PFS” A graphical representation of Option B is listed below in Exhibit 26. EXHIBIT 26: Option B: Partial PFS Projects Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 80 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 81 Option C: Performance-based Contract with Deferred Payments Government provides a portion of upfront funding without third party financing; Success payments fund remainder of service delivery How it works: Government provides upfront, non-contingent funding sufficient to cover a portion of initial PFS project costs until success payments can be paid to the project and cover remaining project costs. Outside funders would likely not be engaged in this option. This option creates shared risk between the government and service provider. Participant PFS Project Funding Source Funder Repayment Type/Level of Risk Outside Funder(s)  N/A  N/A  N/A Government End Payer  Similar to a reimbursement for service model, except (i) a portion of payments (e.g. 20%) are withheld unless specific outcomes are achieved/validated and/or (ii) bonus payments can be disbursed with over-performance of specific outcomes  Issue success payments to service provider  Reputational risk  Unsuccessful outcome  Loss of upfront funding (share in impact risk) Service Provider  Typically contribute “skin-in- game” investment by either (i) contributing to upfront funding or (ii) deferring full receipt of success payments until outcomes are achieved  Recycle success payments to fund project costs in later years  N/A  Reputational risk  Unsuccessful outcome  Loss of investment or budget shortfall (share in impact risk) Risk: Government risks funding project that does not achieve desired impact in event of underperformance. At the same time, the service provider bears impact/performance risk in later years—if the project does not achieve expected levels of success, success payments earned and paid in later years might not sufficiently cover the service provider’s delivery costs. Key Consideration: this approach offers several project efficiencies and limitations, including:  Lower transaction costs – Fewer resources dedicated to outside funder development, due diligence, interest payments for loans, etc.  Potential to simplify reporting requirements - By removing outside funders, the government/service provider/evaluator could potentially streamline reporting and governance mechanisms, prioritizing those of mutual importance. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 82  Limited scale – Project size solely contingent on government resources, which are typically scarce to begin with. By decreasing the ability to leverage private capital with government dollars, project size could be constrained.  Streamlined negotiation of project terms - Bilateral negotiations between the government and service provider may simplify and expedite discussions on project terms (as compared to involving outside funders).  Risk no longer shared with external funding – By partially funding the project on the front-end, government bears the majority of the risk at the onset of the project. Since the remainder of cost coverage (and/or bonus payments) are to be paid subject to outcome delivery, risk shift toward service provider as the project progresses  Unknown replicability & sustainability - Should the government elect to replicate this contracting structure in its other programs, foregoing 100% external third party funding could offer the government greater autonomy and independence in pursuing additional contracts (i.e., no need to raise outside capital each time). However, the payment timing gap, between upfront project funding to execute and delivery of success payments, will still exist. Government will, therefore, have to provide some level of upfront, non-performance-contingent funding for each of these future projects. A graphical representation of Option C is listed below in Exhibit 27. EXHIBIT 27: Option C: Performance-based Contract with Deferred/Bonus Payments Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 83 Independent of how a contract is structured, whether its Options A – C or other alternatives, the government’s and service provider’s respective risk profiles will greatly depend on how several key contract provisions are negotiated. For instance, any contractual obligation to pay for any termination related expenses will influence who bares the greatest risk in the event of project shutdown or underperformance: i. In the event of government non-performance (e.g., non-payment of success payments owed; failure to appropriate), providers and/or funders may negotiate for government to fund all termination related expenses ii. In the event of provider non-performance (e.g., bankruptcy; failure to enroll minimum number of youths), government may negotiate for the provider to fund all termination related expenses. iii. In the event of project underperformance such that a service providers delivery costs are not fully covered through available success payments, government and provider may negotiate over which party pays for the budgetary shortfall. III. EXPLORE STRATEGIES TO SECURE END PAYMENT COMMITMENTS Significant investment in time and relationship building is required by the Counties to identify a willing and able state-level entity to sponsor a potential revenue source for project end payments. While identifying the most viable appropriations strategy is critical to advancing commitment to a project, so is the mechanism that secures and manages those funds and ultimately makes payments based on successful outcome delivery by a service provider. Whether it is a new appropriation from the state legislature or new use of funds from a state agency budget or Title IV-E Federal Reimbursement (via State Agency), various strategies can be explored in order to guarantee end payment for successful outcomes. Exhibit 28 below highlights three viable examples to consider: Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 84 EXHIBIT 28: Approaches to Multi-Year Contingency Contracts Source: Third Sector Capital Partners and Living Cities. For additional information on these alternative, parties should consult https://www.livingcities.org/blog/809-4-ps-of-pay-for-success-policy Deciding on which alternative to pursue is contingent on the risk appetite of a state-level end payer, political negotiations and prioritization (within county, state, and legislative bodies), and the budget upside that project success would entail. While the Steering Committee is exploring the political feasibility of securing a commitment to the project from a state-level end payer, legal counsel for the project should be prepared to understand any legal restriction on multi-year obligations for human services contracts within the state. For easy reference, Exhibit 29 below highlights case studies on funding mechanisms based on launched projects. The first two examples highlight criminal justice projects while the latter two are of other county-level projects. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 85 EXHIBIT 29: Funding Mechanisms for Success Payments of Launched Projects Source: Third Sector Capital Partners In the event appropriations are made and an account is established, there are a few noteworthy items to consider for how success payments might be managed. Consideration for Holding Mechanisms:  Appropriated Account - An appropriated account may carry forward unspent funds from one biennium to another biennium; however, legislative expenditure authority may be required to spend the funds across biennia.  Non-Appropriated Account - non-appropriated accounts are typically not subject to legislative expenditure authority. Non-appropriated accounts may be used by the state when revenue streams or expenditure timelines are difficult to predict.  A future Legislature can repurpose the fund balance in any account (including non- appropriated accounts), therefore funders (if any) and service providers may need assurance that success payments are available when earned. o To reduce risk, additional contractual language can specify that rating agencies will be notified in the event of success payment default or that funders’ funds will not be disbursed to service providers until an appropriation is made that will be available to make success payments. No matter if end payments are secured via an appropriation or policy commitment, projects require formal “triggers” to initiate payments for successful delivery and validation of outcomes. End payments for service delivery have traditionally been negotiated based on two objectives: Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 86  Cost-Savings - payments are disbursed from a holding mechanism should targeted cost- savings be met  Performance Metrics - Payments are disbursed from a holding mechanism based on the evaluation results All PFS projects currently launched in the U.S. make success payments tied to social outcomes that are not tied to realized cost savings. Those projects, however, estimate potential savings or cost diversion (as was done in Section: Economic & End Payer Assessment) to price outcomes and monitor such costs throughout the course of the project. IV: DEFINE ROLE OF A SPECIAL PURPOSE VEHICLE (SPV), IF NEEDED Depending on the structure of a project, multi-party contracts may be best managed by a SPV. The role of a SPV, as is the case in most launched projects, is to govern the flow of funds to and from all organizations (including evaluators and other administrative organizations) that are party to project contracts. SPVs are independent of government, funder, or service providers, but are jointly controlled through board representation. All project funds (e.g. fundraising, funding for service delivery, success payments, etc), typically flow through an SPV, as do decisions related to oversight and governance. SPVs are typically formed as 501(c)3 organizations or Limited Liability Corporations, depending on project preference, legal counsel and eligibility for philanthropic giving (if needed), and can be registered by either government, service provider, or other community organizations (e.g. foundations). See Appendix R for examples of other SPVs formed by relevant launched projects, and how they govern project contracts and funding. Under Oregon law, the legal structure that would correspond to an SPV is most likely a Special Purpose Entity (SPE). As defined in ORS 744.318(17), a SPE means a corporation, partnership, trust, Limited Liability Company or other similar entity formed solely to provide either direct or indirect access to institutional capital markets: (a) For a financing entity or licensed life settlement provider; (b) In connection with a transaction in which the securities in the special purpose entity are acquired by the owner or by qualified institutional buyers as defined in Rule 144 promulgated under the Securities Act of 1933, as amended; or (c) In connection with a transaction in which the securities pay a fixed rate of return commensurate with established asset-backed institutional capital markets. While a juvenile justice project developed by the Counties and state government may not require third party financing from institutional capital markets or issue debt-like securities with a rate of return, SPEs provide the appropriate governing structure for independently managing various multi-party contracts. So, while an SPV is not a type of legal entity in itself, Oregon law appears to provide the legal precedent for a governance and fund administrating entity to be formed as a Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 87 corporation, partnership, trust, 501(c)3 organization, Limited Liability Corporation, or any other entity. Should project leaders register a SPE as a 501(c)3 in the State of Oregon, the Secretary of State's Office has special requirements for registration. Please see Appendices O and P for a copy of requirements, typical time frames for formation, and formal registration form for the State of Oregon. The laws regulating 501(c)3s, including membership and administration, are found in the chapter 65 of the Oregon Revised Statutes.37 Other business organization laws are in chapters 62 through 70.38 Recommended Next Steps None at this moment. Once an end payer and service provider is identified, the Counties should create a cross-agency contracting and legal working group from the state legislature, potential end payer, and other legal and political experts familiar with the budgeting and procurement processes for human services. These individuals should be able to provide specific expertise and perspective regarding securing multi-year end payment commitments, holding and releasing such funds, and legal requirements related to forming any entity that would govern the project and related funds. 37 Oregon State Legislature. Retrieved March 2016 via https://www.oregonlegislature.gov/bills_laws/lawsstatutes/2013ors065.html 38 Oregon State Legislature. Retrieved March 2016 via https://www.oregonlegislature.gov/bills_laws/Pages/ORS.aspx Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 88 Funder Development: Not Yet Demonstrated Not Yet Demonstrated Possible Promising Strong Is there funder interest in continuing to develop a PFS project and invest in the up- front funding required by a PFS contract? Work Stream Goals:  Identify potential funder network to fund project  Develop project materials to educate and engage local funder community on the value proposition of the proposed project Summary of Findings:  Due to not having a committed end payer and identified intervention model or service provider, it is unclear if the project would require third party upfront funding. Detailed engagement with local third party funders is at this phase premature.  As the Counties identify a potential end payer, continued identification of prospective funders can help determine interest in financially supporting specific project work streams (e.g. data building efforts or evaluation) or providing upfront funding for the entire project (if needed). Recommended Next Steps:  Without a committed end payer or identified service provider, it would currently be premature to engage third party funders (if needed).  Continue identifying funders who have exhibited strong interest in supporting a potential project.  Potential PFS projects should leverage the local funder community to provide project funding or work stream support (e.g. evaluation), and to demonstrate community support for the project goals and intervention. I. IDENTIFY POTENTIAL FUNDER NETWORK TO FUND PROJECT Due to not having a committed end payer or service provider, at this stage it would be premature to engage third party funders. Project leaders, including members on the Steering Committee would require guidance from a committed end payer and selected intervention on whether or not to seek third party funding. Third party funding is an optional component of PFS when government resources cannot adequately fund a service provider at the onset of an intervention. Funding from private sources, ranging from local philanthropy to commercial banks, could be a source of capital drawn down by a service provider. Such funds allow a provider to execute its project goals while intervention Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 89 results are being evaluated and before payments are made by an end payer. Traditional PFS projects, including all 11 launched projects, are structured this way. Exhibits 25 and 26 in Section: Legal & Regulatory Assessment highlight the potential flow of third party funds for a project. Should the Counties elect to fundraise upfront capital for the project, it is essential to create a project profile for third party funders and investors. Such a document would guide communications and engagement efforts on topics such as desired outcomes, top questions and key messages. Below are various project topics for the Counties to consider before meaningfully engaging with funders:  Funders want to know that their grants and investments are making a difference in addressing a particular need.  Funders want to see programs reporting on results and outcomes, rather than activities and outputs.  Many performance-oriented funders want to affect long-term change in government systems, particularly in supporting government to make the shift to funding preventative interventions.  Funders want to be part of something that is innovative in its financial approach, while also helping trigger a longer-term commitment for sustainable support from the government. If governments can fund an intervention at the onset of the project or can negotiate a contract with deferred payments with a well-capitalized service provider that can cover its own operational costs for launch, then government should proceed without third party funding. Exhibit 27 in Section: Legal & Regulatory Assessment provides an overview of this structure. Given that the project discussed in this feasibility assessment is considering outcome measurement within a two year time frame, the short time frame allows for the Steering Committee to strongly consider a Performance-Based Contract with a Deferred / Bonus Payments structure highlighted in Exhibit 27. Under this structure, should the project also elect to form a special purpose vehicle it would likely not be to fully manage the exchange of funds, but to govern the administration of the project (e.g. contract, oversight of evaluation, authority to make end payments, periodic reporting to stakeholders, etc). II. DEVELOP PROJECT MATERIAL TO EDUCATE & ENGAGE LOCAL FUNDER COMMUNITY The feasibility assessment provides sufficient context and details to any potential funder or stakeholder interested in learning more about the project. While it may be the case that the project may not require third party funding, outside funders can still play a critical role in catalyzing the development of this initiative. Outside funders – like philanthropic organizations, private financial institutions, and/or individual investors – can pay for Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 90 or subsidize specific project work stream (e.g. data analytics, evaluation, legal services, etc). Oregon is home to a variety of philanthropic organizations committed to evidence-based practices and service alignment, including Meyer Memorial Trust, Oregon Community Foundation, Northwest Health Foundation, and Umpqua Bank. Multnomah and Marion Counties recently began discussions with a core group of foundations to explore innovative ways of aligning public/ philanthropic investments. In July 2017, the Counties in partnership with the Oregon Community Foundation and the two other organizations exploring PFS projects (The Center for Evidence-based Policy at Oregon Health and Science University and Friends of the Children), hosted a philanthropic convening with nearly 50 individuals representing over 20 local organizations. The objective of the gathering was to discuss lessons learned from Oregon’s three existing Pay for Success projects, their alignment with the priorities of philanthropy, and opportunities for promoting innovation and building common infrastructure. Hosting future gatherings with all or a subset of attendees would be recommended in order explore how predictive analytics and opportunities to foster social service innovation and scale effective programs can be supported by philanthropy. Recommended Next Steps  Due to not having a committed end payer or identified a service provider, at this stage it would be premature to engage third party funders. Project leaders, including members on the Steering Committee would require guidance from a committed end payer and selected intervention on whether or not to seek third party funding.  Continue identifying funders who have demonstrated strong interest in supporting a potential project. Interest may be around criminal justice, data investments, outcomes- based contracting, or another aspect of a potential project.  Potential PFS projects should leverage the local funder community to provide project funding or work stream support (e.g. evaluation), and to demonstrate community support for the project goals and intervention. Strong local participation can help open doors to regional and/or national funders, and serve as a public advocate if needed. Should the Counties decide to pursue this project, philanthropic organizations should be engaged to determine their interest in the project and, if/when ready, request their financial support. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 91 Appendices Appendix A: Membership in Feasibility Assessment Working Groups STEERING COMMITTEE Scope Time Commitment Reporting Member Leads Judy Shiprack Janet Carlson Members Faye Fagel Jan Fritz Jeff White Brigid Zani Jan Calvin Christina McMahan John Tydlaska Marissa Madrigal Adam Renon Fariborz Pakseresht Christine Krik Laura Furr Jodi Hack Jessica Vega Pederson Brian Renauer Ryan Labrecque Christopher M. Campbell Oscar Benitez Brian Beachkofski Ernest Brown John Ginther Michael Cwidak-Kusbach Alberto Ramos Will Toaspern Jerry Croan • Project Direction • Oversight • Delegation • Resource Allocation • Risk Management & Troubleshooting Every month and as needed n.a. Title Commissioner Commissioner Director, Juvenile Dept. Deputy County Administrative Officer Chief Financial Office Management Analyst, Juvenile Dept. Consultant Juvenile Services Division Director Economic Development Coordinator Chief Operations Officer Policy Advisor Director Public Policy & Gov't Relationships Director Asst. Director, Program Manager Justice Reform and Youth Engagement State Rep District 19 - Marion State Rep District 47 - Portland Researcher Researcher Assistant Professor Associate Sr. Director Sr. Analyst Associate Associate Associate Analyst Senior Fellow Organization Multnomah County Marion County Marion County Marion County Marion County Marion County Full Circle Consulting Multnomah County Multnomah County Multnomah County Multnomah County Oregon Youth Authority Oregon Youth Authority National League of Cities Oregon Legislature Oregon Legislature Portland State University Portland State University Portland State University Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 92 WORKING GROUP - OUTCOMES & EVALUATION Scope Time Commitment Reporting Member Leads Brigid Zani Adam Renon Members Faye Fagel Vicky Johnson Bobbie Cogswell Jan Calvin Christina McMahan Deena Corso Kim Bernard Sherri Campbell Liang Wu Andrew Grover Phil Cox Paul Bellatty Margaret Braun Kristi Racer Laura Furr Christopher M. Campbell Oscar Benitez Brian Beachkofski Ernest Brown John Ginther Michael Cwidak-Kusbach Alberto Ramos Will Toaspern • Data Sharing • Pop. Targeting & Referrals • Pop. Needs Assessment • Intervention Design • Target Outcomes • Evaluation Plan Daily / weekly Steering Committee on a weekly or bi-weekly basis Title Management Analyst, Juvenile Dept. Policy Advisor Director, Juvenile Dept. Probation Supervisor, Probation Services Probation Officer Consultant Juvenile Services Division Director Sr. Manager, Dept of Community Justice Juvenile Svcs DCJ Data Manager Senior Grants Coordinator Research & Evaluation Asst. Director, Oregon Asst. Director, Community Services Asst. Director, Community Services Sr. Research Analyst Research & Evaluation Program Manager, Justice Reform & Youth Assistant Professor Associate Sr. Director Sr. Analyst Associate Associate Associate Analyst Organization Marion County Multnomah County Marion County Marion County Marion County Full Circle Consulting Multnomah County Multnomah County Multnomah County Multnomah County Multnomah County Youth Villages Oregon Youth Authority Oregon Youth Authority Oregon Youth Authority Oregon Youth Authority National League of Cities Portland State University Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 93 WORKING GROUP - OPERATIONS Scope Time Commitment Reporting Member Leads Brigid Zani Adam Renon Members Faye Fagel Christina McMahan Vicky Johnson Bobbie Cogswell Jan Calvin Deena Corso, Joyce Resare Les Taylor Sherri Campbell Phil Cox Laura Furr Oscar Benitez Brian Beachkofski Ernest Brown John Ginther Michael Cwidak-Kusbach Alberto Ramos Will Toaspern • Intervention Design • Provider Selection • Ops. Plan • Ops. Budget • Pilot development Daily / weekly Steering Committee on a weekly or bi-weekly basis Title Management Analyst, Juvenile Dept. Policy Advisor Director, Juvenile Dept. Juvenile Services Division Director Probation Supervisor, Probation Services Probation Officer Consultant Sr. Manager, Dept of Community Justice Juvenile Svcs Manager, Business Services, Dept of Comm. Justice Youth Services, Dept of Community Justice Senior Grants Coordinator Asst. Director, Community Services Assistant DirectorProgram Manager Justice Reform and Youth Engagement Associate Sr. Director Sr. Analyst Associate Associate Associate Analyst Organization Marion County Multnomah County Marion County Multnomah County Marion County Marion County Full Circle Consulting Multnomah County Multnomah County Multnomah County Multnomah County Oregon Youth Authority National League of Cities Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 94 WORKING GROUP - FINANCE & REGULATORY Scope Time Commitment Reporting Member Leads Jan Fritz John Tydlaska Members Brigid Zani Camber Schlag Bruce Armstrong Joyce Resare Adam Renon Sheri Campbell Brian Smith Mike Jaspin Jacquie Weber Oscar Benitez Brian Beachkofski Ernest Brown John Ginther Michael Cwidak-Kusbach Alberto Ramos Will Toaspern • Economic Model • Funder Outreach • Contracts & Agreements • SPV Development • End-payer Appropriations Daily / weekly Steering Committee on a weekly or bi-weekly basis Title Deputy County Administrative Officer Economic Development Coordinator Management Analyst, Juvenile Dept. Contracts and Procurement Manager Office of Legal Counsel Manager, Business Services, Dept of Comm. Justice Policy Advisor Senior Grants Coordinator Purchasing Manager Economist, Deputy Budget Director Deputy County Attorney Associate Sr. Director Sr. Analyst Associate Associate Associate Analyst Organization Marion County Multnomah County Marion County Marion County Marion County Multnomah County Multnomah County Multnomah County Multnomah County Multnomah County Multnomah County Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Third Sector Capital Partners Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 95 Appendix B: Juvenile Crime Prevention (JCP) Assessment 2006.1 Risk Indicators for Eligible Youths A youth with the JCP score in the “medium” to “high” risk range. Youth with a score between 6 and 13 are medium risk youth. High risk adolescents would have scores 14 to 30. The score is automatically calculated in JJIS. The second level of screening with the JCP items was youth had to have at least 8 of the specific risk factors listed below. When a factor is positive, the youth received a point if that factor was absent. 2.0 SCHOOL ISSUES Case Planning Domain: Education  PF2.1 Significant school attachment/commitment (has significant attachments, beliefs, commitment and/or involvement with and within his/her school; youth motivated to do well in school).  R2.2 Academic failure (recently failed, or currently failing two or more classes; not meeting minimal academic standards; not performing at grade level appropriate to youth’s age).  PF2.7 Family actively involved in helping youth succeed in school (helps with homework, provides transportation to school, talks with teachers, etc.). 3.0 PEER RELATIONSHIPS AND OTHER RELATIONSHIPS Case Planning Domain: Life/Social Skills  PF3.1 Friends disapprove of unlawful behavior (associates on a regular basis with more than one friend who disapproves of unlawful acts such as stealing, physically hurting others, vandalism, etc.).  R3.2 Friends engage in unlawful or serious acting-out behavior (has one or more friends or routine contact with peer(s) who actively engage in unlawful behaviors including delinquency, substance abuse, or violent activities.).  R3.3 Has friends who have been suspended or expelled or dropped out of school (associates with one or more  T3.5 Substance abusing friends (youth hangs out with one or more other youth who use alcohol and/or drugs on a regular basis, for example at least several times per month). 4.0 BEHAVIOR ISSUES Case Planning Domain: Offense Specific  PF4.5 Involved in constructive extra-curricular activities (sports, clubs, student or religious groups, practice of music, theater, or other arts).  R4.10 Behavior hurts youth or puts her/him in danger (check if has been true at any time in the past) (limit to physical harm or threat of harm; e.g., attempted suicide, riding in a Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 96 vehicle with a teenage driver who had been drinking or using drugs, taking other excessive risks). 5.0 FAMILY FUNCTIONING Case Planning Domain: Family  PF5.1 Communicates effectively with family members (shared communication is both verbal and nonverbal and includes establishing and maintaining healthy relationship boundaries).  R5.2 Poor family supervision and control (family does not know where the youth goes, what he or she does, or with whom, and has little or no influence in such matters).  R5.4 History of reported child abuse/neglect or domestic violence.  PF5.10 Has close, positive, supportive relationship with at least one family member (at least one family member has a supportive relationship with the youth, encourages the youth, and provides recognition for achievements). 6.0 SUBSTANCE ABUSE Case Planning Domain: Substance Use  R6.1 Substance use beyond experimental use (uses alcohol and/or other drugs regularly).  R6.2 Current substance use is causing problems in youth's life (youth is having problems with school, the law, family, friends or community related to alcohol/drug use). 7.0 ATTITUDES, VALUES, & BELIEFS Case Planning Domain: Life Skills  T7.3 Youth accepts responsibility for behavior.  T7.6 Youth preoccupied with delinquent or anti-social behavior. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 97 Appendix C: Descriptive Overview of Juvenile Justice Placements County Detention:  Detention centers operates 24 hours a day, 365 days a year within the guidelines and requirements established by both Oregon State Statute and County policy  Detention provides a safe and secure environment for the temporary custody of delinquent youth who: o Have been charged by law enforcement with attempting or committing an offense o Who present the highest risk to community safety o Have willfully failed to appear at one or more juvenile court proceedings o Are on juvenile probation and have failed to comply with one or more of the conditions of probation o Youth receive education, counseling, medical services, physical education and recreation, and cognitive and other skill development programs. County Residential:  For Multnomah, the Juvenile Services division operates a wide array of programs for youth requiring formal or informal community supervision. Programs exist for youth involved in gangs, requiring drug treatment services, and with other specific needs. The Donald E. Long Juvenile Detention Facility is staffed 24 hours per day, 7 days per week to provide a safe and secure setting to detain youth ages 12-17 with an average population of 64 youth.  For Marion, County Residential in this project is referred to as the Guaranteed Attendance Program (GAP). o Staff-secured shelter care facility that serves youth who volunteer to participate in the ninety-day program. o Accommodates up to 30 youth at a time separated into two buildings for male and female youth. The waiting list for a spot in the program averages between 15 and 25 youth at a time. o The program operates afternoons through morning on weekdays and 24 hours on weekends and holidays. All youth placed in the GAP program have day programs elsewhere, including: public school, Marion County Alternative Programs, or jobs in the community. The youth return to GAP daily in the afternoon and spend the evening and night there. o GAP began in November 1995 as a way to address the need for an alternative to Detention where youth could be held accountable in a staff-secured setting, yet still attend their daily programs. Since then the program has changed to a ninety-day treatment program. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 98 OYA Residential Treatment  OYA Residential uses Behavior Rehabilitation Services (BRS) which improves the lives of children and adolescents with debilitating psychosocial, emotional and behavioral disorders by providing behavioral intervention, counseling, and skills-training services  Youth are committed by the County court to the Oregon Youth Authority for placement in a residential treatment program  OYA contracts with private providers for residential treatment services and foster care. OYA probation and parole officers supervise and coordinate case plans and services for youth committed to Oregon Youth Authority and placed either in residential treatment programs or Youth Correctional Facilities and continue from program or facility release to case closure OYA Youth Correctional Facility (YCF)  Youth correctional facility programs provide the highest levels of security and structure within the OYA close custody system. These facilities are sited across the state and serve varied populations. Operating capacities vary from MacLaren Youth Correctional Facility serving 345 offenders to North Coast and Eastern Oregon Youth Correctional Facilities serving 25 each.  Youth live in group settings and many of their daily activities are conducted within living units. A continuum of reformation and rehabilitative services are provided both by OYA employees and contracted providers from the private sector. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 99 Appendix D: Organized Skepticism of Oregon Pay for Success Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 100 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 101 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 102 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 103 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 104 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 105 Other relevant literature authored and distributed by the Oregon Center for Public Policy includes “A Guide to Evaluating Pay for Success and Social Impact Bonds” which may be accessed online at http://www.ocpp.org/media/uploads/pdf/2015/12/rpt201512-guide- pfs-sib.pdf Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 106 Appendix E: Tough Questions on Pay for Success: Core Criticism of the Model 1. PFS forces governments to pay private investors a high rate of return to fund programs that could be paid for directly by government at a lower cost to taxpayers. (Wall Street Bankers shouldn't be making a profit off the backs of people in need.)  The reality today is that high-impact service providers don’t have access to the funding they need.  This is primarily due to the fact that government support for social programs isn’t always tied to results. But it’s also because governments tend not to fund the entire cost of a program up front, leaving service providers with an uncertain and inadequate funding stream.  Pay for Success solves these problems by tapping private funders to fund the full up-front costs of the program and by establishing performance goals that allow all partners to track outcomes over time.  The private funders recoup their original investment only if the program succeeds in meeting the performance goals set out by independent project managers. If the program exceeds the performance goals, the private funders earn a modest return.  Because Pay for Success typically focuses on funding preventative services, their success saves governments money by reducing their need to spend down the road on more costly programs. The longer-term savings more than offset the slightly higher costs of a PFS project.  By embracing the Pay for Success model, more funding will go to programs that work, more lives will be improved, and more accountability will drive smarter government spending. 2. PFS does not shift the risk of funding social programs from taxpayers to investors. In reality, PFS funding is directed almost exclusively to programs that are already proven to work, representing little-to-no risk to investors. (“PFS doesn’t deliver on its promise— it’s selling taxpayers a bill of goods.”)  The investments made by private funders to fund the full up-front costs of a program are absolutely at risk. There is no guarantee whatsoever that the program will meet or exceed the agreed upon metrics, which are established collaboratively by service providers, government, and independent project managers, and set at ambitious but achievable levels. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 107  But this misses the larger point, which is that government should be allocating greater resources to social programs that actually work.  Pay for Success expands funding for effective social programs by tapping new private investments to cover the up-front costs of the programs.  By tracking outcomes over time, governments gain access to better information about the effectiveness of programs and have the ability to make smarter decisions about future spending, because independent project managers establish performance goals for the programs that must be met for government to repay the upfront investment.  Pay for Success helps drive resources toward programs that actually work, delivering better outcomes for people in need and improved accountability for spending. 3. PFS siphons money from social programs to pay a range of third parties that are needed to “manage” the projects. That means less money for people in need and more costs to taxpayers. (“PFS lines the pockets of intermediaries at the expense of people in need.”)  People in need deserve access to programs that actually work.  The benefit of Pay for Success is simple: it expands funding for effective social programs by tapping new private investments to cover the up-front costs of the programs.  Taxpayer dollars are protected, and governments have the ability to make smarter decisions about future spending, because independent project managers identify high- performing providers and establish performance goals for the programs that must be met for government to repay the upfront investment.  Far from delivering less money and more costs to taxpayers, Pay for Success does exactly the opposite by driving greater resources toward programs that actually work, delivering better outcomes for people in need and improved accountability for spending. 4. PFS forces nonprofits and charities to focus on hitting metrics rather than helping people. (“PFS is like education reform – it forces service providers to game the numbers rather than focusing on people”)  All parties involved in a Pay for Success program, including the nonprofits and charities, have a shared goal: to improving the lives of people in need. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 108  While not every aspect of achieving that goal can be precisely measured, we do have better tools than ever to measure outcomes and track them over time.  Doing so does not detract from a focus on individuals—it is a compliment to that commitment, helping drive more resources towards those programs that are truly making an impact for the people and communities they are meant to serve.  The flexible and reliable multi-year funding provided to service providers through Pay for Success allows them to structure programs to serve the unique needs of the communities in which they work, without having to worry about constantly fundraising. Additional Questions Won’t this end up costing taxpayers more, when projects are successful?  No, for two reasons: If programs are unsuccessful, taxpayers don’t pay anything. If they are successful, taxpayers do pay back the initial investment, plus a modest return, but that cost is more than offset by the savings incurred over the course of the program.  Pay for Success helps drive resources toward programs that actually work, delivering better outcomes for people in need and improved accountability for spending. How do you know the projects will be successful?  Like all social programs, when Pay for Success programs are launched we cannot be certain they will be successful—but unlike other programs, with Pay for Success taxpayer dollars will only be spent if the program actually does measurably improve the lives of those it is meant to help.  In order to maximize the likelihood of success, Pay for Success programs are thoughtfully created with all the different organizations involved having a voice in setting goals and expectations.  Pay for Success also taps upfront, reliable, and flexible funding for service providers, allowing them to design programs that uniquely serve the communities in which they work. These programs typically include provision of integrated services aimed at changing outcomes for people and families, which help prevent longer-term problems down the road. Who decides the metrics for success? Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 109  All the partners who are involved in each unique Pay for Success program work collectively to set goals for success.  This work is facilitated by independent project managers that coordinate the diverse partners involved in a Pay for Success project and helps the partners work together to set goals that reflect true success for the program, yet are also realistically achievable. What happens if the Pay for Success program doesn’t achieve its outcomes?  If the program does not achieve its target results, taxpayers do not need to repay those who made the original investment. This model ensures that taxpayer dollars are being spent only on programs that actually work.  However, the goal of Pay for Success is always to drive resources toward programs with a proven track record of success that have the highest potential for achieving better outcomes for people, families, and communities. Why are private funders interested in funding Pay for Success projects?  More than ever, funders are seeking creative opportunities to put their money to work in ways that are proven to help people in need. Private funders have limited opportunities to make investments that measurably improve lives and offer the potential of making their money back or even realizing a modest return. PFS offers funders the opportunity for that kind of win-win situation, while also growing the pool of resources available for high-impact service providers.  With this new model, funders will know the outcomes of the programs they are funding, while also giving back to their communities in meaningful ways by investing in proven preventative interventions that improve lives.  Funders of PFS programs leverage significant resources for providers to track outcomes over a longer period of time and demonstrate a meaningful impact for individuals and communities. Why should we trust financial institutions to solve problems government hasn’t been able to?  Pay for Success is ultimately about finding innovative ways to solve social problems. It is about bringing together a wide range of stakeholders including service providers, government, independent project managers, and funders to create a platform where outside investment can be used to support programs that measurably improve people’s lives. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 110  We know that governments face significant budget constraints, making it harder to allocate necessary funding to help those in need. Pay for Success makes it possible for communities to deliver larger scale and better services for people most in need, without additional burdens on taxpayers.  Funders play a significant role in Pay for Success projects by providing the investment required for the programs to get off the ground. However, the role of the funder is very clearly defined and limited in scope. Government, service providers, community organizations, and the project managers that bring it all together ultimately drive the project forward.  Private funders are looking for innovative and meaningful ways to make a difference. Pay for Success gives them an opportunity to invest in their communities by supporting high- quality programs that improve the lives of people in need, with the ultimate goal of receiving both a financial and social return. Doesn’t the focus on established service providers work to exclude those nonprofits without the upfront working capital to plan and design projects?  Not at all. The goal of Pay for Success programs is to support innovative nonprofit service providers who are working to address some of the biggest social issues of the day. By expanding funding for high-quality programs that actually deliver results, Pay for Success gives service providers an opportunity to scale resources in a way that they never could before, bringing significant new capital to traditionally under-resourced issues. What is the rate of return for funders in a Pay for Success project?  Private funders are looking for innovative and meaningful ways to make a difference. Pay for Success gives them an opportunity to invest in their communities by supporting high- quality programs that improve the lives of people in need, with the ultimate goal of receiving both a financial and social return.  If the program is successful in delivering services, the government repays those who made the original investment. If, and only if, the program exceeds the pre-determined outcomes, the government pays a small return on the investment.  If the program does not achieve its target results, government does not repay those who made the original investment. This model ensures that taxpayer dollars are being spent only on programs that actually work.  At the highest level, the rate of return for this project is XX%. (Or: You can refer to our project fact sheet about the specific terms of the project.) However, because returns are paid only at the highest levels of success, the longer-term savings more than offset the modest rate of return paid. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 111 Is it possible that through the PFS model, private capital might overly influence the decision-making and priorities of government?  Pay for Success projects start with government identifying a problem that needs to be solved. The process of setting up a Pay for Success project is a highly collaborative effort involving government, service providers, community organizations, funders, and a project manager that helps bring it all together.  One of the benefits of Pay for Success is the ability to provide upfront, reliable, and flexible funding to service providers, giving them an opportunity to scale resources and reach more people. This allows government to work in partnership with nonprofit service providers and private investors to effectively address deep-seeded social problems.  All Pay for Success projects are tracked by an independent evaluator, which ensures that the project is working toward meeting and exceeding its goals, while also monitoring that the program is being implemented as agreed upon by all partners in the project. How are you working to prevent false reporting by service providers?  Independent project managers evaluate service providers throughout the duration of the contract. Goals are set collaboratively with service providers to ensure they are ambitious, yet realistic performance measures that strengthen the program and the services it delivers.  By embracing the Pay for Success model, service providers across the country can gain access to more funding for the most effective programs, allowing them to better fulfill their missions and reach more people in need. How will changes in political landscapes affect the success of PFS projects?  Pay for Success projects are usually designated for specific time frames that can span several years, and are agreed upon by all interested parties, including the service provider, government, private investors and independent project managers. By design, changes in political landscapes should not affect Pay for Success projects once they are launched.  Pay for Success projects have been launched or are in development in diverse areas of the country and have been championed by elected officials across party lines. What is the role of government vs. that of the private sector to solve our greatest societal issues? Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 112  The premise of Pay for Success programs is to bring together uncommon partners for the common good.  Private funders help enable these innovative collaborations that support a better, more robust partnership—between government, service providers, community organizations, and independent project managers—to solve society’s biggest challenges. By combining the expertise of all partners, Pay for Success helps drive resources toward programs that actually work, delivering better outcomes for people in need and improved accountability for government spending. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 113 Appendix F: Service Provider Request for Information (RFI) Template INTERNAL NOTE: The template included below is a component of the government RFI processes. The contents highlight RFI Purpose (brief background), the county’s priorities (target population, outcomes, and necessary intervention components), and requested information (proposed services, impact measurement, service costs, evaluation, and data access). For completion of the RFI process, government should consider standard legal language for public records, RFI disclosure, and any other legal requirements. Lastly, consideration should be given for the submission timeline (time for follow up questions or hosting of any meetings for respondents) and vendor response requirements (automated/digital response template or paper submissions). PURPOSE The purpose of this Request for Information (RFI) is to help Marion County (“County”) obtain information regarding intensive in-home interventions for youth at risk of an out-of-home placement that would be well suited for an outcomes-based contract. The County seeks to implement an outcomes-based contract to fund services for at-risk youth in the County. The County seeks to obtain information on promising interventions that have a strong evidence base or research driven best practices to support achievement of desired outcomes. COUNTY’S PRIORITIES Marion County Juvenile Department is seeking to provide intensive in-home intervention for youth at risk of out-of-home placement. With this intervention we expect the target youth (described below) to have fewer new criminal referrals, reduced out-of-home placements, engagement or re-engagement in education or vocational training and stability in their home community. A. Target Population Eligible youth will meet the following criteria: 1. ages 13 through 18 2. placedoncountyprobation 3. assessed as medium through high-risk based on Juvenile Crime Prevention (JCP) criteria (see Appendix A) 4. have 8 or more of these 15 specific JCP items: a) Substance use beyond experimental use (uses alcohol and/or drugs regularly) b) Current substance use is causing problems in youth's life (youth is having problems with school, the law, family, friends or community related to alcohol/drug use) c) Youth accepts responsibility for behavior d) Youth pre-occupied with delinquent or anti-social behavior e) Friends disapprove of unlawful behavior f) Friends engage in unlawful or serious acting-out behavior Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 114 g) Has friends who have been suspended or expelled or dropped out of school h) Substanceabusingfriends i) Communicates effectively with family members (both verbal and nonverbal shared communication with healthy relationship boundaries) j) Poor family supervision and control (don't know where youth goes, what youth does, and has little influence in such matters) k) Hasclose,positive,supportiverelationshipwithatleastonefamilymember l) Significant school attachment/commitment (has significant involvement/commitment to school) m) Academic failure (recently failed, or currently failing two or more classes) n) Family actively involved in helping youth succeed in school o) Involved in constructive extra-curricular activities (sports, clubs, music, theater, arts, etc.) Based on historical trends, the eligible target population is estimated to be approximately 100 county youth throughout the duration of a calendar year. As a voluntary service, we expect less than that number to accept services. B. Intervention The County will choose interventions that are evidence-based, or utilize research driven best practices and have demonstrated success in achieving measurable outcomes that can be scaled up while maintaining fidelity to the intervention model. Mandatory requirements of a successful intervention are as follows: 1. Bicultural/bilingual competency in service delivery 2. Intensiveservicesdeliveredinanin-homesetting 3. CapacitytoserveyouththroughoutMarionCounty 4. Willingness to participate in rigorous impact evaluation (e.g. propensity score matching) C. Outcomes The County has identified the following outcomes for the target population to be measured in any contract: 1. Reductioninout-of-homeplacements: a) CountyDetention b) CountyGuaranteedAttendanceProgram(GAP) c) OregonYouthAuthorityResidential d) Oregon Youth Authority Youth Correctional Facility 2. Reduction in new Criminal Referrals 3. ImprovementinSocialImpactsofJuvenileOffending a) Reduction in alcohol and other substance use Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 115 b) Increase in attendance, credit accumulation, grades, pro-social behavior, and positive peer association INFORMATION REQUESTED In line with its outlined priorities, the County requests the following information from providers of juvenile justice interventions that would be well-suited for an outcomes-based contracting model. Please be as specific as possible in responding to the questions below and provide relevant information: A. Proposed Services. Target Population 1. Please respond to each of the following components regarding your organization’s existing and proposed program(s): a) Describe how your organization proposes to address risk factors of the youth in the Target Population defined in Section A of the County’s Priorities, and their families to achieve the desired outcomes. b) Describe how your organization proposes to increase positive youth development factors, strengths, and assets in the Target Population defined in Section A of the County’s Priorities, and their families to achieve the desired outcomes. 2. Given the estimated eligible population that might volunteer for the intervention: a) Describe how your organization will provide the intervention to the maximum number of youth volunteering and what your organization’s capacity is to deliver the service. b) Describe how your organization will make the service available to meet the needs of eligible youth from any city or community within Marion County. Intervention 1. Briefly provide an overview of the existing or proposed program components, and the evidence, research, or promising practice basis for the service/interventions provided. a) If your organization provides this service currently or has done so in the past, describe how the intervention services are delivered. If your organization has not provided this intervention, describe the proposed service delivery model. b) Describe how your organization meets the needs of monolingual Spanish youth and families, and delivers services in accordance with accepted culturally competent service delivery. c) Describe how your organization meets the needs of families and provides youth and family support outside of traditional business hours. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 116 d) Describe the length of the program and how successful completion of the program is identified or defined. B. Impact Measurement. The specific outcomes of interest to the County are detailed in Section C of County’s Priorities. 1. Describe how your organization anticipates achieving the outcomes outlined in Section C. 2. Describe your organization’s capacity to evaluate the service delivery and the program development design process to target interventions to achieve the desired outcomes. C. Service Costs. 1. Provide a budget for the services delivered on the form provided with this RFI (Attachment “XYZ”). Include in your budget narrative descriptions and explanations of the FTE and positions required, calculations for materials and services, and a justification of any indirect costs. In addition, identify the cost per participant. D. Evaluation. 1. Describe any experience your organization has with independent/third-party evaluators. Include challenges arising during evaluation and lessons learned. E. Data Access 1. What mechanisms does your organization rely on to track and evaluate outcomes? Distinguish between handwritten notes, digital data, or government records to which your organization has access. 2. If your organization currently accesses government data, describe the sources and how your organization uses them. If your organization does not currently access government data, what sources will support more effective targeting of the intervention and measurement of outcomes? 3. Describe how your organization will track achievement of the outcomes described in Section C. 4. What experience does your organization have with handling protected data (FERPA, HIPAA, etc.)? Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 117 Appendix G: Evaluator Process & Requirements INTERNAL NOTE: This document and all evaluator work stream should be undertaken once the following are completed: 1) agreement on target population and outcome definitions, 2) service provider identification and 3) finalized structure and duration of intervention. The template included below is a component of the evaluator selection processes. The contents highlight RFI Purpose (brief background), the county’s priorities (target population, outcomes, and necessary intervention components), and requested information (evaluation design, relevant data, service costs, evaluation provider). For completion of the RFI template, government should consider standard legal language for public records, RFI disclosure, and any other legal requirements. Lastly, consideration should be given for the submission timeline (e.g. time for follow up questions or hosting of any meetings for respondents) and vendor response requirements (e.g. automated/digital response template or paper submissions). PURPOSE The purpose of this RFI is to help Marion County (the County) obtain information to support procurement of third-party evaluation services for juvenile justice programs. The County seeks to implement an outcomes-based contract to fund services for at-risk youth in the County. A concurrent RFI has been issued to obtain information on promising programs and interventions that have a strong evidence base to support achievement of desired outcomes. COUNTY’S PRIORITIES Marion County Juvenile Department is seeking to provide intensive in-home intervention for youth at risk of out of home placement. With this intervention we expect the target youth (described below) to have fewer new criminal referrals, reduced detention admissions, engagement or re-engagement in education or vocational training and remain in their home community. The target population to be served in the intervention are youth who meet the following criteria: 1. ages 13 through 18 2. placedoncountyprobation 3. assessed as medium through high-risk based on Juvenile Crime Prevention (JCP) criteria (see Appendix A) 4. have 8 or more of these 15 specific JCP items: a) Substance use beyond experimental use (uses alcohol and/or drugs regularly) b) Current substance use is causing problems in youth's life (youth is having problems with school, the law, family, friends or community related to alcohol/drug use) c) Youth accepts responsibility for behavior d) Youth pre-occupied with delinquent or anti-social behavior e) Friends disapprove of unlawful behavior f) Friends engage in unlawful or serious acting-out behavior g) Has friends who have been suspended or expelled or dropped out of school Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 118 h) Substanceabusingfriends i) Communicates effectively with family members (both verbal and nonverbal shared communication with healthy relationship boundaries) j) Poor family supervision and control (don't know where youth goes, what youth does, and has little influence in such matters) k) Hasclose,positive,supportiverelationshipwithatleastonefamilymember l) Significant school attachment/commitment (has significant involvement/commitment to school) m) Academic failure (recently failed, or currently failing two or more classes) n) Family actively involved in helping youth succeed in school o) Involved in constructive extra-curricular activities (sports, clubs, music, theater, arts, etc.) Based on historical trends, the eligible target population is estimated to be approximately 100 county youths throughout the duration of a calendar year. As a voluntary service, we expect less than that number to accept services. The County will choose programs that are evidence-based and have demonstrated success in achieving measureable outcomes, that can be scaled up while maintaining fidelity to the program model. Mandatory requirements of a successful program are as follows: 5. Bicultural/bilingual competency in service delivery 6. Intensiveservicesdeliveredinanin-homesetting 7. CapacitytoserveyouththroughoutMarionCounty 8. Willingness to participate in rigorous impact evaluation (e.g. propensity score matching) The County has identified the following outcomes for the target population to be measured in any contract: 4. Reductioninout-of-homeplacements: a) CountyDetention b) CountyGuaranteedAttendanceProgram(GAP) c) OregonYouthAuthorityResidential d) Oregon Youth Authority Youth Correctional Facility 5. Reduction in new Criminal Referrals a) Improvement in in Social Impacts of Juvenile Offending Reduction in alcohol and other substance use b) Increase in attendance, credit accumulation, grades, pro-social behavior, and positive peer association EVALUATOR RESPONSIBILITIES In order to ensure that the County only pays for services that achieve the stated outcomes, the County will also procure an evaluator to measure these outcomes for youth receiving the intervention. The anticipated evaluation process is included below. 1. Work with County to establish data sharing agreement(s) across government data bases and/or with a service provider Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 119 2. Acquire and merge data from multiple sources (e.g. Juvenile Justice, Education, Service Provider etc.) to create unified view of youth 3. Design an appropriately rigorous statistical analysis method (without random assignment of services) to evaluate the effectiveness of the intervention. Compare outcomes for youth receiving contracted program against expected outcomes, propensity matched control group, or other estimate of the counterfactual to assess if statistically significant improvements are produced. 4. Execute the program evaluation reporting by presenting the degree of impact of the intervention on the outcomes of interest. These estimates may be the basis for payments for the intervention. 5. Generate report on outcomes to determine whether contractual conditions for outcome payments have been met. The County views prior experience with juvenile justice research in Oregon as a prerequisite for successfully completing the evaluation. INFORMATION REQUESTED In line with its outlined priorities, the County requests the following information from interested parties on evaluation options that would be well-suited for an outcomes-based contract. Please be as specific as possible in responding to the below questions and provide relevant information: A. Evaluation Design. Based on the project design and outcomes described above, propose an evaluation methodology best suited to attributing impact to the intervention without using randomized assignment for services. Please describe your experience with any evaluation methodologies recommended. B. Relevant Data. Marion County expects to provide a data-access agreement with evaluators for Juvenile Justice Information system data. Provide a description of what other data would be required for your proposed methodology. Please identify where you expect to acquire those data and include description of any prior data-access agreements you have completed for other projects. C. Evaluation Costs. Provide a budget for evaluation services on the form provided with this RFI (Attachment “XYZ”). Include in your budget narrative descriptions and explanations of the FTE and positions required and a justification of any indirect costs. In addition, identify the cost per participant, if the size of the population served impacts the total costs. D. Evaluator Qualifications. Please describe your organization’s capacity to deliver the services discussed in the Evaluator Responsibilities section. In that description include the following: 1. The Institutional Review Board structure used to oversee human subjects research. 2. Relevant experience with public sector clients or juvenile justice interventions, specifically in the state of Oregon. 3. Ifavailable,shareasampleevaluationreport. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 120 Appendix H: Cost of Placement in OYA Residential and YCF Source: Oregon Youth Authority. Research and Evaluation. Evaluation of rates of escalation to OYA among Multnomah County youth enrolled in the Youth Villages Intercept Program. March 2016 Table 7 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 121 Appendix I: Oregon Victimization Costs by Crime Type Source: Oregon Criminal Justice Commission. Cost-Benefit Methodology. July 2011 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 122 Appendix J: County Administrative Cost for New Criminal Referrals Probation Administrative Cost for New Criminal Referrals after Probation Placement Source: Marion County (A) Probation officer salary with benefits (B) Annual hours at 40 hours a week (C) Imputed hourly wage (= A / B) (D) Hours required to process criminal referral with petition (E) Cost of administrative time for criminal referral (= C x D) $102,740 2,080 $49 8 $392 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 123 Appendix K: Restitution Amount per Criminal Referral Jan-June 2013 July-Dec 2013 # of youths 23 18 Total $ restitution $60,893 $143,814 Jan-June 2014 16 $9,184 July-Dec 2014 26 $40,234 Average 21 $63,531 $ / crime $2,648 $7,990 $574 $1,547 $3,190 Source: Marion County For purposes of the cost / benefit analysis, using $3,200 as average restitution amount Incidence of Restitution Offenses Among Target Population Youth Source: Marion County For the purposes of the cost / benefit analysis using 40% as incidence rate Target Population 2013 2014 Avg Youth with offense resulting in restitution 41 42 42 Probation eligible youth 114 113 114 Percent of population with restitution 36% 37% 37% Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 124 Appendix L: Criminal Referrals per Youth on Probation by Placement Type Source: Marion County Note: Youth can age out of juvenile system. Arrests after age 17 are not reflected here. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 125 Appendix M: Total Cost Assumptions in Absence of an Intervention (by Scenario) Note: Headcount and length of stay are based on Exhibit 10 with adjustment for voluntary entry Cost per day figures are based on averages for Marion County and Oregon Youth Authority (2015) New referrals during probation period is based on Marion County youth (2012-2014) Restitution amount and incidence based on Marion County youth (2013-2015) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 126 Note: Headcount and length of stay are based on Exhibit 10 with adjustment for voluntary entry Cost per day figures are based on averages for Marion County and Oregon Youth Authority (2015) New referrals during probation period is based on Marion County youth (2012-2014) Restitution amount and incidence based on Marion County youth (2013-2015) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 127 Note: Headcount and length of stay are based on Exhibit 10 with adjustment for voluntary entry Cost per day figures are based on averages for Marion County and Oregon Youth Authority (2015) New referrals during probation period is based on Marion County youth (2012-2014) Restitution amount and incidence based on Marion County youth (2013-2015) Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 128 Appendix N: Literature Review: Quantifying Social Costs The cost methodology of the feasibility assessment to determine cost of placement and social costs attempts to estimate the costs borne by the incarcerated individual and by society, including would-have-been victims. There are a handful of studies that assess these topics, with conclusions listed below. The feasibility reviewed the July 2011 Cost-Benefit Methodology report issued by the Criminal Justice Commission of the State of Oregon, the Washington State Institute for Public Policy’s Benefit-Cost analysis for Juvenile Justice, since their analysis is conducted in a State similar to Oregon and is highly rigorous, and the Justice Policy Institute’s “$ticker $hock Calculating the Full Price Tag for Youth Incarceration,” report. Criminal Justice Commission Cost-Benefit Methodology Overview The report from the Criminal Justice Commission (CJC) quantifies costs avoided by programs that prevent future crime. In so doing, they answered two questions: What are the costs of crime to the taxpayer that can be avoided? What are the benefits to the would-have-been victim? Their analysis broke those questions out further into: 1. Taxpayer savings a. Costofarrest b. Costofconviction c. Cost of incarceration d. Costofprobation e. Costofpost-prisonsupervision 2. Victimbenefits a. OutofPocket i. Lost property ii. Lost productivity (reduction in future earnings) iii. Mental health iv. Social services v. Medical care b. QualityofLife The following table from the CJC report lists the benefits that both the taxpayer (government) and the victim (society) would accrue if one felony conviction of different types is avoided. Because of the target population of this feasibility, this CJC report is most informative for the magnitude of benefits that accrue if a Robbery, Aggravated Assault, or Property Crime is committed. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 129 Importantly, the report notes “it costs the same to arrest an offender for a robbery or an assault but robbery arrests are less likely to end in a conviction so the cost per felony conviction of a robbery arrest is higher. Put another way, on average it takes more robbery arrests to get a conviction than it does for an assault.” Ultimately, the CJC reported that with an average impact of 27% for the re-entry programs studied, that the reduction in felony crime provided a benefit to taxpayers of $8,631 and to would-have been victims of $14,388. Project Application While the analysis described above is for felony convictions avoided by re-entry programs, which is different than what this feasibility study focuses on, it does provide a contextual starting point – a basis for beginning to form an order of magnitude for social costs. This feasibility assessment, however, finds county-specific information for the exact target population to be served as a more accurate measure to determine societal costs. To be specific, the cost / benefit analyses in Exhibit 19 in Section: Economic & End Payer Assessment delve into unique headcount, criminal referral counts, criminal referral type, and specific restitution amounts – inputs that ultimate result in social costs that are lower but more precise than the one presented in the broader CJC analysis. Washington State Institute for Public Policy Benefit-Cost Analysis The CJC numbers offer a reasonable start to estimating societal costs, but are specific to re-entry programs’ savings from avoided felonies. Because different populations experience different negative outcomes, and because services vary widely in their intervention models, it is helpful to gauge how costs are evaluated for juvenile justice program in general. The benefit-cost analysis of the Washington State Institute for Public Policy (WSIPP) for juvenile justice programs is one such study that provides a range and average of benefits to costs for programs in general. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 130 Overview The WSIPP examined 28 different juvenile justice programs to determine “what works” to improve outcomes using a meta-analysis and to understand whether the benefits of each program exceeded its costs. The benefits quantified included labor market outcomes; earnings; educational attainment; avoided child abuse and neglect; alcohol, drug, and tobacco use; medical costs, teen birth outcomes, public assistance outcomes, and many other benefits.39 Total benefits of these programs, to both taxpayer and non-taxpayer, ranged from -$9,824 to $40,800. Subtracting costs from those benefits yielded a range from -$12,615 to $36,930. The average benefit minus cost from the range of programs was $10,226. Although five of the 28 programs resulted in greater costs than benefits (and in some cases negative benefits even before costs were subtracted), the majority of programs resulted in some positive benefit after costs were subtracted.40 This average number is in line with what we see from the CJC re-entry benefit cost analysis. Project Application The Steering Committee for this feasibility assessment believes lowered drug abuse, increased positive education outcomes, and the effects of MST should be scrutinized as potential benefits from a project. The average benefit-cost of $10,226 by WSIPP above includes both drug abuse and education outcomes. We also looked at what benefit-cost programs focused solely on Substance Abuse and Education had, to understand that context. Substance Abuse  The Substance Abuse Treatment for Youth studied by the WSIPP included three programs, only one of which was effective at producing benefits larger than costs, the Teen Marijuana Check-Up program. Substance Abuse Treatment for Adults, on the other hand, showed a large number of benefit producing programs, with a range from $35,452 to -$17,315. The average benefit after costs were subtracted was $5,731.  Since it is unknown how effective an intervention for this project would be in reducing substance abuse or if the achieved outcomes are the same as this in the WSIPP programs, these estimates are excluded from the cost benefit analysis of the feasibility assessment. Education 39 The full methodology used in the benefit-cost analysis can be found here: http://www.wsipp.wa.gov/TechnicalDocumentation/WsippBenefitCostTechnicalDocumentation.pdf 40 Those non-positive benefit-cost programs are: Intense Supervision (parole), Diversion with services (vs. simple release), Other chemical dependency treatment for juveniles (non-therapeutic communities), Scared Straight, and Intensive Supervision (probation). Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 131  The 49 K-12 Educational programs analyzed by WSIPP, similarly, showed mostly positive benefit-cost numbers, with a range from $32,831 to -$13061, and on average having benefits outweigh costs by $6,182.  Similar to the limitations to included substance abuse estimates in the cost/benefit analysis of the feasibility assessment, it is unknown how effective an intervention for this project in achieved the education outcomes reviewed in the WSIPP programs. Multisystemic Therapy (MST) The WSIPP analysis found MST to have a benefit of $9,316. While the data point serves as good reference, there are a number of limitations would prevent the cost benefit analysis of the feasibility assessment to include these values.  As highlighted in Stephanie Klietz’s “Cost-Benefit Analysis of Multisystemic Therapy with Serious and Violent Juvenile Offenders,” the WSIPP study was less than ideal, because the analysis did not use studies that randomized to treatment or control conditions. The WSIPP study also did not include as wide of a range of governmental or crime victimization expenditures as Klietz would have preferred, and measured benefits over a 12 month period, which may exclude reduced recidivism after the first year. Further, Klietz identified that although the WSIPP study was “extremely rigorous” it was “limited because long-term clinical outcomes were estimated and then used to determine long-term costs and benefits.”  Klietz measured 176 youth that were part of an original clinical trial for an additional number of years (on average 13.7 years for the entire population). Measuring the benefits and costs Klietz found $50,000 (assuming one victimization per conviction) to $202,000 (assuming multiple victimizations) benefits after costs were subtracted. This is equivalent to between $8 and $27 return to taxpayers for every dollar spent.  Kleitz implicitly assumes that MST has a 13.7 year effect – that any difference in the benefit-cost ratio between the treatment and control groups was due to MST, even nearly 14 years after receiving initial treatment. Because Klietz tracked the youth over 13.7 years, statistically significant findings for MST in studies over shorter time frames (such as in this feasibility assessment) should not be considered comparable. That would assume that MST benefits over 13.7 years are as effective in a shorter time span. To truly verify this, one would want to understand how statistically significant the impact of MST was, not just the benefit-cost of the follow- up period.  Klietz shows that the average number of offences per recidivist is 3.43 for MST and 4.54 for the Individual therapy control group. She does not, however, provide any analysis of the statistical significance of the difference between these two numbers, or even if there is statistical significance. While the expected number of offenses Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 132 per participant (not recidivists) is more divergent at 1.72 per MST participant and 3.68 per Individual in the Individual therapy control condition, there is no statistical information provided. Because of this, it is unclear if the difference between the two groups was caused by random variation or to which factor to attribute the benefit- costs of $50,000 to $200,000 to. “$ticker $hock Calculating the Full Price Tag for Youth Incarceration” Overview In the report by the Justice Policy Institute, Oregon showed the following costs for confinement: Cost per day: $263; Cost per 3 months: $23,670; Cost per 6 months: $47,340; Cost per Year: $95,995. It is noted that these represent the highest cost confinement option provided to the researchers, and do not include societal or youth-level costs that may accrue over a lifetime. Citing Mark Cohen from Vanderbilt University in his paper, “New Evidence on the Monetary Value of Saving a High Risk Youth,” the report claims that “the present value of saving a 14-year-old high risk juvenile from a life of crime to range from $2.6 to $5.3 million.” While this certainly is “sticker shock,” it is important to note that this number is for a youth who would have otherwise had a ‘life of crime.’ Project Application When determining the cost-avoidance of social and youth outcomes of this project, we must take into account all the individuals in the target population, not just those who are the worst offenders, as well as the impact of the program, which is unlikely to be 100%. Because the target population defined in Exhibit 6 will not be completely filled with individuals that would have otherwise had a ‘life of crime,’ and because social programs are not perfectly effective, the $2.6 to $5.3 million estimate of cost avoidance is determined to be much too high. Conclusion Given these analysis, we believe it is safe to assume an order of magnitude close to $10,000- $20,000 for avoided costs that accrue to the individual and society. However, there seems to be a wide range of effective programs and academic disagreement over how to account for benefits that may accrue in the distant future. As such, the conservative approach applied to the cost / benefit analysis outlined in this feasibility report (see Section: Economic & End Payer Assessment), is most reasonable given its application to a specific set of youths, their behaviors (as determined by criminal referrals), and severity borne by society (as determined by restitution amounts). Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 133 Appendix O: Requirements for Establishing a Special Purpose Entity (SPE) If required, below is an overview of general requirements the formation of a SPE would entail for either a Limited Liability Corporation or nonprofit 501(c)(3).: Timeline Considerations Limited Liability Corporation (LLC) 2-4 weeks to set up  May need a 501(c)(3) fiscal sponsor  May not be able to receive certain grants, unless it has a 501(c)(3)fiscal sponsor Nonprofit 501(c)(3) May take up to 1 year to obtain IRS approval; until then it needs a 501(c)(3) fiscal sponsor Documents required for an SPE could include all or some of the items listed below: EXHIBIT 30: SPE Formation Documents Typically Drafted for Launched Projects a. Budget b. By-Laws c. Articles of Incorporation d. Clerk of Certifications e. Employer Identification Number (IRS) f. Action by Directors g. Action by Incorporator h. Conflict of Interest (Internal) i. Good Standing j. Certificate of Authority Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 134 Appendix P: Oregon Registration Form of a Nonprofit 501(c)(3) Registering a new business entity is done via the Secretary of State’s Office. Registration can be done online (http://sos.oregon.gov/business/Pages/register.aspx). A full form copy is listed below. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 135 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 136 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 137 Appendix Q: Launched Project Example – Santa Clara County Flow of Services, Funds and Contracts: List of contracts including Government Payer: • Pay for Success Contract: Santa Clara County vs. Abode • Medi-Cal Contract / Ramp-Up Agreement: Santa Clara County vs. Abode • Evaluator Agreement: Evaluator vs. Abode (County party to the agreement) Project’s Executive Steering Committee: Purpose/Primary Responsibilities The Executive Steering Committee is tasked with providing overall strategic guidance to the project and evaluating compliance with (and approving amendments to) the funding plan, evaluation plan, and project budget. • Review Quarterly Financials • Amend Project Budget (as long as it does not increase aggregate budget) • Amend the Evaluation Plan (as long as it does not change the success definition or success payment terms) • Review Project Budget after allocation of shared savings Services Payments Success Payments Funders 1 Provide up-front financing Abode Services Repays funders if outcomes me Medi-Cal Contract / Ramp-Up Agreement Distribute Medi-Cal Reimburse- ments Loan and/or Grant Agreement(s) Delivers Mental Health t services Abode Services 3 Provides payment for services 265 $8M Success Payments if outcomes met County of Santa Clara Delivers Services Target Population Pay for Success Contract Based on evaluation results, determines payment amount Designs evaluation methods and measures outcomes Abode Services Evaluator Agreement 4 Evaluator Pay for Success Contract Provides technical assistance and financial arrangement Third Sector Capital Partners, Inc. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 138 • Oversee/revise modifications to program operations made by Operating Committee • Review Auditor reports/performance Members • Two representatives of the County – voting members • Two representatives of Abode Services, at least one of which must be the Executive Director or Associate Director – voting members • Additional representatives may be invited by the County, Abode, or any of the funders – non- voting members Meeting Frequency/Schedule • Quarterly in Y1, Semi-annually thereafter • Also at special meetings when needed/requested by Operating Committee (must meet within a week of request) Project’s Operating Committee: Purpose/Primary Responsibilities The Operating Committee is responsible for reviewing intake, ramp-up, and general program implementation, ensuring that all aspects of the program are running smoothly and working together to address any challenges that arise. • Review and approve Funding Requests from Abode • Review key program metrics, including (but not limited to): referrals, enrollments, housing placements, success months, etc. • Modify aspects of the program – including the housing plan – as long as such changes do not have a material impact on the project budget, success definition, funding plan or other key program elements. • Make recommendations to the Executive Steering Committee about changes to the evaluation plan, funding plan, or project budget. Members • One representative of the County – voting member • Associate Director of Abode Services (or designee) – voting member • Other attendees can include additional representatives of the County and Abode, as well as any funder representative that wishes to attend. None of the additional attendees are voting members. Meeting Frequency/Schedule • Monthly meetings • Special meetings, when called at least 2 days in advance. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 139 Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 140 Appendix R: Understanding the Conduits for Funding and Structuring a Project Most Pay for Success (PFS) project funds typically flow through a Special Purpose Vehicle (SPV). This memo summarizes the SPV, funding, oversight, and governance structures in the Massachusetts Juvenile Justice Initiative and the Cuyahoga County Partnering for Family Success Program, to the extent public. Pay for Success Key Players Service Provider delivers services with the goal of achieving the agreed-upon outcomes. The Service Provider receives partial to complete cost coverage and may receive performance payments. Government initiates the PFS contract. The Government is also the end payer for successful outcomes. Funders provide working capital to the Service Provider, Evaluator, Validator (if applicable), Programmatic Manager, Fiscal Service Agent, and Government Advisor. Evaluator leads rigorous evaluation design and measures progress towards outcomes based on contract requirements. Validator (Optional) verifies the method and findings of the project evaluation as performed by the Evaluator. Programmatic Manager manages the flow of information over the course of the project and is hired by the SPV. The Programmatic Manager works with the Service Provider and Evaluator to ensure that the project progresses as outlined in the contract. Fiscal Service Agent manages and records the flow of funds through the SPV and ensures that payments are made as outlined in loan agreements. Government Advisor provides technical assistance on the PFS contract and mechanisms. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 141 MASSACHUSETTS JUVENILE JUSTICE PAY FOR SUCCESS INITIATIVE Project Overview The Commonwealth of Massachusetts, Roca, Inc., and Third Sector Capital Partners, working with lenders Goldman Sachs, through its Social Impact Fund, The Kresge Foundation, and Living Cities and philanthropic grantors Laura and John Arnold Foundation, New Profit Inc., and The Boston Foundation, launched a Pay for Success (PFS) initiative in January 2014 to reduce recidivism and improve employment outcomes for young men at high risk of re-offending in the Boston, Chelsea, and Springfield, Massachusetts areas. The initiative will allow Roca to provide its high-impact intervention to 929 at-risk young men aged 17 to 24 who are in the probation or parole system, are exiting the juvenile justice system, or are leaving the custody of the Suffolk, Essex, Hampden, and Middlesex Houses of Correction or the state's Department of Correction. Roca’s intervention aims to reduce recidivism and increase employment through intensive street outreach and targeted life skills, education, and employment programming. The Roca intervention is delivered over an intensive two-year period followed by two years of follow-up engagement. Massachusetts will make up to $28 million in success payments for this seven-year project, which is the largest investment in a PFS initiative in the U.S. to date. Key Players Service Provider—Roca, Inc. is a nonprofit that has delivered an evidence-based high impact intervention to young people in the Chelsea and Springfield, Massachusetts areas for 25 years. Roca’s model is based on proven behavioral change theories and trains high- risk young men in job readiness, educational readiness, and life skills. The Roca intervention establishes transformative relationships and uses targeted life skills, education, and employment programming to support young men in developing the skills necessary to reduce violence and create positive behavioral changes. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 142 Senior Lender Junior Lenders Grantors Provider Deferred Upfront Funding Project Cost Financing Government Payor Success Payment Youth Services Inc. 501(c)3 Special Purpose Vehicle Service Provider Independent Evaluator Evaluation Validator Programmatic Manager Fiscal Service Agent PM Deferred Government Payer—The Commonwealth of Massachusetts makes payments for success if the independent third-party evaluator and validator determine that Roca’s program has reduced the number of days that participating young men spend in prison, has improved their job readiness, and has increased their employment. The project intermediary/SPV will then distribute any success payments from the Commonwealth to funders. At higher levels of success the funders can receive a small percentage return on their funding in return for assuming the up-front financial risk. Project Intermediary—Third Sector Capital Partners, Inc. is a nonprofit advisory firm that works with government, service providers, and funders to develop and launch PFS projects. As project intermediary for this initiative, Third Sector arranged project funding and is responsible, through the SPV created for these purposes, for overseeing project implementation, distributing funding to Roca and managing repayment to funders. New Profit Inc., a national venture philanthropy fund will provide additional management support. Funders—Goldman Sachs, Kresge Foundation, Living Cities, Laura and John Arnold Foundation, New Profit, Inc. and The Boston Foundation provided a total of $18 million in upfront funding for the project. Independent Evaluator—The Urban Institute is an independent evaluation and research institute and is responsible for implementing a statistical evaluation to measure the impact of the Roca intervention. The project will use a Randomized Controlled Trial, the gold standard of rigorous evaluation. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 143 Independent Validator—Public Consulting Group (PCG) will be responsible for assessing the proposed evaluation methodology and verifying whether outcome targets are met. PCG was selected by Massachusetts through a formal procurement. Financial Flows Underpinning Services Funding for the Massachusetts initiative includes commercial and philanthropic funding as well as risk sharing, in the form of deferred service fees, from the service provider (Roca) and project manager (Third Sector Capital Partners, Inc.). The external funders assume project risk by financing services up front with the promise of repayment from government sources only in the event of success.  Goldman Sachs will provide $8 million in senior loan financing at an interest rate of 5%  Kresge Foundation will provide $1.33 million in junior loan financing at an interest rate of 2%  Living Cities will provide $1.33 million in junior loan financing at an interest rate of 2%  Laura and John Arnold Foundation will provide $3.34 7 million in grants  New Profit will provide $1.81 million in grants  The Boston Foundation will provide $300,000 in grants Innovatively, both Roca and Third Sector deferred operational fees to provide a stake in the project’s success. Third Sector deferred $50,000 (15%) of its management fees; it will only be paid that portion of its fees if results are achieved. Roca deferred $3.26 million (15%) of its service fees; it will only be paid that portion of its fees if results are achieved. In total, the SPV, Youth Services Inc. will receive approximately $19.65 million in upfront funding, drawn down from external funders on a quarterly basis and paying current interest. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 144 Senior Lender Junior Lenders Grantors Provider Deferred $8.00M $2.66M $5.45M $3.50M PM Deferred $0.05M A total of $19.65M in upfront funding and deferred fees were collected by SPV A total of $25.56M in project fees will be serviced by the SPV over 6 years Project Loan/Grant Funding Project Fees Servicing Youth Services Inc. 501(c)3 Special Purpose Vehicle $23.3M $0.71M $0.16M $0.37M $0.56M $0.25M Service Provider Independent Evaluator Evaluation Validator Programmatic Manager Fiscal Service Agent Financial Arranger The combination of upfront external funding and utilization of Jobs Readiness Payments from the Commonwealth will service the $25.56 million in project costs over six years. These project costs are made up of the following:  Roca’s operational costs of $23.3 million  Independent Evaluator fees of $0.71 million  Evaluation Validator fees of $0.16 million  Programmatic Management fees of $0.37 million  Fiscal Services, including auditing, tax management and accounting fees of $0.56 million  Financing arrangement fees of $0.25 million Success payments will be made between the second and seventh years of this initiative based on decreases in incarceration, increases in job readiness, and increases in employment attained by the young men participating in the Roca program. Success Payments made in earlier years will help service project fees that are not serviced by the external funding provided by the $19.66 million in upfront funding and the deferred fees from Roca and Third Sector. The Commonwealth will make three types of Success Payments:  Payments for decreases in incarceration represent the majority of the success payments and are based on a graduated payment schedule where the Commonwealth pays increasing amounts for each day that participants avoid incarceration as compared to similar young men who are not in the program. The payment rates are based on associated savings to the Commonwealth as shown below. The minimum reduction in incarceration necessary for payments to be made is 5.2%. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 145  Payments for increases in job readiness are $1000 for each participant in each quarter that a Roca participant engages with a Roca youth worker nine or more times, with each engagement helping young men address barriers to employment and move toward economic independence.  Payments for increases in employment are $750 for each participant in each quarter that a Roca participant is employed as compared to similar young men who are not in the program. The graphic below illustrates the composition, proportionality and timing of inflows to the SPV over the project’s six years. Prog. Manager Deferred Service Payments Provider Deferred Service Payments Initial Funder Capital Financing Government Success Payments Year1 Year2 Year3 Year4 Year5 Year6 Wrap- up The graphic below illustrates the composition, proportionality and timing of outflows to the SPV over the project’s six years. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 146 Additional Success Payments Funder Interest Payments Funder Capital Repayment Prog. Manager & Evaluator Fees Provider Working/Start-up Capital & Fees Year1 Year2 Year3 Year4 Year5 Year6 Wrap- up Repayment Flows at Project Wind-Up Massachusetts will repay funders only if Roca’s services are proven to produce positive societal outcomes and savings for the Commonwealth, as confirmed by the final Evaluation Report produced by the Urban Institute. The Evaluator is implementing a Randomized Controlled Trial (RCT) to measure individual employment, job readiness, and incarceration outcomes The project’s target impact is a 40% decrease in days of incarceration. At this level of impact, the project would generate budgetary savings to the Commonwealth equal to the cost of delivering services. If the project achieves its target impact, funders will be repaid using success payments from the Commonwealth’s Social Innovation Financing Trust Fund as follows:  The Goldman Sachs Social Impact Fund will be repaid its principal funding and a base annual interest rate of 5.0%;  The Kresge Foundation and Living Cities will be repaid their principal funding and a base annual interest rate of 2.0%;  Roca and Third Sector will be paid their deferred service fees; In addition, at higher levels of impact, project and funders will receive the following fees:  Roca: Up to approximately $0.75 million  Goldman Sachs: Up to approximately $0.93 million  The Kresge Foundation and Living Cities: Up to approximately $500,000 ($250,000 each) Any remaining PFS payments, which could be up to $4.9 million depending on the level of impact achieved, will be used to recycle philanthropic funding. Laura and John Arnold Foundation will use recycled funding to support future PFS initiatives. New Profit and The Boston Foundation will use recycled funding to support scaling of Roca. Any remaining capital left in the SPV will go to support the project’s operations. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 147 Senior Lender Junior Lenders Grantors Provider Deferred PM Deferred $9.66M $3.26M $4.56M $4.20M $0.07M Government Payor $28M Youth Services Inc. 501(c)3 Special Purpose Vehicle Programmatic Manager sends Evaluation Report to Government Evaluator sends Final Evaluation Report to the Programmatic Manager Programmatic Manager Independent Evaluator Evaluation Validator Success Payment Repayment of Loan/ Deferred Fee/Grant Evaluation Report Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 148 Oversight and Governance Mechanisms The Operating Committee meets on a monthly basis to highlight any concerns, discuss trends, and monitor progress of Roca’s target population under the Pay for Success Contract.  Attendance: Each Operational Meeting shall be attended by the following o Project Manager, representative of Youth Services Inc. o Roca Inc, represented by the Executive Director for Massachusetts and/or its Pay for Success Administrator o the State’s Department of Youth Services, represented by the Assistant Commissioner of Operations o the State’s Executive Office’s Administrative of Finance, represented by the Secretary o A representative from each of Office of the Commissioner of Probation (OCP), Department of Correction (DOC), Essex County Sheriff’s Department (ECSD), Hampden County Sheriff’s Department (HCSD), Middlesex Sheriff’s Office (MSO), Suffolk County Sheriff’s Department (SCSD), and Massachusetts Parole Board (MPB) shall have the right but not the obligation to attend the Operational Meetings o Subject to reasonable advance notice to the Parties, each Lender and Grantor may attend and observe a maximum of two Operational Meetings per calendar year o The independent Evaluator may be requested by the Operating Committee members to attend  Agenda: The agenda for each Operational Meeting shall be prepared by the Programmatic Manager and shall include at least the following: o A description of any significant changes to the Roca Intervention Model or the Roca Services that are being considered or implemented; o A discussion of the most recent Roca Report, a quarterly report prepared by Roca o A discussion of any indictments or incarcerations of Roca Youth; o A discussion of any Roca Youth that have been released from prison and re-enrolled in the Roca Services; o A discussion of the Referral Process and any changes that should are being considered o A discussion of the Project Manager’s role and any changes that should are being considered o A discussion of any changes to the anticipated funding needs of Roca or Youth Services Inc. in connection with their performance under this Contract.  Voting: The Operational Meeting participants will attempt to resolve all issues by unanimous consent of the Commonwealth, Roca and the Project Manager, on behalf of Youth Services, Inc. Any issues that cannot be resolved by the Operating Committee members may be forwarded to the Oversight Committee. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 149 Independent Evaluator Monthly Report Monthly Report Service Provider Quarterly and Annual Reports Convenes Monthly Meetings Operations Committee Programmatic Manager Chairs Quarterly Meetings Governance Committee Funders The Governance Committee meets on a quarterly basis to discuss any amendments to the funding plan, the Pay for Success Contract, the Evaluation Plan and/or Contract and the relevant funding agreements  Attendance: o Youth Services Inc. will be represented by the Programmatic Manager o Roca will be represented by the Founder and CEO and/or Pay for Success Administrator o The Commonwealth will be represented by the Commissioner of the Department of Youth Services and Secretary of the Executive Office of Administration and Finance. o The Independent Evaluator o Each Lender and Grantor is entitled to have one representative attend meetings  Agenda: Updates to and discussion of Project operations, evaluation, progress, and any other relevant issues  Voting Governance: The Oversight Committee will attempt to resolve all issues by unanimous consent of the Commonwealth, Roca and YSI. Any issues that the Oversight Committee cannot resolve by unanimous consent may be submitted by agreement of the Parties for resolution by the Massachusetts Office of Dispute Resolution, provided that no such resolution shall affect payments due hereunder without the consent of the Lender Committee. The Parties shall share equally in the cost of retaining such expert. SPV Liabilities Non-public information. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 150 CUYAHOGA COUNTY PARTNERING FOR FAMILY SUCCESS PROJECT Project Overview The County of Cuyahoga, Ohio, encompassing the greater Cleveland area, launched the nation’s first county-level Pay for Success (PFS) project in partnership with FrontLine Service, a comprehensive continuum of care service provider for homeless persons in Ohio. The Partnering for Family Success Program, the first PFS project in the combined areas of homelessness and child welfare, will deliver intensive 12-15 month treatment to 135 families over five years to reduce the length of stay in out-of-home foster care placement for children whose families are homeless. In Cuyahoga County, for children placed in out-of-home foster care to be reunited with their families, their caregivers must be able to provide a safe and stable home environment. Unfortunately, many of these caregivers are homeless, struggle with domestic violence, substance abuse and mental illness and are in need of critical services to address these issues in order to be able to care for their children. As a result, their children—nearly two-thirds of whom are under the age of six—spend significantly longer lengths of stay in out-of-home foster care and are left profoundly impacted by the loss of consistent caregivers. This extended time in the child welfare system results in poor outcomes for the County’s most vulnerable families and leads to higher costs to the County. Cuyahoga County is the first county in the nation to establish a special fund for PFS. The Social Impact Financing Fund was approved in July 2014, with co-sponsorship by County Executive Ed FitzGerald and County Councilman Dale Miller. Private funders and philanthropic organizations will provide a total of $4 million in upfront funding for the Partnering for Family Success Program. Cuyahoga County will repay these funders only if FrontLine’s services are proven to shorten the length of stay in out-of-home foster care. Cuyahoga County will pay a maximum of $5 million in success payments for this five-year project. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 151 Key Players S e r Senior Lender v i c e Junior Lenders Upfront Funding Grantors Government Payor Success Payment Cuyahoga PFS LLC Special Purpose Vehicle P r o v i d e r — F rontLine Service is a nonprofit agency that operates the most comprehensive continuum of care for homeless people in Ohio, serving more than 20,000 adults and children each year. FrontLine’s primary intervention in the Partnering for Family Success Program is Critical Time Intervention (CTI), an evidence-based model for assisting vulnerable populations through major place-based transitions. FrontLine will link families to housing and will use CTI to provide emotional and practical support during the critical transition of leaving a homeless shelter to becoming stably housed. FrontLine will also provide age- appropriate and evidence-based trauma services that will strengthen healthy and secure caregiver-child relationships. By providing caregivers critical access to housing before they are reunited with their children, this program will help caregivers to more successfully receive mental health services, complete substance abuse counseling, access job interviews and will allow for an increased amount of child visitations with their caregivers in a safe and stable home. These factors will empower the County’s Division of Children and Family Services to grant reunification decisions to these families sooner and allow children to exit foster care earlier. Housing Providers—The Cuyahoga Metropolitan Housing Authority (CMHA), Emerald Development & Economic Network, Inc. (EDEN), and Famicos Foundation (Famicos) are local housing providers that will provide public housing and/or voucher- based housing resources for the Program’s families. In a landmark action, CMHA has created a high-priority public housing preference for caregivers enrolled in the Program. EDEN and Famicos have carved out annual housing for the Program out of variable annual vacancy spots. Project Cost Financing Service Provider Independent Evaluator Programmatic Manager Fiscal Service Agent Gov’t PFS Advisor Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 152 Referral Partners:  County’s Office of Homelessness oversees the Norma Herr Women’s Homeless Shelter, which will serve as a main referral source of caregivers for the Program.  The Domestic Violence & Child Advocacy Center is a domestic violence and child advocacy social service provider that provides housing shelter for female caregivers who experience domestic violence. It will serve as an additional referral source of caregivers for the Program.  County’s Division of Children and Family Services will confirm that caregivers sourced from both homeless shelters have an active child welfare case that is eligible for the Program. Project Manager & Fiscal Agent—Enterprise Community Partners, Inc. (Enterprise) is a national organization that creates and advocates for affordable homes in thriving communities linked to jobs, good schools, health care and transportation. Enterprise’s generational goal is to end housing insecurity in the U.S. As the project manager and fiscal agent for the Partnering for Family Success Program, Enterprise will disburse upfront payments from the funders to FrontLine and for other project costs through Cuyahoga PFS, LLC, an affiliated entity. In this capacity, Enterprise will also disburse any success payments from Cuyahoga County to the funders. Enterprise will monitor whether the Partnering for Family Success Program runs according to the PFS contract and will work to resolve any issues that arise throughout the life of the project, leveraging its Cleveland- based staff and in-house national housing experts. Independent Evaluator—Case Western Reserve University will conduct a 5-year RCT and 2-year process evaluation of FrontLine’s evaluation. Government Payer—Cuyahoga County will make success payments if the independent evaluator, Case Western Reserve University, determines that children enrolled in the Partnering for Family Success Program (the treatment group) have avoided more out-of- home placement days compared to children placed in a comparison group that is not served by FrontLine. The Project Manager will distribute any success payments from Cuyahoga County to the funders. At higher levels of success, some funders may receive a modest return for assuming the upfront funding risk. The County will make payments if, and only if, Case Western Reserve University’s evaluation proves the program can effect a reduction in foster care days spent by these children. Government Advisor—Third Sector Capital Partners, Inc. is a nonprofit advisory firm that builds the capacity of government, service providers and funders to develop and launch PFS projects nationwide. Since 2012, Third Sector has facilitated Cuyahoga County’s exploration of PFS contracting, advised the implementation and ramp-up design, led the Program’s contract construction and arranged upfront funding. Third Sector has supported capacity building within the County and community to develop future PFS projects. As an ongoing Project Advisor, Third Sector will support the Project Manager during Program implementation. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 153 Funders—A consortium of five funders, including The Reinvestment Fund, The George Gund Foundation, The Cleveland Foundation, Nonprofit Finance Fund and Sisters of Charity Foundation of Cleveland have provided upfront funding of $4 million for the program. Financial Flows Underpinning Services A consortium of five funders have provided upfront funding of $4 million for the program:  The Reinvestment Fund will provide $1,575,000 in senior loan funding at 5% interest.  The George Gund Foundation will provide $1,000,000 in subordinate loan funding at 2% interest for $725,000 and at 0% interest for $275,000.  The Cleveland Foundation will provide $750,000 in subordinate loan funding at 2% interest.  Nonprofit Finance Fund will provide $325,000 in subordinate loan funding at 2% interest.  Sisters of Charity Foundation of Cleveland will provide two $75,000 recoverable grants and will also provide subordinate loan funding of $200,000 at 2% interest. A total of $4M in upfront funding were collected by SPV A total of $3.525M in project fees will be serviced by the SPV over 5 years Project Loan/Grant Funding Project Fees Servicing Senior Lender Junior Lenders $2.275M Grantors $1.575M $0.150M Cuyahoga PFS LLC Special Purpose Vehicle $2.327M $0.425M $0.400M $0.170M $0.200M Service Provider Independent Evaluator Programmatic Manager Fiscal Service Agent Gov’t PFS Advisor Of that upfront funding, approximately $3.53 million will service the following project fees:  FrontLine’s operational fees of $2.327 million  Case Western Reserve University’s evaluation fees of $0.425 million Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 154  Enterprise’s programmatic management fees of $0.400 million  Enterprise’s fiscal services fees of $0.170 million  Third Sector’s financial arrangement fees of $0.2.000 million In additional to servicing project fees, the upfront funding also serviced quarterly interest rate payments. The graphic below illustrates the composition, proportionality and timing of inflows to the SPV over the project’s five years. Initial Funder Capital Financing Government Success Payments Year 1 Year 2 Year 3 Year 4 Year 5 Wrap- up The graphic below illustrates the composition, proportionality and timing of outflows to the SPV over the project’s five years. Additional Success Payments Funder Interest Payments Funder Capital Repayment Prog. Manager & Evaluator Fees Provider Working/Start-up Capital & Fees Year 1 Year 2 Year 3 Year 4 Year 5 Wrap-up Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 155 Oversight and Governance Mechanisms The Operating Committee meets on a monthly basis to highlight any concerns, discuss trends, and monitor progress of FrontLine’s target population under the Pay for Success Contract and to facilitate successful outcomes. In particular, the Operating Committee shall focus on the enrollment and referral process of Clients into FrontLine’s Program, the housing resources available to the Target Population, identifying and monitoring program trends, and monitoring the progress of the Target Population under this Contract.  Attendance: Each Operational Meeting shall be attended by the following o Enterprise Community Partners, Inc., the Programmatic Manager representative of Cuyahoga PFS LLC o FrontLine Services, represented by the Executive Director, Chief Operating Officer and/or its Pay for Success Director o The County, represented by the Director of the Department of Children and Family Services and/or the Pay for Success Department of Children and Family Services/Office of Homeless Services Coordinator o Case Western Reserve University, the independent evaluator, represented by the Lead Evaluator, Dr. David Crampton o It is anticipated that such other members may include representatives from Cuyahoga County Office of Homeless Services, Cuyahoga County Office of Job and Family Services, the Domestic Violence and Child Advocacy Center, Cuyahoga Metropolitan Housing Authority, the Emerald Development and Economic Network o Each Lender and Grantor may request to attend any meeting by written request to the Project Manager  Agenda: The agenda for each Operational Meeting shall be prepared by the Programmatic Manager and shall include at least the following: o A discussion of the most recent FrontLine Report, a quarterly report prepared by FrontLine and the most recent monthly report from the Independent Evaluator o A description of any significant changes to the FrontLine Services that are being considered or implemented; o A discussion of the referral processes at the Department of Children and Family Services and the Domestic Violence Children Advocacy Center o A discussion of the housing resources available to families enrolled in FrontLine’s Services and access and placement process with CMHA, Famicos, EDEN and the Continuum of Care and any changes that should be or are being considered or implemented o A discussion of the engagement of Clients into the FrontLine Program and any changes that should be or are being considered or implemented; o A discussion of the retention levels of Clients into FrontLine Services and any changes that should be or are being considered or implemented; o A discussion of the implementation and operation of the RCT and any changes that should be or are being considered or implemented Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 156 o A discussion of the FrontLine participation in DCFS case conference and performance during team decision meetings and any changes that should be or are being considered or implemented; o A discussion of any critical incidents involving Clients over the past month and related County communications planning; o A discussion of any changes to the anticipated funding needs of FrontLine or SPV in connection with their performance under this Contract; o A discussion of issues related to systemic barriers to family reunification o A discussion of issues related to income and benefits as related to Clients and access thereto; and o A discussion of the evaluation of Clients to date. Voting: The Operating Committee will serve as an advisory committee to facilitate programmatic adjustments in the interest of improving the provision of services and/or the efficiency of the project and will not have any authority to bind the Parties in any way under this Contract. Monthly Report Service Provider Repayment Flows at Project Wind-Up As detailed in the Evaluation Plan, Independent Evaluator will implement a randomized controlled trial to measure the specific outcomes for purpose of calculating Success Payments. The RCT approach compares the total number of foster care days avoided by the “FrontLine Treatment Children,” defined as those children intended to be enrolled in the FrontLine Program and whose mothers are receiving FrontLine Services, with “Control Children” defined as those children placed in the control group and consequently not enrolled in the FrontLine Program. Independent Evaluator Monthly Report Programmatic Manager Chairs Quarterly Meetings Governance Committee Quarterly and Annual Reports Funders Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 157 Operations Committee Convenes Monthly Meetings In his final report, the Independent Evaluator will detail the calculation of the total reduction in foster care days experienced by the treatment group as compared with the control group. For each day reduced, the County will pay a per-diem success payment of $75. The County will make a maximum of Success Payment in the amount of $5 million. At levels above target impact level, the junior lenders will be able to participate in up to $1 million of additional success payments from the county. Senior Lender Limitations on Liability (a) Limited Recourse of County and FrontLine Against SPV Success Payment Repayment of Loan/ Deferred Fee/Grant Evaluation Report Junior Lenders Grantors $1.6M $3.3M $0.15M Government Payor Cuyahoga PFS LLC Special Purpose Vehicle $5M Programmatic Manager sends Evaluation Report to Government Evaluator sends Final Evaluation Report to the Programmatic Manager Programmatic Manager Independent Evaluator  All obligations of SPV under this Contract are subject to the SPV’s obligations under the Loan Documents and Grant Agreements.  The SPV shall not be obligated to take or omit to take any action which is in breach or inconsistent with the Loan Documents or Grant Agreements, as shall be determined by SPV in its sole discretion.  SPV shall be liable under this Contract solely for SPV’s misappropriation of funds under this Contract or commission of fraud with respect to the handling of funds in its custody.  SPV may rely on the genuineness of all signatures on all documents delivered to SPV.  SPV’s obligations under this Contract do not benefit from any recourse whatsoever to any member, director or officer of SPV.  Absent a misappropriation of funds or commission of fraud by SPV, County and FrontLine Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 158 shall have access only to the funds within the SPV Operating Account, subject to the rights and claims of third parties.  The other Parties’ sole remedy against SPV under this Contract, in the absence of a misappropriation of funds or the SPV’s commission of fraud in the handling of funds in its custody, is to terminate SPV under this Contract.  In the event the SPV is found to have misappropriated funds under this Contract, the SPV’s monetary liability shall be limited to the amount that is determined to have been so misappropriated. (b) Limited Recourse of the Funders Against SPV/Borrower  Absent fraud or misapplication of the funds, the Loan is payable solely from the Success Payments deposited to the Operating Account pursuant to the terms of the Pay for Success Contract. The Loan is a limited obligation of the Borrower, and there is no recourse to any director or officer of the Borrower.  The Borrower, shall be personally liable solely for Borrower’s misappropriation of funds or commission of fraud with respect to the handling of funds in its custody  Borrower may rely on the genuineness of all documents delivered to the Borrower  Absent a misappropriation of funds or commission of fraud by Borrower (in which case Lenders shall have all remedies available to it at law or in equity), Lender shall have recourse against Borrower only to the extent of the funds within the Operating Account and to no other assets.  If Borrower shall be found liable for fraud or the misappropriation of funds, then its liability shall be limited to the amount of such mishandled funds. Academic Carve-Outs Subject to the Data User Agreements and any publicity provision requirements of County of any Funder (as defined under the PFS Contract), Evaluator may use the project findings that are public information for educational and academic purposes. Third Sector Capital Partners, Inc. is a nonprofit advisory firm. The information contained herein reflects Third Sector’s current understanding of project characteristics. This understanding is based solely upon the information provided to Third Sector by project parties and additional information that is in the public domain. Third Sector makes no representation or warranty as to the accuracy or completeness of the information contained herein. Third Sector submits this information solely in connection with your consideration of the Marion and Multnomah Pay for Success Project. The funds (and any potential return) described herein are not securities. No representations or warranties of any kind are intended or should be inferred with respect to the project described herein. The plans, intentions or expectations disclosed herein may not be achieved and, even if achieved, actual results or future events could differ materially from plans, Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 159 intentions and expectations. Any obligation to update the information contained herein is expressly disclaimed. Oregon Juvenile Justice Pay for Success (PFS) Feasibility Assessment – Marion & Multnomah Counties 160