Fostering Opportunities - Jefferson County Pay for Success
Provide a narrative summary of what is contained in this registration, or how it differs from prior registrations. (optional)
OSF Registration Prior to Data Collection
Revised Date 10/29/2018
Fostering Opportunities: Jefferson County Pay for Success Project
Elysia Clemens, PhD
RQ1: What is the impact of Fostering Opportunities on credit accumulation?
RQ2: What is the impact of Fostering Opportunities on attendance?
RQ3: What is the impact of Fostering Opportunities on students’ odds of being suspended at all, and on the number of times students are suspended?
RQ4: What is the impact of Fostering Opportunities on students’ being on-track for graduation at the end of the project?
Hypothesis for each Confirmatory Research Question
RQ1: Fostering Opportunities will have a positive impact on credit accumulation for students entering the study between grades 7-10, who are in the foster care system as compared to business-as-usual.
RQ2: Fostering Opportunities will have a positive impact on attendance as for students entering the study between grades 7-10, who are in the foster care system as compared to business-as-usual.
Existing Data – Select one option
Registration prior to creation of data: As of the date of submission of this research plan for preregistration, the data have not yet been collected, created, or realized.
Explanation of existing data
Administrative data from Jefferson County Public Schools and Jefferson County Human Services will be the primary data sources for this study. Jefferson County Public Schools will request transcripts for students enrolled in the study and transfer to another school district. The project team is actively monitoring transfer patterns and has discussed with the adjacent school district, Denver Public Schools, setting up a data sharing agreement for this project. The project team anticipates that Denver Public Schools will provide data if requested.
Data Collection Procedures
Population of subjects
The study population are 7th to 10th grade students who will attend the Jefferson County School District between 2019 and 2022.
• Involvement in foster care in Jefferson County Human Services
• Enrollment in Jefferson County Public Schools
• School year grades 7th to 10th at time of randomization
• Youth with juvenile delinquency cases
• Youth who exit the school district prior to randomization, or within three days of randomization
Our target sample is readily identifiable and accessible because of their status of being in foster care in Jefferson County Human Services. Recruitment and randomization will occur at the start of each semester. The recruitment window is defined by the school district’s school enrollment period (i.e., October 1 for the fall semester, February 15th for the spring semester). All recruitment, randomization, and enrollment in the study will be complete during these windows, from Fall of 2019 to Spring of 2022.
The study will be fully enrolled when 80 youth are served in the treatment group at a given time, which is based on four specialists serving a caseload of 20 students each. We project the following target sample sizes based on churn in students receiving full services during the pilot study and the feasibility of enrolling additional control students throughout the period of time where funds are dedicated to implementation. The churn rate from the pilot study was 40% per year, which we have translated into an estimate of 20% churn per semester.
Table 1. Estimated Treatment Condition Size Assuming 20% Churn Per Semester.
Year 2019 Spring Year 2020 Fall
Year 2020 Spring Year 2021 Fall
Year 2021 Spring
Cohort 1 80 End Year 1 End Year 2
Cohort 2 16 End Year 1 End Year 2
Cohort 3 16 End Year 1 End Year 2
Cohort 4 16 End Year 1
Cohort 5 16 End Year 1
Sample size and Sample Size Rationale
Level of Analysis
A total of 364 students will be recruited over the course of three school years (Table 1).
Table 2. Estimated Total Sample Size Assuming Balanced Design for Preliminary Analyses and 40% Treatment/60% Control by the Final Analyses. The Preliminary Analyses will be conducted using data through Fall of Year 2.
Preliminary Analyses: One Year Post- Randomization Final Analyses: One Year Post- Randomization Final Analyses: Two Years Post- Randomization
Treatment 96 144 112
Control 96 220 168
Total 192 364 280
Note. A balanced design is presented for the preliminary analyses because there are more treatment slots available during the initial enrollment period and we want to ensure: (a) we only randomize each student once and (b) we are serving the maximum number of students. If it is feasible to enroll additional control students, then we will do so. We further assume in a given year there are only 80 slots available for services for students. In fact, we hope there will be 90 slots as we expect a 5th lead specialist for the team of school-based specialists will be able to maintain a 50 percent caseload; however, we are assuming just 80 actual spots to continue to be somewhat conservative with respect to enrollment targets.
The statistical power analysis was conducted using the PowerUp software package. The power analysis was based on the following assumptions: probability of Type I error of .05 (i.e., alpha), two-tailed test, statistical power of .8, proportion of sample randomized to treatment of .4, percent of variance explained by covariates of .2, and a total sample size of 364. This analysis yielded a minimal detectable effect size of .24 for effects, one-year post-randomization.
The sample size will be smaller for assessing the effects of the intervention two years post randomization due to limitations of committed funding for service delivery. The minimum detectable effect size for a sample of 280, using the same assumptions as above, is .27.
Baseline Data and Detectable Percent Increase in Real World Terms
Baseline data indicate a 76% attendance rate, with a pooled standard deviation of .35. The power analysis suggests that we will be able to detect an 8% increase in attendance rate or a change to 84%. There are approximately 175 instructional days in a school year, so this increase would mean attending 14 more days. This target is still below the district average attendance rates which are 94% for middle school students and 90% for high school students.
Baseline data indicate a 67% course pass rate, with a pooled standard deviation of .36. The power analysis suggests that we will be able to detect a 9% increase in course pass rate or a change to 76%. Students typically enroll in and need to pass 6 courses a year to be on-track for high school graduation, so this increase means an increase from passing 4 of those courses to passing 4.5 courses on average.
Two Years Post-Randomization. The sample size is smaller, and we can detect a 9% increase in attendance and a 10% increase in course pass rates.
All available students will be randomized into treatment or control conditions until the end of the recruitment period, Fall of 2021.
• Domain 1 Attendance is defined in terms of class periods attended. All reasons for missing class (e.g., excused, unexcused) are treated as absences. For descriptive purposes, an attendance rate will be defined as: number of class periods attended divided by total possible class periods.
• Domain 2 Course Credit Accrual is defined in two ways because course passing applies to middle school and high school students; whereas, on-track for graduation only applies to high school students. For the confirmatory question, we will focus only on course passing. For the exploratory question we will additionally assess on-track for graduation for high school students.
o Course Passing is defined categorically by the school district. For descriptive purposes, a course passage rate will be defined as the number of courses passed divided by the total number of courses.
o On-Track to Graduate is defined as whether a student has accumulated enough credits to be on-track to graduate within four-years of initially entering 9th grade. This definition will be operationalized based on the graduation requirements in the school district the student is attending, or was last enrolled in, at the time of evaluation.
• Domain 3 Suspension Incidents is an unduplicated count of suspension incidents (not days). A suspension incident is inclusive of in-school and out-of-school suspensions.
• Treatment vs. Control
• School level
The following covariates will also be considered
• Age at entry into foster care
• Incident of abuse or neglect within one year prior to randomization
• Number of foster care placements within one year prior to randomization
• Placement type at the time of randomization (e.g., kindship placement, county foster care home)
• Demographic or prior year data on attendance or course pass rate where baseline equivalence was expected but not met.
Formula or precise description of method (if more complicated like factor analysis, note here and describe in analysis plan section)
See outcome variables descriptions above.
Describe who is aware of the experimental manipulations within a study
No blinding involved
The purpose of this study is to estimate the one- and two-year impacts of the “Fostering Opportunities” intervention on school attendance and course pass rate. The intervention is being implemented with Pay for Success financing. In this case the Colorado Governor’s Office of State Budget and Planning and Community First Foundation are financing the implementation of the project.
The study is a randomized control trial where 7th to 10th grade students who are in foster care at entry into the study will be randomly assigned to the treatment or business as usual conditions.
The study will take place in Jefferson County (Jeffco), Colorado. The intervention is implemented by “specialists” hired by the school district. These specialists check in weekly with students, ensure caregivers and child welfare case workers have timely and accurate information about students’ educational progress, and consult with teachers on trauma-informed approaches to helping the students be successful in school. These specialists have a caseload of about 20 students and follow them through planned and unplanned school changes within Jefferson County Schools and to adjacent school districts. The intervention and the study designs assume that some students will transfer out of the school district, and procedures are in place to continue some aspects of service delivery and to track student outcomes.
How and at what level
The Independent Evaluators will use a computer-generated random number to assign child welfare case IDs to (a) Fostering Opportunities or (b) Business as Usual. Random assignment will be stratified by grade level, and sibling pairs will be randomized by alternating the random assignment based on the lowest- and the highest-grade level of the sibling group.
Describe one or more confirmatory analyses. All others must be noted as exploratory or hypothesis generating. Include description of predictor/independent variables and outcome variables.
We will use two separate ANCOVAs to estimate the impact of Fostering Opportunities on students’ attendance and course pass rates. Covariates in the model will include: (a) cohort (see Table 1) and (b) school level. A sibling effect will be included if more than 10 percent of the sample are siblings. Early enrollment data suggests that the threshold for a sibling effect is likely to be met.
For the confirmatory questions, we conducted a power analysis. The power analysis assumed that 20 percent of the variance would be accounted for by covariates. In addition to the covariates listed above the following will also be considered (a) age at entry into foster care (b) incident of abuse or neglect within in one year prior to randomization (c) number of foster care placements within one year prior to randomization, and (d) placement type at the time of randomization (e.g., kinship placement, county foster care home) and (e) demographic or prior year data on attendance or course pass rate where baseline equivalence was expected but not met. The decision rule for inclusion of a given criteria shall be Rubin’s matching criteria (see Baseline Equivalence Section).
• Gender: male = 1, female = 0
• Race: dummy variables for all race and ethnicity categories
• Placement type: dummy variables for all placement categories
• Cohort: dummy variables for all cohort categories
• School level: dummy variables for all school level categories
• Sibling: Yes = 1, No = 0
• Suspension: Yes = 1, No = 0
• Abuse or neglect within one year prior to randomization: Yes = 1, No = 0
Project codebooks will be available for all variables included.
criteria (e.g. p-values, model fit), one or two tailed tests, accounting for multiple conditions/tests