Fostering Opportunities Jefferson County, CO Pay for Success

OSF REGISTRIES Help Donate Join Login Fostering Opportunities - Jefferson County Pay for Success Public registration Overview Files Wiki Components 0 Links 0 Analytics Comments 0 Collapse Summary Summary Provide a narrative summary of what is contained in this registration, or how it differs from prior registrations. (optional) OSF Registration Prior to Data Collection Revised Date 10/29/2018 Title Fostering Opportunities: Jefferson County Pay for Success Project Authorship Elysia Clemens, PhD Research Questions Confirmatory RQ1: What is the impact of Fostering Opportunities on credit accumulation? RQ2: What is the impact of Fostering Opportunities on attendance? Exploratory RQ3: What is the impact of Fostering Opportunities on students’ odds of being suspended at all, and on the number of times students are suspended? RQ4: What is the impact of Fostering Opportunities on students’ being on-track for graduation at the end of the project? Hypothesis for each Confirmatory Research Question RQ1: Fostering Opportunities will have a positive impact on credit accumulation for students entering the study between grades 7-10, who are in the foster care system as compared to business-as-usual. RQ2: Fostering Opportunities will have a positive impact on attendance as for students entering the study between grades 7-10, who are in the foster care system as compared to business-as-usual. Sampling Plan Existing Data – Select one option Registration prior to creation of data: As of the date of submission of this research plan for preregistration, the data have not yet been collected, created, or realized. Explanation of existing data Administrative data from Jefferson County Public Schools and Jefferson County Human Services will be the primary data sources for this study. Jefferson County Public Schools will request transcripts for students enrolled in the study and transfer to another school district. The project team is actively monitoring transfer patterns and has discussed with the adjacent school district, Denver Public Schools, setting up a data sharing agreement for this project. The project team anticipates that Denver Public Schools will provide data if requested. Data Collection Procedures Population of subjects The study population are 7th to 10th grade students who will attend the Jefferson County School District between 2019 and 2022. Inclusion rules • Involvement in foster care in Jefferson County Human Services • Enrollment in Jefferson County Public Schools • School year grades 7th to 10th at time of randomization Exclusion rules • Youth with juvenile delinquency cases • Youth who exit the school district prior to randomization, or within three days of randomization Recruitment Our target sample is readily identifiable and accessible because of their status of being in foster care in Jefferson County Human Services. Recruitment and randomization will occur at the start of each semester. The recruitment window is defined by the school district’s school enrollment period (i.e., October 1 for the fall semester, February 15th for the spring semester). All recruitment, randomization, and enrollment in the study will be complete during these windows, from Fall of 2019 to Spring of 2022. The study will be fully enrolled when 80 youth are served in the treatment group at a given time, which is based on four specialists serving a caseload of 20 students each. We project the following target sample sizes based on churn in students receiving full services during the pilot study and the feasibility of enrolling additional control students throughout the period of time where funds are dedicated to implementation. The churn rate from the pilot study was 40% per year, which we have translated into an estimate of 20% churn per semester. Payments None Study Timeline Table 1. Estimated Treatment Condition Size Assuming 20% Churn Per Semester. Fall Year 2019 Spring Year 2020 Fall Year 2020 Spring Year 2021 Fall Year 2021 Spring Year 2022 Cohort 1 80 End Year 1 End Year 2 Cohort 2 16 End Year 1 End Year 2 Cohort 3 16 End Year 1 End Year 2 Cohort 4 16 End Year 1 Cohort 5 16 End Year 1 Sample size and Sample Size Rationale Level of Analysis Individual level. Sample Size A total of 364 students will be recruited over the course of three school years (Table 1). Table 2. Estimated Total Sample Size Assuming Balanced Design for Preliminary Analyses and 40% Treatment/60% Control by the Final Analyses. The Preliminary Analyses will be conducted using data through Fall of Year 2. Preliminary Analyses: One Year Post- Randomization Final Analyses: One Year Post- Randomization Final Analyses: Two Years Post- Randomization Treatment 96 144 112 Control 96 220 168 Total 192 364 280 Note. A balanced design is presented for the preliminary analyses because there are more treatment slots available during the initial enrollment period and we want to ensure: (a) we only randomize each student once and (b) we are serving the maximum number of students. If it is feasible to enroll additional control students, then we will do so. We further assume in a given year there are only 80 slots available for services for students. In fact, we hope there will be 90 slots as we expect a 5th lead specialist for the team of school-based specialists will be able to maintain a 50 percent caseload; however, we are assuming just 80 actual spots to continue to be somewhat conservative with respect to enrollment targets. Power Analysis The statistical power analysis was conducted using the PowerUp software package. The power analysis was based on the following assumptions: probability of Type I error of .05 (i.e., alpha), two-tailed test, statistical power of .8, proportion of sample randomized to treatment of .4, percent of variance explained by covariates of .2, and a total sample size of 364. This analysis yielded a minimal detectable effect size of .24 for effects, one-year post-randomization. The sample size will be smaller for assessing the effects of the intervention two years post randomization due to limitations of committed funding for service delivery. The minimum detectable effect size for a sample of 280, using the same assumptions as above, is .27. Baseline Data and Detectable Percent Increase in Real World Terms Baseline data indicate a 76% attendance rate, with a pooled standard deviation of .35. The power analysis suggests that we will be able to detect an 8% increase in attendance rate or a change to 84%. There are approximately 175 instructional days in a school year, so this increase would mean attending 14 more days. This target is still below the district average attendance rates which are 94% for middle school students and 90% for high school students. Baseline data indicate a 67% course pass rate, with a pooled standard deviation of .36. The power analysis suggests that we will be able to detect a 9% increase in course pass rate or a change to 76%. Students typically enroll in and need to pass 6 courses a year to be on-track for high school graduation, so this increase means an increase from passing 4 of those courses to passing 4.5 courses on average. Two Years Post-Randomization. The sample size is smaller, and we can detect a 9% increase in attendance and a 10% increase in course pass rates. Stopping rule All available students will be randomized into treatment or control conditions until the end of the recruitment period, Fall of 2021. Variables Outcome variables • Domain 1 Attendance is defined in terms of class periods attended. All reasons for missing class (e.g., excused, unexcused) are treated as absences. For descriptive purposes, an attendance rate will be defined as: number of class periods attended divided by total possible class periods. • Domain 2 Course Credit Accrual is defined in two ways because course passing applies to middle school and high school students; whereas, on-track for graduation only applies to high school students. For the confirmatory question, we will focus only on course passing. For the exploratory question we will additionally assess on-track for graduation for high school students. o Course Passing is defined categorically by the school district. For descriptive purposes, a course passage rate will be defined as the number of courses passed divided by the total number of courses. o On-Track to Graduate is defined as whether a student has accumulated enough credits to be on-track to graduate within four-years of initially entering 9th grade. This definition will be operationalized based on the graduation requirements in the school district the student is attending, or was last enrolled in, at the time of evaluation. • Domain 3 Suspension Incidents is an unduplicated count of suspension incidents (not days). A suspension incident is inclusive of in-school and out-of-school suspensions. Manipulated variables • Treatment vs. Control Measured variables • Cohort • School level • Sibling The following covariates will also be considered • Age at entry into foster care • Incident of abuse or neglect within one year prior to randomization • Number of foster care placements within one year prior to randomization • Placement type at the time of randomization (e.g., kindship placement, county foster care home) • Demographic or prior year data on attendance or course pass rate where baseline equivalence was expected but not met. Indices Formula or precise description of method (if more complicated like factor analysis, note here and describe in analysis plan section) See outcome variables descriptions above. Design Plan Study Type  Experiment Blinding Describe who is aware of the experimental manipulations within a study  No blinding involved Study Design The purpose of this study is to estimate the one- and two-year impacts of the “Fostering Opportunities” intervention on school attendance and course pass rate. The intervention is being implemented with Pay for Success financing. In this case the Colorado Governor’s Office of State Budget and Planning and Community First Foundation are financing the implementation of the project. The study is a randomized control trial where 7th to 10th grade students who are in foster care at entry into the study will be randomly assigned to the treatment or business as usual conditions. The study will take place in Jefferson County (Jeffco), Colorado. The intervention is implemented by “specialists” hired by the school district. These specialists check in weekly with students, ensure caregivers and child welfare case workers have timely and accurate information about students’ educational progress, and consult with teachers on trauma-informed approaches to helping the students be successful in school. These specialists have a caseload of about 20 students and follow them through planned and unplanned school changes within Jefferson County Schools and to adjacent school districts. The intervention and the study designs assume that some students will transfer out of the school district, and procedures are in place to continue some aspects of service delivery and to track student outcomes. Randomization How and at what level The Independent Evaluators will use a computer-generated random number to assign child welfare case IDs to (a) Fostering Opportunities or (b) Business as Usual. Random assignment will be stratified by grade level, and sibling pairs will be randomized by alternating the random assignment based on the lowest- and the highest-grade level of the sibling group. Analysis Plan Describe one or more confirmatory analyses. All others must be noted as exploratory or hypothesis generating. Include description of predictor/independent variables and outcome variables. Statistical Models We will use two separate ANCOVAs to estimate the impact of Fostering Opportunities on students’ attendance and course pass rates. Covariates in the model will include: (a) cohort (see Table 1) and (b) school level. A sibling effect will be included if more than 10 percent of the sample are siblings. Early enrollment data suggests that the threshold for a sibling effect is likely to be met. For the confirmatory questions, we conducted a power analysis. The power analysis assumed that 20 percent of the variance would be accounted for by covariates. In addition to the covariates listed above the following will also be considered (a) age at entry into foster care (b) incident of abuse or neglect within in one year prior to randomization (c) number of foster care placements within one year prior to randomization, and (d) placement type at the time of randomization (e.g., kinship placement, county foster care home) and (e) demographic or prior year data on attendance or course pass rate where baseline equivalence was expected but not met. The decision rule for inclusion of a given criteria shall be Rubin’s matching criteria (see Baseline Equivalence Section). Transformations • Gender: male = 1, female = 0 • Race: dummy variables for all race and ethnicity categories • Placement type: dummy variables for all placement categories • Cohort: dummy variables for all cohort categories • School level: dummy variables for all school level categories • Sibling: Yes = 1, No = 0 • Suspension: Yes = 1, No = 0 • Abuse or neglect within one year prior to randomization: Yes = 1, No = 0 Project codebooks will be available for all variables included. Follow-up Analyses NA Inference Criteria criteria (e.g. p-values, model fit), one or two tailed tests, accounting for multiple conditions/tests p<.05, two tailed Data Exclusion Exclusion criteria and how outliers will be handled Missing Data Listwise deletion will be used for outcome data. For baseline data and covariates, a dummy variable will be used in combination with a flag for missing data. Exploratory Analysis – if any RQ3: To address the exploratory question related to suspensions, hurdle regression will be used to simultaneously model the odds of clearing the hurdle (i.e., being suspended at all) and, of those who were suspended, the odds of an increased number of suspensions. This hurdle regression analytic approach was selected because baseline data illustrated that just over half of the students were not suspended at all (56%), and the model is specifically designed for count data. The analysis yields two separate p-values for all effects, indicating statistical significance for both the odds of the event and the rate separately. Controls in the model will include: (a) an additional random effect to account for potential similarities in response among siblings (if appropriate based on sample size) and (b) school level. The effect of the intervention on the odds of being suspended at all is an experimental analysis. However, the effect of number of suspensions is non-experimental because the composition of the group that ‘cleared the hurdle’ and were suspended at least once occurred post random assignment. RQ4: Logistic regression will be used to model the odds of being on-track to graduate at the end of the project period. For this analysis, data from all cohorts and all semesters of the project, except the building year, will be included – regardless of length of time since a given student was randomized. A random effect to account for potential similarities in response among siblings (if appropriate based on sample size) will be included as a control in the model. Descriptive Analyses for Exploratory Purposes: To provide context for understanding the RCT findings, we will report basic descriptive statistics for data collected on all variables at all times of observation, including means, standard deviations, and proportions for all binary variables. For each programmatic outcome (i.e., Research Questions 1-4), we will present interaction plots over time, stratified by Fostering Opportunities/non-Fostering Opportunities groups, to compare the growth or decay in means and proportions between the two groups. Because some predictors can change within individuals, time-change plots will be constructed, in which the change in each response is plotted against the change in each predictor (Diggle et al., 2013). Plots will allow us to determine whether the relationships between predictors and responses are dynamic and develop over time. These analyses will inform program improvement. Other Baseline Equivalence of Groups We will use administrative data from the prior school year to assess baseline equivalence of students between treatment and control groups. We will determine whether there is a difference of more than .25 pooled standard deviation between the means for the outcome and demographic measures available prior to the start of the pilot, and we will confirm the ratios of variances of those outcomes remain between .50 and 2.00, satisfying the matching criteria of Rubin (2001). Discrete demographics (race, gender, special education status, and whether or not the student was in foster care during the prior year) will be compared using chi-squared tests of association (Agresti, 2012). Should baseline equivalence not be met, then we will include the variables where equivalence was not met as covariates in the statistical models. Correspondence with Ethical Standards for Research The research protocol has been reviewed and approved by the University of Denver’s Institutional Review Board (IRB). A Data Protection Agreement has been executed between Jefferson County Public Schools and the University of Denver to govern the data sharing and management processes. Jefferson County Human Services will facilitate parental active consent for releasing child welfare and educational data for this project. The intervention is oversubscribed, meaning that there are not sufficient resources to implement the intervention for all eligible students. Contributors Colorado Evaluation and Action Lab, Kristin Klopfenstein, Elysia Versen Clemens, Elizabeth Shahan, Alison Sheesley, and Whitney LeBoeuf Description RCT evaluation of interventions designed to promote on-time graduation for children in foster care. Registration type Open-Ended Registration Date registered October 29, 2018 Date created October 29, 2018 Registered from osf.io/8hvke Category Project Affiliated institutions Laura and John Arnold Foundation Citation osf.io/3mgwy Copyright © 2011-2019 Center for Open Science | Terms of Use | Privacy Policy | Status | API TOP Guidelines | Reproducibility Project: Psychology | Reproducibility Project: Cancer Biology This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use. Accept